Full Screen

Robots.txt Files

More Free Lessons in

Search Engine Optimization (SEO) View All →

Get cutting-edge digital marketing skills, know-how and strategy

This micro lesson is from one of our globally recognized digital marketing courses.

Start a FREE Course Preview Start a FREE Course Preview
Global Authority

The Global Authority

12 years delivering excellence


245,000+ Members

Join a global community


Associate Certification

Globally recognised


Membership Included

Toolkits, content & more

Digital Marketing - Study Notes:

Instructions for robots

The robots.txt file is a simple text file placed on your web server, which tells web crawlers whether or not they should access a file on your website. The robots.txt file controls how search engine spiders see and interact with your web pages.

If there are any areas of your site that you would rather a search engine didn’t crawl – i.e. that you don’t want it to index – you can issue a robots.txt file that disallows the URL path that you ultimately want to block.

If you disallow any page that contains a forward slash, you can block all access to the site. You can also block individual folders or image files if you wish them not to be crawled. (See slide ‘Instructions for robots’ for examples of these instructions.)

You can check if a site has a robots.txt file from any browser. The robots.txt file is always located in the same place on any website, so it is easy to determine if a site has one. Just add "/robots.txt" to the end of a domain name.

Back to Top
Shane Lyons

Shane Lyons is head of Search & Analytics at Mediaworks, an award-winning media and communications agency. He has been working in Digital both in Ireland and abroad for the last 8 years and now specializes in SEO and Analytics.

Data protection regulations affect almost all aspects of digital marketing. Therefore, DMI has produced a short course on GDPR for all of our students. If you wish to learn more about GDPR, you can do so here:

DMI Short Course: GDPR

The following pieces of content from the Digital Marketing Institute's Membership Library have been chosen to offer additional material that you might find interesting or insightful.

You can find more information and content like this on the Digital Marketing Institute's Membership Library

You will not be assessed on this content in your final exam.


    SEO Setup
    Shane Lyons
    Skills Expert

    This module introduces the key concepts for setting up SEO and the elements to consider before beginning an SEO project. It outlines the key technical elements you need to consider when implementing SEO, how to implement on-page SEO, and how to implement SEO in a mobile environment. It explains how a business can improve its SEO strategy and local SEO. It also demonstrates how to measure and report on the effectiveness of your SEO campaign and the key SEO metrics to measure.