pdf 862 KB
12 years delivering excellence
Join a global community
Toolkits, content & more
The robots.txt file is a simple text file placed on your web server, which tells web crawlers whether or not they should access a file on your website. The robots.txt file controls how search engine spiders see and interact with your web pages.
If there are any areas of your site that you would rather a search engine didn’t crawl – i.e. that you don’t want it to index – you can issue a robots.txt file that disallows the URL path that you ultimately want to block.
If you disallow any page that contains a forward slash, you can block all access to the site. You can also block individual folders or image files if you wish them not to be crawled. (See slide ‘Instructions for robots’ for examples of these instructions.)
You can check if a site has a robots.txt file from any browser. The robots.txt file is always located in the same place on any website, so it is easy to determine if a site has one. Just add "/robots.txt" to the end of a domain name.Back to Top
Shane Lyons is head of Search & Analytics at Mediaworks, an award-winning media and communications agency. He has been working in Digital both in Ireland and abroad for the last 8 years and now specializes in SEO and Analytics.
Data protection regulations affect almost all aspects of digital marketing. Therefore, DMI has produced a short course on GDPR for all of our students. If you wish to learn more about GDPR, you can do so here:
The following pieces of content from the Digital Marketing Institute's Membership Library have been chosen to offer additional material that you might find interesting or insightful.
You can find more information and content like this on the Digital Marketing Institute's Membership Library
You will not be assessed on this content in your final exam.
ABOUT THIS DIGITAL MARKETING MODULE
This module introduces the key concepts for setting up SEO and the elements to consider before beginning an SEO project. It outlines the key technical elements you need to consider when implementing SEO, how to implement on-page SEO, and how to implement SEO in a mobile environment. It explains how a business can improve its SEO strategy and local SEO. It also demonstrates how to measure and report on the effectiveness of your SEO campaign and the key SEO metrics to measure.
Ready to learn more about Search Engine Optimization (SEO)?
Sign up for a FREE trial, and get access to more great content to help you level up your digital marketing career.