Full Screen

How Search Engines Work

More Free Lessons in

Search Engine Optimization (SEO) View All →

Get cutting-edge digital marketing skills, know-how and strategy

This micro lesson is from one of our globally recognized digital marketing courses.

Start a FREE Course Preview Start a FREE Course Preview
Global Authority

The Global Authority

12 years delivering excellence


245,000+ Members

Join a global community


Associate Certification

Globally recognised


Membership Included

Toolkits, content & more

Digital Marketing - Study Notes:

Automated software

How do search engines actually work? Well, there are so many websites and so much content on the internet that search engines need to use a system through which they can comb through different websites, and discover and understand the content on them. This is achieved using automated software known as spiders, crawlers, or bots.

These terms can be quite interchangeable and to a wide extent they do a similar job. But they do have marginally different functions:

  • Spider: A spider is a program run by a search engine to build a summary of a website’s content. Spiders create a text-based summary of content and an address (URL) for each webpage.
  • Crawler: A crawler visits each webpage on a website and determines all of the hyperlinks on every individual page.
  • Robot: A robot is an automated computer program that visits websites and performs predefined tasks. Its job is to understand how to crawl and index pages on a given website.

Three steps

There are three basic steps a search engine takes when searching for content: crawling, indexing, and ranking.

  • Step 1 - Crawling: Crawling is when the search engine spiders or crawlers browse the web to identify and capture available content.
  • Step 2 - Indexing: Indexing is where online content is analyzed, organized, and stored in huge databases, so that users can search it efficiently.
  • Step 3 – Retrieval or Ranking: Ranking is the position where online content appears in the Search Engine Results Page (SERP) based on a specific user query.


During the indexing stage, search engines use an algorithm to help them process and understand web information. The algorithm operates within a set of rules and it’s a unique formula, and search engines use this formula to determine the significance of each individual web page.

Part of Google’s algorithm is called RankBrain. RankBrain is a machine-learning artificial intelligence system that helps Google process some of its search results, in particular rare or one-of-a-kind queries. Machine learning is where a computer teaches itself how to do something, rather than being taught by humans or following programming.

RankBrain was not introduced as a new way for Google to rank search results; it is simply part of Google’s overall search algorithm, a computer program that’s used to sort through the billions of pages it knows about and find the ones deemed most relevant for particular queries.


All our SEO efforts are focused on rank. The higher you rank, the more organic traffic you’re going to drive through to your site. It is also important to consider that factors such as location, language, and context will also effect whether a certain webpage ranks or not.

Aside from rank, there are a couple of other metrics that are important to monitor when optimizing websites:

  • Page authority: The ranking strength of a specific page
  • Domain authority: The ranking strength of an entire website domain
Back to Top
Shane Lyons

Shane Lyons is head of Search & Analytics at Mediaworks, an award-winning media and communications agency. He has been working in Digital both in Ireland and abroad for the last 8 years and now specializes in SEO and Analytics.

Data protection regulations affect almost all aspects of digital marketing. Therefore, DMI has produced a short course on GDPR for all of our students. If you wish to learn more about GDPR, you can do so here:

DMI Short Course: GDPR

The following pieces of content from the Digital Marketing Institute's Membership Library have been chosen to offer additional material that you might find interesting or insightful.

You can find more information and content like this on the Digital Marketing Institute's Membership Library

You will not be assessed on this content in your final exam.


    SEO Setup
    Shane Lyons
    Skills Expert

    This module introduces the key concepts for setting up SEO and the elements to consider before beginning an SEO project. It outlines the key technical elements you need to consider when implementing SEO, how to implement on-page SEO, and how to implement SEO in a mobile environment. It explains how a business can improve its SEO strategy and local SEO. It also demonstrates how to measure and report on the effectiveness of your SEO campaign and the key SEO metrics to measure.