How Search Engines Work

Digital Marketing - Study Notes:

Automated software

How do search engines actually work? Well, there are so many websites and so much content on the internet that search engines need to use a system through which they can comb through different websites, and discover and understand the content on them. This is achieved using automated software known as spiders, crawlers, or bots.

These terms can be quite interchangeable and to a wide extent they do a similar job. But they do have marginally different functions:

• Spider: A spider is a program run by a search engine to build a summary of a website’s content. Spiders create a text-based summary of content and an address (URL) for each webpage.
• Crawler: A crawler visits each webpage on a website and determines all of the hyperlinks on every individual page.
• Robot: A robot is an automated computer program that visits websites and performs predefined tasks. Its job is to understand how to crawl and index pages on a given website.

Three steps

There are three basic steps a search engine takes when searching for content: crawling, indexing, and ranking.

• Step 1 - Crawling: Crawling is when the search engine spiders or crawlers browse the web to identify and capture available content.
• Step 2 - Indexing: Indexing is where online content is analyzed, organized, and stored in huge databases, so that users can search it efficiently.
• Step 3 – Retrieval or Ranking: Ranking is the position where online content appears in the Search Engine Results Page (SERP) based on a specific user query.

RankBrain

During the indexing stage, search engines use an algorithm to help them process and understand web information. The algorithm operates within a set of rules and it’s a unique formula, and search engines use this formula to determine the significance of each individual web page.

Part of Google’s algorithm is called RankBrain. RankBrain is a machine-learning artificial intelligence system that helps Google process some of its search results, in particular rare or one-of-a-kind queries. Machine learning is where a computer teaches itself how to do something, rather than being taught by humans or following programming.

RankBrain was not introduced as a new way for Google to rank search results; it is simply part of Google’s overall search algorithm, a computer program that’s used to sort through the billions of pages it knows about and find the ones deemed most relevant for particular queries.

Rank

All our SEO efforts are focused on rank. The higher you rank, the more organic traffic you’re going to drive through to your site. It is also important to consider that factors such as location, language, and context will also effect whether a certain webpage ranks or not.

Aside from rank, there are a couple of other metrics that are important to monitor when optimizing websites:

• Page authority: The ranking strength of a specific page
• Domain authority: The ranking strength of an entire website domain