Search engines: crawling, indexing and search results

Google first discovers the address, then tries to fetch and process the content. A known address, a successful server response and a fully rendered page are different stages. They therefore need to be distinguished when investigating access.

How does Google find a new page?

Google can discover a new address through links from pages it already knows or through a sitemap. Google needs to know the address before Googlebot can try to fetch the page. Discovery means the address is known; it does not mean the content has already been read.

Imagine a business publishing a new service page. A relevant link from its service overview gives both visitors and the search engine a route to the page. If the page can only be found by typing its exact address, that route is missing. A sitemap can help Google discover addresses but does not guarantee that they will be crawled or indexed. This is explained in Google’s sitemap guidance.

What happens when Googlebot fetches the page?

Googlebot requests the page’s content from the server and receives a response. A successful response may contain the page’s HTML, while a server error, login requirement or restriction can prevent access. An address can therefore be known without Google having been able to read the intended content.

Robots.txt is used to control what crawlers may fetch. The file is not the same as a directive to keep an address out of search results. A noindex directive needs to be readable to work as intended. Google’s explanation of robots.txt explains this distinction.

What does rendering mean?

Rendering means Google processes the page’s code, including JavaScript, to obtain the content the page displays. On some websites, the text is already present in the server’s HTML response. On others, JavaScript needs to run before the main text or links become available.

Illustrative example: a product name is visible in the browser but missing from the initial HTML. The check then needs to include the rendered version. Text looking correct in a screenshot does not, on its own, show which content Google has been able to process. Blocked resources or execution errors can produce a different result. Google explains how it processes JavaScript and why fetching and rendering need to be distinguished.

Indexing means that Google analyses the content and may store information about the page in its search index. The index is used when the search engine selects answers to a search. Being able to open a page, or Google having fetched it, therefore does not prove that it has been indexed.

What is analysed when a page is indexed?

Google analyses the page’s text and content signals, including its title, images and image alt text. Language and geographical relevance can also provide context. Indexing therefore involves more than saving a URL: the system needs to process what that address actually presents. This is described in Google’s explanation of indexing.

Illustrative example: a window replacement page needs an understandable offering and a title describing window replacement. A filename or photograph of a window does not replace text explaining the scope of the work. This example shows the information readers need to find; it is not a formula for how Google weights each element.

What happens when several addresses show the same content?

Google may choose a canonical, a representative address, from duplicate or very similar pages. The website can indicate its preferred address, but this is a signal to Google, not a binding decision.

If the same service page exists at two addresses, the website owner first needs to decide which should be the main address. Internal links, sitemaps and canonical markup should consistently reflect that choice. A canonical directive is also no substitute for clear, distinct content when the pages are meant to serve different purposes. Google’s canonical guidance describes the methods and their limitations.

Why is not every fetched page indexed?

A fetched page may be excluded from the index because of a readable noindex directive, duplicate content or insufficient content quality. When Google selects another address as canonical, that address may represent the shared content. These are examples of possible causes, not a diagnosis of a particular page. Google’s URL Inspection documentation describes these limitations. A status saying that a page has been fetched but not indexed identifies a state; it does not, on its own, establish the cause.

For your own website, the status needs to be connected to the exact address and the time of Google’s latest check. The guide to Google indexing takes that diagnosis further. The key distinction here is that discovery, fetching and indexing are not the same event.

An indexed page is assessed in relation to the search being made. Google uses several systems to understand queries and content and select relevant results. A page’s position can therefore differ between queries and search contexts.

Illustrative example: “what is technical SEO” asks for an explanation, while “help with technical SEO in Gothenburg” may be a step towards choosing a provider. An explanatory guide and a service page serve different purposes. The business needs to provide the right answer on each page, not simply repeat the same keywords on both.

Do the keywords have to match exactly?

No. A relevant page does not need to repeat the search query’s exact words. Google describes systems that work with relationships between words and concepts. At the same time, a shared word does not mean that the page fulfils the task behind the search. A clear explanation and an accurate scope help readers more than repetition. Google’s overview of ranking systems describes this understanding of language, among other things; it does not specify a recommended keyword density.

Why do two people get different search results?

The search context, such as location, language and device, can affect the results Google shows. A comparison therefore needs to record the exact query and search environment. A single test from one computer is not a measure of every user’s results. Google’s explanation of the search process describes how content is selected from the index for the particular search.

Can advertising buy a better organic position?

No. Paid advertising does not buy a higher position in Google’s organic search results. Google separates advertising from its automated assessment of organic results. An advertisement and an indexed page can therefore appear for the same search without the advertising purchase determining the page’s organic position. See Google’s description of result selection.

Start with what you can actually observe, then choose the appropriate check. Absence from a single search result does not reveal whether the problem concerns discovery, indexing or the page’s relevance to the query.

Search Console’s URL Inspection tool distinguishes information about the indexed version from a live test of the current page. The live test can help investigate access and processing but cannot predict every indexing decision. The URL Inspection documentation explains what the different sources of information show. The table above points to the next check; it is not a diagnosis of your website.

Choose an important page and document its address, intended purpose and current status before changing anything. This makes it possible to follow up on the right problem and distinguish a completed action from an actual outcome.

When the page is readable but its answer does not meet the search need, the on-page SEO guide helps you review content, headings and links together. When the problem requires a broader technical review, SOVRA Digital’s technical SEO work explains how to define the scope. The next step is chosen according to the confirmed need.

What happens when an indexed page changes?

A published change and Google’s updated record are two different things. The new version needs to be fetched and processed before it can affect what Google knows about the page. Google’s recrawling guidance describes requests for individual URLs and sitemaps for multiple pages, without guaranteeing indexing.

Illustrative example: you change a service’s scope on Tuesday. Visitors see the new text after publication, but Search Console may still display information from an earlier crawl. Compare the change date with the report date before concluding that the change has been assessed. Use the Google indexing guide to check the specific address.