What are the three stages of search?
Every page that shows up in a search result has passed through three stages. First a crawler fetched it. Then the search engine read what it fetched and decided to keep it in the index. Last, when someone typed a search, the engine picked it from the index and chose where to place it. People call these stages crawling, indexing and ranking.
The stages run in that order and each one depends on the one before. A page that was never crawled cannot be indexed, because the search engine has nothing to read. A page that was crawled and left out of the index cannot rank, because ranking only looks inside the index. When a page is missing from search, the first question is always which stage it stopped at.
That question is what Search Console answers for real sites, and what the Index view answers in the lab. The fixes are completely different for each stage, which is why it pays to name the stage before you change anything.
What happens during crawling?
Crawling is fetching. The crawler keeps a list of addresses to visit, takes one, checks the site's robots.txt file to see whether it is allowed, and asks the server for the page. It reads the HTML that comes back, collects the links and adds any new addresses to its list.
A page can fail at this stage in a few ways. Nothing may link to it, so the crawler never learns the address. The robots.txt file may block it. The server may answer with an error, or with a redirect to somewhere else. In each case the crawler does not end up with the page's content.
Google crawls on its own schedule. A popular site that changes every hour gets visited far more often than a bakery that edits its menu once a month. You cannot set the schedule, though a sitemap and Search Console's URL Inspection tool can ask for a page to be looked at sooner. The lab crawls everything in one go, in a fixed order, so you can see the whole process at once.
# robots.txt on crumbandcrust.example
User-agent: *
Disallow: /hours.html
# The crawler may fetch every page except /hours.htmlWhat happens during indexing?
Indexing is reading and deciding. Once the search engine has the page, it pulls out the title, headings, text, links and images, runs any JavaScript it needs to, and works out what the page is about. Then it decides whether to keep the page in the index.
Pages are left out at this stage for reasons you can usually see. The page may carry a noindex robots meta tag, which is a direct request to leave it out. It may be a copy of another page, so the engine keeps one version and points the rest at it. It may answer "page not found" with a normal success code, which Google calls a soft 404. Or it may have so little on it that there is nothing worth keeping.
Google does not promise to index every page it crawls, even a perfectly fine one. Search Console's Pages report lists not-indexed pages with reasons such as "Excluded by 'noindex' tag" or "Crawled, currently not indexed". The second one means the page was read and judged not worth adding yet, and the usual answer is to make the page more useful.
What happens during ranking?
Ranking happens at the moment of a search. The engine works out what the person most likely means, finds the indexed pages that could answer it, and orders them. Google describes the inputs in broad terms: the meaning of the words, how relevant each page is, signs of quality and trust, how usable the page is, and the searcher's context, such as their location, language and device.
This means there is no single ranking for a page. The Crumb and Crust hours page might come first for someone on Mill Lane typing "crumb and crust opening times" and appear nowhere for someone in another city typing "bakery hours". Positions also change from day to day as other pages change.
Nobody outside the search engine can promise a position, and no single change to a page reliably moves it up. What you can do is make sure the page is crawled, indexed and clear about what it offers, then watch the Performance report in Search Console to see which searches it actually appears for.
Why does the lab never rank anything?
The lab covers crawling and indexing in full and stops there on purpose. Ranking depends on the person searching, every competing page on the web, and signals no small simulation could copy honestly. A pretend ranking would teach you to chase numbers the lab made up.
When a page in the lab reaches "indexed", read it as "eligible". The page has done everything a page can do to be considered. The Result view then shows how its result would most likely look if it appeared, which you can influence with your title and description. Where it would appear is a question the lab leaves open, as a real search engine does.
How is robots.txt different from noindex?
These two are mixed up more than anything else in SEO, and they act at different stages. The robots.txt file controls crawling: it tells crawlers which addresses they may fetch. A noindex tag controls indexing: it sits in the page and asks the engine to leave that page out of results.
Because they act at different stages, they can cancel each other out. If you block a page in robots.txt and also add noindex, the crawler never fetches the page, so it never sees the tag. Meanwhile, if other pages link to the blocked address, Google can still list the bare address with a note that no information is available. The lab shows this as "url-only".
The rule that follows is short. To keep a page out of results, let it be crawled and give it noindex. Use robots.txt for areas a crawler has no reason to fetch at all, and remember it is a polite request that well-behaved crawlers follow, with no protection for anything private. Private pages need a password.
<!-- In the <head> of cakes.html: keeps the page out of results -->
<meta name="robots" content="noindex">How can you tell which stage a page is stuck at?
Work through the stages in order and stop at the first one that fails. In the lab, the Crawler view tells you about crawling, and the Index view tells you about indexing. On a real site, Search Console's URL Inspection tool does both for one address.
- Never found: the address is missing from the Crawler log, or the Index view says it was never discovered. Add a link to it, or list it in the sitemap.
- Blocked: the Crawler log shows it skipped because of a
robots.txtrule. Change the rule if the page should be found. - Fetched but left out: the Index view gives a reason such as noindex, duplicate or thin. Fix that reason on the page itself.
- Indexed: the page is eligible. Anything further is about how clear and useful it is to the people searching.
What do AI tools get wrong about these stages?
AI assistants often tell you to "block the page in robots.txt" to remove it from Google. That stops crawling and leaves indexing to chance, and it hides any noindex tag from the crawler. Some go further and suggest a Noindex: line inside robots.txt. Google stopped supporting that in 2019 and the line does nothing.
They also promise outcomes the stages cannot deliver, such as "this change will get you to page one within a month". Getting indexed is something you can check and fix. Ranking is decided fresh for each search and nobody can guarantee it. When an AI answer mixes the two, fix the crawling and indexing parts and set the promise aside.
Field notes
- Crawling means fetching, indexing means reading and deciding to keep, and ranking happens at the moment of a search.
- A robots.txt rule stops the fetch, while noindex asks for a fetched page to be left out of results.
- A page blocked in robots.txt can still be listed from links, because the crawler never sees its noindex.
- When a page is missing, check each stage in order and stop at the first one that fails.
Easy to get wrong
A robots.txt block stops crawling only. The address can still be listed if other pages link to it, and the crawler never sees a noindex tag on a page it is not allowed to fetch.
A page left out of the index cannot appear for any search. Fix the earliest stage that fails before working on later ones.
Google ignores Noindex: lines in robots.txt and has since 2019. Use the robots meta tag in the page for indexing.
Questions people ask
If my page is indexed, will people see it?
Only if it is chosen for a search they make. Indexed means eligible. The Performance report in Search Console shows which searches it actually appeared for and how often.
How long after I fix a page will Google notice?
It depends on how often Google crawls your site, from hours to a few weeks. URL Inspection in Search Console has a Request indexing button that asks for one page to be crawled again, which can help after a fix.
Does Bing work the same way?
The same three stages apply to Bing and other search engines, and they read the same robots.txt rules and robots meta tags. Bing Webmaster Tools gives you reports much like Search Console.
Can a page rank without being crawled?
A blocked page can sometimes appear as a bare address if many pages link to it, with no title or description from the page itself. That is rarely useful to anyone, and it is a sign the block should be reconsidered.