What does a search engine actually do?
A search engine does three jobs, one after the other. It finds pages by following links from pages it already knows. It reads each page it finds and keeps a processed copy in a huge database called the index. Then, when someone types a search, it picks pages from the index that answer the question and decides the order to show them in.
Those three jobs are usually called crawling, indexing and ranking. They happen at different times. Crawling and indexing run all day in the background, whether anyone is searching or not. Ranking happens in the fraction of a second after someone presses Enter, and it uses only what is already in the index. A page the search engine never found, or found and chose to leave out, cannot appear for any search at all.
That order explains most of the work in this course. Before anyone can argue about which bakery comes first for “sourdough near Mill Lane”, the Crumb and Crust menu page has to be reachable, readable and worth keeping. Search engine optimisation, usually shortened to SEO, is mostly the patient work of making sure those first two jobs go smoothly, and that the page says clearly what it offers.
How does a search engine find a page in the first place?
The program that fetches pages is called a crawler, or sometimes a bot or spider. Google's is called Googlebot and Microsoft's is Bingbot. A crawler starts from addresses it already knows, downloads each page, pulls out every link on it and adds the new addresses to its list of places to visit. Over time, following links from page to page, it reaches most of the public web.
Links are the main road in. If the bakery's home page links to the menu, the crawler finds the menu the next time it reads the home page. If no page on the bakery's site or anywhere else links to the menu, the crawler has no way to guess the address. A page like that is often called an orphan page.
There are two other ways in. A sitemap is a file that lists the addresses you want crawled, and you can tell search engines where it lives. Search engines also offer free tools, such as Google Search Console and Bing Webmaster Tools, where a site owner can submit a sitemap and ask for a single address to be crawled. Both help, and later lessons cover them. Links still do the most work, because the same links guide visitors around the site as well.
What does the crawler read on a page?
The crawler downloads the same HTML a browser receives. From it, the search engine pulls out the parts that describe the page: the <title> element, the meta description, the headings, the main text, the alt text on images, the links and where they point, and any instructions meant for crawlers, such as a robots meta tag.
Modern search engines also run the page's JavaScript, much as a browser does, so text added by a script can usually be seen. That rendering step can happen later than the first fetch, and it can fail if a script breaks. Text that is in the HTML from the start is the safest choice for anything you want found.
Google also reads pages as a phone would. Since the move to mobile-first indexing, the version of a page that a phone gets is the version that counts. A page that hides its prices on small screens is, as far as the index is concerned, a page without prices.
<head>
<title>Menu and prices | Crumb and Crust</title>
<meta name="description" content="Today's bread, buns and cakes on Mill Lane, with prices.">
</head>
<body>
<h1>Menu and prices</h1>
<a href="/about.html">About the bakery</a>
</body>What is the index, and why are some pages left out?
The index is the search engine's library: a processed copy of the pages it decided to keep, organised so it can look up words and topics quickly. Being in the index makes a page eligible to appear in results. It does not decide where the page appears, and for most searches most indexed pages never appear on the first screen.
Plenty of pages that a crawler fetches never make it into the index, and usually for sensible reasons. The page may ask to be left out with a noindex tag. It may be a near copy of another page, so the search engine keeps one and skips the rest. It may answer with an error, or show almost nothing worth keeping. Each of these has its own lesson later in the course.
Search Console shows this split for real sites, with a list of indexed pages and a list of pages that are not indexed, each with a reason. The Index view in this lab works the same way, in plain words of its own.
How is a search result put together?
A standard result has three parts: the site's name and address on top, a blue title link, and a short description underneath. The title link usually comes from your <title> element, and the description often comes from your meta description. Neither is guaranteed. Search engines rewrite titles that are missing, vague or stuffed with keywords, and they often replace the description with a sentence from the page that fits the search better.
Some results carry extras, such as an event's date and place, or a product's price. Those come from structured data, a small block of code that describes the page in a fixed vocabulary. Valid structured data makes a page eligible for the extras. The search engine still decides whether to show them.
The Result view in this lab draws a plain sketch of a desktop result, a phone result and the preview card apps show when someone shares the link. It shows the most likely version, based on your files, and says where each part came from.
What is the crawler lab on this page?
The lab is a pretend website made of a few small files you can edit: HTML pages, and later robots.txt, sitemap.xml and a redirects file. A pretend crawler starts at the home page, follows links, obeys the rules a real crawler would, and reports what it found. Nothing leaves this page, and the same files always give the same result.
The lab never ranks anything. Real ranking depends on the person searching, where they are, what else exists on the web and hundreds of signals no small simulation could copy honestly. The lab sticks to what can be checked: whether a page can be found, read and indexed, and how its result could look.
- Result: a sketch of the page in search results on desktop and phone, plus its share preview.
- Crawler: every address the crawler fetched or skipped, in order, with an Inspect button.
- Index: which pages are indexed, which are left out, and the reason for each.
- Checks: problems and warnings across the whole site, from missing titles to broken links.
What can you control, and what can you not?
You control a lot of the groundwork. You decide whether every page can be reached by a link, whether crawlers are allowed in, whether each page has a clear title and description, and whether the text answers what people are looking for. You also control whether the site works on a phone and loads quickly.
You do not control the order of results. Nobody outside the search engine does, and any person or tool that promises a first-place ranking is guessing at best. You also cannot pay a search engine to rank higher in the normal results. Paid ads exist, and they are labelled as ads.
The calm approach is to do the groundwork well, write pages that are useful to the people you want to reach, and then measure what happens with Search Console instead of guessing. That is the approach this course takes.
What do AI tools get wrong about how search works?
AI tools often repeat advice that stopped being true years ago. A common one is adding a <meta name="keywords"> tag; Google has ignored it for more than fifteen years. Another is to submit your site to search engines every week, when a single sitemap and good links are enough. Some tools still suggest a target keyword density, such as using a phrase in three per cent of your words, which no major search engine has ever confirmed and which tends to make text read badly.
They also blur crawling and indexing. An AI answer may tell you to block a page in robots.txt to keep it out of search, which stops the crawler reading the page but can still leave the bare address in results. And many will confidently promise rankings. Treat any sentence that says a change will put you on page one as a sign that the rest of the answer needs checking too.
Field notes
- A search engine finds pages through links, keeps a copy of some in its index, then picks results at search time.
- The crawler reads the same HTML a browser gets, so the title, headings and text all come from your code.
- Only pages in the index can appear in results, and being indexed never guarantees a high place.
- The title link usually comes from your title element, and the snippet often comes from your meta description.
Easy to get wrong
An address written as plain text is not a link, so crawlers do not follow it. Use an <a> element with an href, and give it words that say where it goes.
Crawlers discover pages through links and sitemaps. A page nothing points to can sit unseen for a long time, however good it is.
Google ignores the keywords meta tag. The title, the headings and the text people can read are what describe the page.
Questions people ask
How long does it take for a new page to appear in search?
Anywhere from a few hours to a few weeks. It depends on how often the crawler visits your site and how easily it can reach the new page. A link from your home page and an entry in your sitemap both help; Search Console can also ask for a single page to be crawled.
Do I have to pay to be in search results?
No. Crawling and indexing are free, and you cannot pay for a better position in the normal results. Paid ads are a separate system and are labelled as ads.
Is the crawler in this lab the same as Googlebot?
No. It is a small pretend crawler that follows the same published rules for links, robots.txt, noindex and sitemaps, so the lessons carry over. Real crawlers also render JavaScript, revisit pages on their own schedule and weigh far more signals.
Does every page on my site need to be in search?
No. Pages such as a basket, a members' area or a thank-you page after an order are of no use to someone searching. Later lessons show how to keep those out cleanly.