On this page
- The three stages at a glance
- Stage 1: Crawling
- Rendering: seeing the page like a browser
- Stage 2: Indexing
- Stage 3: Ranking
- Serving results and SERP features
- How AI answers use search engines
- How Google discovers new pages
- What Google looks at when ranking pages
- How local search results work
- How other search engines differ
- Common reasons search engines skip pages
- A simple checklist to help search engines understand your site
- A short history of how search engines evolved
- How search engines understand language
- How search engines handle duplicate content
- How freshness and updates affect rankings
- How search engines use structured data
- What this means for your website
- How search engines fight spam
- How search engines treat new websites
- How mobile devices changed search
- How search engines measure page experience
- Search engine terms explained
- How to use this knowledge to rank better
- What search engines cannot do well
- How to check if Google has found your page
- FAQs
Every time you type a search, Google looks through an index of hundreds of billions of pages and returns results in a fraction of a second. That speed is only possible because the hard work happens long before you search.
This guide explains each step in plain language: how crawlers find pages, how search engines read and store them, how ranking systems choose what to show, and how newer AI answers fit into the picture. Along the way you will see what each stage means for your own website.
A written SEO and AI search plan for your site in 1 business day.
Get my SEO plan Chat on WhatsApp- Crawling is discovery: bots follow links and sitemaps to find pages.
- Indexing is understanding: search engines read, render and store pages they judge useful.
- Ranking is choosing: systems weigh relevance, quality, usability and context for each search.
- If a page is not crawled or indexed, it cannot rank, however good it is.
The three stages at a glance
| Stage | What happens | What can go wrong |
|---|---|---|
| Crawling | Bots discover URLs and download pages | Blocked by robots.txt, no internal links, server errors |
| Indexing | Pages are rendered, analysed and stored | Duplicate content, noindex tags, thin pages, JavaScript issues |
| Ranking and serving | The best pages are chosen and shown for each search | Weak relevance, low quality, poor experience, stronger competitors |
Stage 1: Crawling
Crawling is how search engines discover pages. Programs called search crawlers, such as Googlebot and Bingbot, request URLs, download the content and look for links to more pages.
How crawlers find new pages
They follow links from pages they already know, read XML sitemaps you submit and revisit known URLs to check for changes.
What robots.txt does
A robots.txt file tells crawlers which parts of a site not to crawl. It is a request, not a lock, and it controls crawling rather than indexing.
Crawl budget
Large sites have a limit on how much Google will crawl at once, shaped by server speed and how important pages seem. Small sites rarely need to worry about crawl budget.
Server responses
Each request returns a status code. 200 means OK, 301 means moved, 404 means not found and 5xx means the server failed.
Rendering: seeing the page like a browser
Many websites use JavaScript to load content. Google renders pages in a recent version of Chrome to see the final layout and text, similar to what a visitor sees. If key content or links only appear after complex scripts run, they can be delayed or missed. Server side rendering and plain HTML links avoid most of these problems. Read more about rendering.
Stage 2: Indexing
After a page is crawled and rendered, the search engine analyses it and decides whether to store it in the index. Indexing is not guaranteed. Google skips pages it sees as duplicate, low quality or blocked by a noindex tag.
What gets analysed
Text, headings, images and their alt text, structured data, links, language and the page's main topic.
Canonicalisation
When several URLs show the same content, Google picks one canonical URL to index and treats the rest as duplicates.
Entities and meaning
Search engines connect pages to known entities, such as businesses, people and places, to understand what a page is really about.
Stage 3: Ranking
When someone searches, ranking systems sort indexed pages to find the most helpful results. Google uses many systems and signals. It does not publish the full list, but it has explained the main groups.
| Signal group | Examples |
|---|---|
| Meaning of the query | Understanding words, synonyms, spelling and intent |
| Relevance | How well the page content matches the query |
| Quality | Expertise, trust, original value and helpful content |
| Usability | Mobile friendliness, page speed, HTTPS, no intrusive pop ups |
| Context | Location, language, device and search settings |
| Authority | Links and mentions from other trusted sites |
For local searches, ranking also weighs distance, relevance and prominence, which is why the map pack can change from one street to the next.
Serving results and SERP features
The final step is showing results. Modern search results include much more than ten blue links, and each feature has its own way of choosing content.
AI Overviews
Generated summaries with links to sources, shown for some queries. See our guide to AI Overviews.
Featured snippets and People Also Ask
Short answers pulled from pages that explain a topic clearly.
Local pack
A map with three local businesses for searches with local intent.
Shopping, images and video
Product listings, image packs and video results for visual or buying searches.
Knowledge panels
Facts about people, brands and places from the Knowledge Graph.
How AI answers use search engines
AI search tools such as AI Overviews, ChatGPT search and Perplexity generate answers by retrieving web pages and summarising them. They still depend on crawling and indexing, and they favour sources that rank well, explain topics clearly and look trustworthy. Many AI crawlers also respect robots.txt rules, so your settings affect whether your content can be used.
How Google discovers new pages
Links from other pages
Most pages are found by following links from pages Google already knows, both on your site and on other sites.
XML sitemaps
A sitemap lists the URLs you want crawled and can help new pages get found faster.
URL submissions
Search Console's URL Inspection tool lets you request indexing for an important page.
Feeds and structured data
Product feeds, news sitemaps and other feeds help Google find fresh content quickly.
What Google looks at when ranking pages
| Area | What it means | Example signals |
|---|---|---|
| Meaning of the query | Understanding what the searcher wants | Synonyms, spelling corrections, intent |
| Relevance | How well a page matches the query | Topic coverage, headings, keywords used naturally |
| Quality | How helpful and trustworthy the page is | Expertise, accuracy, originality, links from trusted sites |
| Usability | How easy the page is to use | Mobile friendliness, speed, safe browsing |
| Context | Who is searching and where | Location, language, search settings |
Google describes these areas in its own guidance on how search works. No single factor decides rankings; they work together.
How local search results work
For local searches, Google's guidance names three main factors: relevance, distance and prominence. Relevance is how well a business profile matches the search. Distance is how far the business is from the searcher or the location in the query. Prominence is how well known the business is, based on reviews, links, articles and other information across the web.
That is why a complete Google Business Profile, steady reviews and local links matter so much for map pack rankings. Our local SEO checklist covers each step.
How other search engines differ
| Search engine | Main markets | Notes |
|---|---|---|
| Most of the world | Largest share globally | |
| Bing | US, UK and other markets | Also powers some other search tools and AI features |
| Baidu | Mainland China | Needs Chinese content and fast access in China; see our Baidu SEO guide |
| Naver | South Korea | Mixes web results with its own blogs and cafes; see our Naver SEO guide |
| Yahoo Japan | Japan | Organic results use Google technology; see our Yahoo Japan SEO guide |
| YouTube | Global video search | Ranks videos on relevance and viewer satisfaction |
Common reasons search engines skip pages
- 1A noindex tag tells search engines not to index the page.
- 2Robots.txt blocks crawlers from the page or its resources.
- 3The page is a duplicate of another page, so Google indexes only one version.
- 4The content is thin or offers little unique value.
- 5The page is hard to reach, with no internal links pointing to it.
- 6Server errors or slow responses stop crawling.
If your pages are missing, follow our guide on why your website is not showing up on Google.
A simple checklist to help search engines understand your site
- 1Create a clear site structure with important pages close to the homepage.
- 2Link related pages together with descriptive anchor text.
- 3Submit an XML sitemap in Search Console.
- 4Use unique title tags and headings on every page.
- 5Add structured data for your business, products or articles.
- 6Keep pages fast and mobile friendly.
- 7Write helpful content that fully answers what people search for.
A short history of how search engines evolved
| Era | What changed |
|---|---|
| Early web directories | Sites were listed by hand in categories |
| Keyword matching | Engines ranked pages by how often words appeared, which was easy to manipulate |
| Link based ranking | Google's PageRank used links as votes of trust |
| Quality updates | Updates targeted thin content and link spam |
| Mobile first indexing | Google began using the mobile version of pages for indexing |
| Language understanding | Systems like BERT helped Google understand the meaning of queries |
| AI answers | AI Overviews and AI Mode summarise information from many sources |
Each step made shortcuts less effective and rewarded pages that genuinely help searchers.
How search engines understand language
Modern search engines do not just match words. They try to understand meaning, context and intent. A search for 'apple' could mean the fruit or the company, so Google looks at other words in the query, your location and what most people want when they search that phrase.
Google has described using language models such as BERT and MUM to understand queries and content. For website owners, the lesson is simple: write naturally, cover topics fully and use the words your customers use. There is no need to repeat a keyword many times, because search engines recognise synonyms and related terms.
How search engines handle duplicate content
- 1The crawler finds several URLs with the same or very similar content.
- 2Search engines group the duplicates together.
- 3They choose one URL as the canonical version to show in results.
- 4Your canonical tags, redirects, internal links and sitemaps all influence that choice.
- 5The other versions are usually not shown, although they are not normally penalised.
Keep one clear version of each page and point canonical tags, links and sitemaps to it. See common URL problems that hurt SEO.
How freshness and updates affect rankings
Fresh topics
For news, events and fast changing subjects, newer content often ranks better.
Evergreen topics
For stable topics, a well updated older page can outrank new pages.
Meaningful updates
Changing a date without improving content does not help. Add new facts, sections and examples.
Core updates
Google regularly updates its core ranking systems, which can move rankings across many sites.
Read more in Google algorithm updates.
How search engines use structured data
Structured data is code that labels information on a page, such as a product's price, a recipe's cooking time or a business's opening hours. Search engines use it to understand content more precisely and to show rich results like star ratings, FAQs and event dates. AI search tools can also use these clear labels when building answers.
Use Google's Rich Results Test to check your markup, and only mark up information that is visible on the page.
What this means for your website
- 1Make sure important pages are linked from your menu or other pages so crawlers can find them.
- 2Submit an XML sitemap in Google Search Console and Bing Webmaster Tools.
- 3Check the Page indexing report for pages that are not indexed and fix the reasons.
- 4Use clean, fast, mobile friendly pages that render without relying on heavy scripts.
- 5Give every page a clear topic, title and headings that match what people search.
- 6Add structured data to help search engines understand your business and content.
- 7Earn links and mentions from relevant sites to build authority.
How search engines fight spam
Automated spam detection
Google uses systems such as SpamBrain to detect spam pages and link schemes.
Spam policies
Google publishes policies against cloaking, hidden text, scaled low value content, link schemes and site reputation abuse.
Manual actions
Human reviewers can apply penalties, which appear in Search Console's Manual actions report.
Link devaluation
Many spam links are simply ignored rather than counted.
Following white hat practices keeps your site safe. See SEO tips for beginners for safe basics.
How search engines treat new websites
A brand new website has no history, links or reviews, so search engines know little about it. Pages may be crawled and indexed within days, but ranking for competitive terms usually takes months while trust builds. New sites do best by targeting specific long tail searches first, publishing genuinely helpful pages and earning a few quality links from real partners and directories.
Submitting a sitemap, verifying Search Console and linking new pages from existing ones all help search engines discover and understand a new site faster.
How mobile devices changed search
- 1Google now indexes the mobile version of pages first.
- 2Fast loading on mobile networks matters for user experience and Core Web Vitals.
- 3Local searches on phones often show maps and call buttons first.
- 4Voice searches on phones tend to be longer and more conversational.
- 5Pages that hide content on mobile risk losing visibility.
Read mobile first indexing explained for details.
How search engines measure page experience
| Signal | What it measures |
|---|---|
| Largest Contentful Paint | How quickly the main content loads |
| Interaction to Next Paint | How quickly the page responds to taps and clicks |
| Cumulative Layout Shift | How much the layout jumps while loading |
| HTTPS | Whether the page is served securely |
| Mobile friendliness | Whether the page works well on phones |
Google says page experience is one of many signals; great content still matters most. See how to improve page speed.
Search engine terms explained
| Term | Meaning |
|---|---|
| Crawler or bot | Software that visits pages and follows links |
| Index | The database of pages a search engine has stored |
| Algorithm | The set of systems that decide rankings |
| SERP | The search engine results page |
| Canonical | The preferred version of a page among duplicates |
| Crawl budget | How many pages a search engine will crawl on a site in a given time |
| Rich result | A search result with extra details, such as stars or prices |
Browse more terms in our SEO glossary.
How to use this knowledge to rank better
- 1Make every important page reachable through internal links.
- 2Keep pages fast, secure and mobile friendly.
- 3Write content that fully answers the search, in natural language.
- 4Use clear titles, headings and structured data.
- 5Earn links and mentions from trusted, relevant sites.
- 6Keep business information consistent for local search.
- 7Review Search Console regularly for crawl and indexing issues.
Want help putting this into practice? Our technical SEO service fixes crawling and indexing issues.
What search engines cannot do well
Read text inside images reliably
Important words should be in real text, not only in images or banners.
See content behind logins
Pages that need a login or form submission are usually not indexed.
Guess your location without signals
Show addresses, service areas and local details clearly.
How to check if Google has found your page
- 1Search site:yourdomain.com/page-url in Google.
- 2Open the URL Inspection tool in Search Console and enter the page address.
- 3Check whether it says "URL is on Google" and look at the crawl and indexing details.
- 4If it is not indexed, read the reason and fix it.
- 5Click Request indexing for important new or updated pages.
If your whole site seems missing, our guide on why a website is not showing up on Google walks through the common causes.