Indexing explained: how Google crawls, indexes and reports pages

How a URL travels from discovery to Google's index, what each Page indexing status means, and which ones need fixing, based on Google's documentation.

Indexing is Google storing a page so that it can appear in search results. Before that can happen, Google has to discover the URL, crawl it and usually render it. Afterwards it decides whether the page duplicates another URL and whether it is worth keeping at all. The Page indexing report in Search Console shows where each of your URLs stopped along that path, and Google's reason.

From URL to search result

Google's description of how Search works breaks the journey into steps you can check one at a time.

Discovery. There is no central list of web pages, so Google keeps looking for new ones. It finds most of them through links from pages it already knows and through sitemaps that site owners submit.
Crawling. Googlebot fetches the URL, provided robots.txt allows it. Google limits how fast it crawls so it does not overload your server, and server errors make it slow down further.
Rendering. Google runs the page's JavaScript with a recent version of Chrome, because some content only exists after scripts run.
Indexing. Google analyses the content, groups pages that are duplicates or near-duplicates, chooses one of them as the canonical, and stores it. It can decide not to index a page at all, for example because the content is low quality or a robots meta rule forbids it.
Serving. For each search, Google picks results from the index. Being indexed makes a page eligible to appear; it does not promise that it will.

Google is explicit that it does not guarantee any of these steps for any page. What you can do is remove the obstacles and give it good reasons to keep going.

Reading the Page indexing report

The report splits the URLs Google knows about into Indexed and Not indexed, and lists a reason for every group of non-indexed URLs (Page indexing report). Two pieces of Google's own guidance should shape how you read it:

Not indexed is not necessarily bad. Google tells site owners not to expect every URL to be indexed. Duplicates, redirected addresses and pages without meaningful content are supposed to stay out. What matters is that the key pages are in.
Look at the Source. Each reason shows whether the cause lies with the website or with Google's systems. In general, you can only fix the reasons whose source is "Website".

So the useful question is never "why is my not-indexed number high?". It is "are any of the pages I need on this list, and why?".

What each status means

Google's reasons fall into five groups. The right response depends far more on the group than on the exact wording.

You told Google not to index it

Status	What happened	Action
URL marked 'noindex'	The page carries a noindex rule	None if intended; remove the rule if the page should rank
URL blocked by robots.txt	robots.txt stops Google from crawling it	Blocking crawling does not guarantee the page stays out of the index
Alternate page with proper canonical tag	The page points to another URL as its canonical, and Google agrees	None
Page with redirect	The URL redirects elsewhere, so it is not indexed itself	None, as long as the target is the page you want indexed

Google could not fetch it

Status	What happened	Action
Server error (5xx)	Your server returned a 500-level error	Fix the server; repeated errors also slow crawling
Redirect error	A redirect chain was too long, looped, or led to a bad or empty URL	Point the redirect straight at the final URL
Blocked due to unauthorized request (401), access forbidden (403) or other 4xx issue	The page refused Googlebot	Remove the barrier if the page should be public
Not found (404)	The URL returned 404	Fine for removed pages; fix the links if the page still exists elsewhere

Google chose another URL

Status	What happened	Action
Duplicate without user-selected canonical	Google sees the page as a duplicate and picked a different canonical	Usually none; Google calls this working as intended
Duplicate, Google chose different canonical than user	You named a canonical, but Google preferred another URL	Make your signals agree (see below)

Google has not decided in your favour yet

Status	What happened	Action
Discovered - currently not indexed	Google knows the URL but has not crawled it yet, often rescheduled to avoid overloading the site	Link to it from crawled pages; on very large sites, manage crawling
Crawled - currently not indexed	Google crawled the page but has not indexed it, and it may or may not in future	Make the page clearly useful and distinct, or merge it into a stronger page

The content looks wrong

Status	What happened	Action
Soft 404	The page answers 200 but looks like an error or empty page	Return a real 404 for missing pages, or give real pages real content
Page indexed without content (warning)	Indexed, but Google could not read the content	Check what Googlebot receives with URL Inspection
Indexed, though blocked by robots.txt (warning)	Indexed even though robots.txt blocks it	To keep it out, allow crawling and add noindex

The last row catches many people out. Google can only see a noindex rule on a page it is allowed to crawl, so blocking a page in robots.txt and adding noindex at the same time means the noindex is never read (Block indexing with noindex).

When Google picks a different canonical

Canonical tags are signals, not commands. Google weighs several of them: redirects and rel="canonical" are strong signals, inclusion in a sitemap is a weak one, and it prefers HTTPS addresses over HTTP (Consolidate duplicate URLs). When your signals disagree, for instance a canonical tag pointing to one URL while your internal links and sitemap use another, Google picks for itself. Make every signal name the same URL, and link internally to that URL. Do not use robots.txt or noindex to choose a canonical; noindex removes the page from Search entirely.

Checking one URL with URL Inspection

The Page indexing report shows groups; URL Inspection shows one URL in detail (URL Inspection tool). It reports whether the URL is indexed, the canonical you declared and the one Google selected, when Google last crawled the page and whether it crawled as a mobile or desktop user agent, whether robots.txt allows crawling, whether the fetch worked, and whether indexing is allowed.

Two cautions. First, the default view describes the last indexed version, which may be out of date; Test live URL fetches the page now, which is how you confirm a fix, though a passing live test does not guarantee indexing. Second, when robots.txt blocks crawling, "Indexing allowed?" always shows yes, because Google cannot see a noindex rule it was not allowed to fetch.

Request indexing asks Google to crawl the URL again. It is subject to a daily quota, and Google says that requesting a crawl several times for the same URL will not make it happen faster (Ask Google to recrawl). For many URLs, submit an up-to-date sitemap instead.

How long does indexing take?

Google gives ranges rather than promises. Indexing a new page can take a few days and is never instant, even after a direct request. New sites can take about a week or so to be crawled and indexed, and a recrawl can take anywhere from a few days to a few weeks. When you fix an issue and press Validate fix, the check typically takes up to about two weeks. If an important page is still missing after a month, work through the technical indexing playbook rather than requesting indexing again.

Helping Google find and keep your pages

Link new pages from pages Google already crawls, such as your home page, hub pages or related articles. Links are the main way Google discovers URLs.
Keep sitemaps to canonical URLs, written as full absolute addresses, with at most 50,000 URLs or 50MB uncompressed per file. Google uses <lastmod> when it is consistently accurate and ignores <priority> and <changefreq> (Build a sitemap).
Give each piece of content one URL, with redirects, canonical tags, sitemaps and internal links all agreeing.
Return a real 404 or 410 for pages you removed, rather than an empty page with a 200 status.
Worry about crawl budget only at scale. Google's crawl budget guide is aimed at sites with over a million pages that change weekly, or over 10,000 pages that change daily.

How AutoPSEO helps

AutoPSEO runs URL Inspection from inside the app and counts against Google's daily allowance of 2,000 inspections per property. The Index Tracker keeps a list of up to 1,800 URLs you choose and re-inspects up to 200 of them a day, the longest-unchecked first. It warns when one was deindexed in the last 30 days or has not been crawled for 120 days, and can raise an "Index flip" alert in Slack, Teams or a webhook when a tracked URL drops out of the index or comes back. IndexNow notifies Bing and the other engines that support it; Google does not take part in IndexNow, so for Google the sitemap and internal links remain the way in.

Articles written by the content engine are delivered to your own site, which publishes and links them like any other page. When your site declares an IndexNow key, AutoPSEO pings IndexNow with each new or updated address on your host (Delivery).