How to diagnose technical indexing issues

A triage order for the Page indexing report in Search Console: what to fix first, how to confirm causes with URL Inspection, and when to stop.

Diagnose indexing problems from the pages you need, not from the report's totals. Build a list of the URLs that matter, find out which of them are not indexed and why, then work through the causes in order of severity: server and access errors first, then accidental blocks, canonical conflicts, error-like pages, and last the pages Google simply chose not to index. Confirm each fix with a live test in URL Inspection before you ask Google to validate it.

This playbook assumes you know the statuses described in Indexing explained.

Step 1: build the list of URLs that matter

The Page indexing report counts every URL Google knows about, including parameters, old addresses and pages you never meant to publish. A large "Not indexed" number is normal, and Google says plainly that not every URL should be indexed (Page indexing report). So start from your own list:

Pages that earn traffic: the top 50 to 100 in Performance → Pages over three months.
Pages that earn money: product, pricing, service and location pages, whatever their traffic.
New pages: anything published in the last 90 days.
Your sitemaps: they should list exactly the canonical URLs you want indexed. You can filter the Page indexing report by sitemap to see how many of its URLs are indexed, and each sitemap in the Sitemaps report links to the indexing coverage of its URLs.

Step 2: find which of them are not indexed

For a short list, inspect each URL in URL Inspection. For a longer one, open the Page indexing report filtered to your sitemap and go through each reason, checking which of your important URLs appear in its examples. Record each URL with its reason and the reason's source. Google's rule of thumb: in general you can only fix issues whose source is listed as "Website".

Step 3: work through the causes in order

Fix causes in this order. Each level can hide the next, and the first ones affect the most pages at once.

Order	Status	Check first	Typical fix
1	Server error (5xx)	Server logs and hosting status at the times Google crawled	Fix capacity or crashes; repeated 5xx responses also make Google crawl more slowly
2	Redirect error	The full redirect chain for the URL	Redirect in one step to the final URL; remove loops
3	Blocked due to unauthorized request (401), access forbidden (403) or other 4xx	Login walls, firewall or bot-protection rules	Let Googlebot reach public pages
4	URL marked 'noindex'	The page's robots meta tag and its `X-Robots-Tag` header	Remove noindex from pages that should rank
5	URL blocked by robots.txt	The robots.txt rule that matches	Remove the rule for pages that should be crawled
6	Duplicate, Google chose different canonical than user	Which URL your canonical tag, redirects, sitemap and internal links each name	Make every signal point to the same URL
7	Soft 404	What the page shows when Googlebot fetches it	Real 404 for missing pages; real content for real pages
8	Crawled - currently not indexed	Whether the page adds something your other pages and competitors do not	Improve it, or merge it into a stronger page
9	Discovered - currently not indexed	Whether any crawled page links to it	Link to it from crawled pages; on very large sites, manage crawling

Diagnosing one URL with URL Inspection

For each important URL in your list, run the same five checks (URL Inspection tool):

Read the verdict and the reason. "URL is unknown to Google" means Google has not seen the address at all, so the job is discovery: link to it from pages Google already crawls and list it in your sitemap.
Compare the canonicals. If the Google-selected canonical differs from the user-declared one, Google is indexing another URL in place of this one. Look at that other URL before changing anything.
Check the crawl. Note the last crawl date, whether Google crawled as a smartphone or a desktop, whether crawling was allowed, and whether the page fetch succeeded. One caution: when robots.txt blocks crawling, "Indexing allowed?" always says yes, because Google cannot see a noindex on a page it is not allowed to fetch.
Run Test live URL. The default view describes the last indexed version, which may be old. The live test fetches the page now and shows a screenshot of the rendered page, which is the quickest way to spot content that only appears for logged-in users or after a script fails.
Request indexing only after a fix. Google keeps a daily quota and says repeated requests for the same URL do not speed anything up (Ask Google to recrawl).

Notes on the harder statuses

Blocked by robots.txt is not the same as kept out. robots.txt controls crawling, not indexing; a blocked URL that other pages link to can still be indexed without its content. To keep a page out of Google, allow crawling and use noindex, in a meta tag or an X-Robots-Tag header (Block indexing). Remember that removal happens only when Google recrawls the page, which can take a while.

Canonical conflicts are usually self-inflicted. Google weighs redirects and rel="canonical" as strong signals, sitemap inclusion as a weak one, and prefers HTTPS (Consolidate duplicate URLs). The typical conflict is a canonical tag naming one URL while the navigation, the sitemap and internal links all use a variant (trailing slash, upper case, a tracking parameter, http://). Make them agree and the conflict usually resolves at the next crawl.

Soft 404s come in two kinds. Some are genuinely missing pages, such as an "item not found" template that returns 200; those should return 404 or 410. Others are real pages that look empty to Google, such as a category with no products or a page whose content loads after a script fails; those need content Google can see, which the live test's screenshot will confirm.

Crawled - currently not indexed is a judgement, not an error. Google crawled the page and decided, for now, not to keep it; it may or may not be indexed later. Ask honestly what the page adds. Thin variations of other pages, tag pages with two posts, and articles that repeat what ten other sites say are the usual residents. Make the page clearly better or different, or merge it into a page that already is.

Discovered - currently not indexed is usually about links. Google knows the URL but has not crawled it yet. On most sites the fix is to link to the page from pages Google already crawls and to keep it in the sitemap. Google's crawl budget guide is written for a narrower group: sites with over a million pages that change weekly, or over 10,000 pages that change daily (Managing crawl budget). If that is you, cut the URL inventory: consolidate duplicates, block crawling of URLs that should never be fetched, return 404 or 410 for removed pages, and fix soft 404s, which keep being crawled.

Step 4: validate the fix

When a reason's URLs are fixed, open the reason in the Page indexing report and press Validate fix, once. Google first checks a few sample pages; if the problem is still there, validation stops. Otherwise it rechecks every known affected URL and marks each as passing or failing. Google says validation typically takes up to about two weeks and sometimes much longer. It can also notice fixes during normal crawling without a validation request.

When to stop

Stop chasing a status when the URLs behind it are ones you do not need indexed (parameters, filtered views, internal search results, old redirects) or when a page has been improved, linked and resubmitted and Google still declines it after a couple of months. At that point the useful move is usually to merge the page into a stronger one and redirect it, not to request indexing again.

How AutoPSEO helps

AutoPSEO turns step 2 into a standing check. The Index Tracker holds up to 1,800 URLs of your choice, re-inspects up to 200 of them every day through the URL Inspection API (oldest check first, within Google's allowance of 2,000 inspections per property per day), and marks pages deindexed in the last 30 days and pages Google has not crawled for 120 days. An Index flip alert posts to Slack, Microsoft Teams or a webhook when a tracked URL drops out of the index or comes back (alerts). URL Inspection and Sitemaps work from inside the app, and Bing crawl issues show the same kind of problems from Bing's side.

For new pages, IndexNow tells Bing and the other engines that support it about changed URLs; Google does not take part in IndexNow, so sitemaps and internal links remain the way to reach Google. Articles from the content engine are delivered to your own site, which publishes and links them like any other page; when your site declares an IndexNow key, AutoPSEO pings IndexNow with each published or updated address on your host and records what IndexNow answered (Delivery).