Programmatic SEO indexing: how to get thousands of template pages crawled, indexed and kept there

You publish a few thousand pages, and weeks later Search Console shows a fraction of them indexed. That is unsettling, but part of it is expected. This section sets out what Google says you should expect and which parts of the result are in your hands; the rest of the article works through each one.

Google's help for the Page indexing report says not to expect every URL on your site to be indexed, only the canonical pages, and that it is fine for a URL to stay out for the right reasons, such as a noindex tag, a duplicate or a 404 for a removed page. For a programmatic set, three things decide how much gets in:

• Discovery: whether Google finds each URL, through sitemaps and links.
• Distinction: whether each URL is a different page, or a duplicate Google folds into another.
• Worth: whether the page holds enough to keep. Discovery problems usually show up as URLs not yet crawled; worth problems as pages crawled and then left out.

The guide to programmatic SEO covers building pages that pass the third test; this article covers the first two and how to read the results.

No. Publish in batches, starting with the rows that have the best data and the clearest demand, and give each batch its own sitemap. The Page indexing report has a sitemap filter that shows only the URLs listed in one specific sitemap, so a sitemap per batch tells you how each batch is doing.

Each batch should go live with its hub: a page that lists and links every page in the batch, linked in turn from your navigation or a parent hub, so no page depends on the sitemap alone. Each page should also link to a few neighbours, so a crawler that lands on one finds the rest. Then give the batch a few weeks, look at what share is indexed and which statuses the rest carry, and fix the template before you publish more.

Say you have 4,000 city pages ready. Publishing the 300 with the most data first, in their own sitemap, shows within weeks whether the template holds up; if most of the 300 sit in Crawled, currently not indexed, the other 3,700 would only add to the pile. Server speed matters as well: Google's crawl budget guide says that if it can load and render your pages faster, it might be able to read more content from your site.

Five rules cover most of what goes wrong with sitemaps for a large set:

• Split by limit and by type. A single sitemap holds at most 50,000 URLs or 50 MB uncompressed. Larger sets go in several files listed in a sitemap index, which can hold up to 50,000 sitemap entries, and Search Console accepts up to 500 sitemap index files per site. Split by page type or batch rather than alphabetically, so the per-sitemap filter means something.
• Keep referenced files in place. Sitemaps listed in an index must sit in the same directory as the index or lower in the site.
• List only canonical, absolute URLs. Google asks for fully-qualified URLs and generally shows canonical URLs in results, so leave out redirects, duplicates and noindexed pages.
• Keep lastmod honest. Google uses it only if it is consistently and verifiably accurate, and counts changes to main content, structured data or links, not a new copyright date. It ignores priority and changefreq altogether.
• Tell Google where they are, in the Sitemaps report or with a Sitemap: line in robots.txt. Google treats a sitemap as a hint, not a promise to crawl it.

Google's crawl budget guide is written for very large sites and for sites with a large share of URLs in Discovered, currently not indexed, which a fast-growing set can become at any size. It asks you to eliminate duplicate content to focus crawling on unique content rather than unique URLs. In a programmatic set, duplicates come from the template more often than from the writing: the same pair in two orders, sort and filter parameters, a trailing slash and its absence, two spellings of one city. Pick one address for each page, send the others to it with a 301 redirect or a canonical tag, and build internal links and sitemap entries from that address only. When Google disagrees with your choice, the Page indexing report says so under Duplicate, Google chose different canonical than user; Duplicate without user-selected canonical means you named none.

Empty rows are the other leak. A page that says nothing was found while returning a 200 status is what the report calls a soft 404, and Google recommends a real 404 for pages that truly don't exist. The crawl budget guide adds that soft 404 pages continue to be crawled and waste your budget, so a template that renders an empty shell for a missing row costs crawling as well as quality. It also warns that long redirect chains have a negative effect on crawling, so when you merge pages, point every old address straight at the final one.

They do different jobs, and putting both on one URL stops the noindex from working:

• robots.txt stops crawling. Use it for URL spaces Google should never fetch, such as filter and sort combinations. The crawl budget guide prefers it to noindex for this, because Google still requests noindexed pages and then drops them. It does not keep a page out of results: the Page indexing report notes a very small chance that a blocked page is still indexed.
• noindex stops indexing. Use it for pages people need but search should not show, such as a sparse row you keep for signed-in users. Google has to crawl the page to see the rule; if robots.txt blocks it, the noindex is never read and the page can still appear in search results.
• A 404 or 410 ends it. For rows you have dropped, return one of these and take the URL out of sitemaps and hubs. Google lists a 404 for a removed page with no replacement as a good reason for a URL not to be indexed.

noindex is also slow to act on pages Google rarely visits: its documentation says it may take months for Googlebot to revisit a page, depending on the page's importance.

Filter the report by each batch's sitemap and read the statuses as signals about the template rather than about single pages:

• Discovered, currently not indexed in bulk usually means Google has found more URLs than it has chosen to crawl so far; the guide to Discovered, currently not indexed covers the causes.
• Crawled, currently not indexed in bulk usually points at the pages themselves: thin rows or pages too much alike. Google says there is no need to resubmit these URLs, and the guide to Crawled, currently not indexed explains what to change instead.
• Duplicate statuses point at canonical handling, Soft 404 at empty rows, and the noindex and robots.txt statuses should match only what you meant to exclude.

The report's list of example URLs stops at 1,000, so for a large set look for patterns rather than checking every row. Use URL Inspection on a sample: it shows how Google sees one page, but each property has daily limits on inspections and on indexing requests, a request does not guarantee indexing, and the live test does not check whether the page is a duplicate. For many URLs, Google's own advice is to submit a sitemap.

IndexNow lets you tell search engines directly that a URL was added, updated or deleted, instead of waiting for them to notice. The engines listed on indexnow.org include Bing, Yandex, Naver, Seznam.cz and Yep, and a URL submitted to one is shared with the others. Google is not on that list, so for Google your sitemaps and links remain the route.

• Up to 10,000 URLs go in one POST request, all from one host.
• You prove ownership with a key file, by default at the root of the site; a key file elsewhere only covers URLs under its own folder.
• A 200 response means the engine received the URLs, nothing more: each engine still decides what to crawl.

For a programmatic set, submit each batch when it goes live, and again when rows change or pages are removed, since IndexNow covers deletions too. AutoPSEO's content engine pings IndexNow for the URLs it delivers to your site; the delivery documentation explains how your site declares its key.