Documentation
Turning on the content engine: the site scan and proposal
Turn the content engine on and it scans your sitemaps, Search Console data and pages, then proposes languages, content types, a format and topic rules.
Turning the content engine on for a property starts a full scan of your site. The scan reads robots.txt, every sitemap, three months of Search Console data and the pages themselves, then proposes how your articles should look: languages, content types, an article format and topic rules. No topic is picked before the first scan finishes, and nothing in the proposal applies until you tick it.
Before you turn it on
- Search Console is connected for the property (Connecting Search Console). Without it the scan still reads sitemaps and pages, but traffic per page is missing and no topics come from Search Console.
- You have full access to the property. Viewers can look at the Content page but cannot switch the engine on or change it.
- The account has an active plan or trial (pricing). When a plan lapses, published articles stay where they are and nothing new is started.
- AutoPSEO-Bot can read the site (see below).
- Property Context is filled in, so the writer knows what the business does and never has to guess. The Ready to write checklist reminds you.
Turn the engine on
- In the property's sidebar, open Content (in the Research group).
- Choose Turn on the content engine.
- AutoPSEO saves the default settings and opens the Your site page with "The engine is on. Your site is being scanned now."
| Setting on the Content page | Default |
|---|---|
| New articles per week | 2 |
| Approve every article before it goes live | On |
| Who writes the topics in the queue | AutoPSEO, at the pace above |
| Language articles are written in | English |
| Where articles are published | AutoPSEO hosting at <name>.autopseo.com, until you connect your own site |
Change any of them under Settings on the Content page. To send articles to your own site instead of AutoPSEO's address, set up the delivery webhook.
What the scan reads
The Scan panel shows each phase while it runs:
| Phase | What happens |
|---|---|
| Reading the sitemaps | robots.txt first, then every sitemap it lists plus /sitemap.xml, following sitemap indexes and reading gzip files, on your own host only |
| Reading Search Console | Three months of searches per page |
| Taking over your site's articles | Only when your site's webhook can hand its articles over (delivery) |
| Reading the pages | Each page fetched and parsed, the most visited first |
| Working out the proposal | Up to 60 articles per section, 400 in all, plus the home page, measured |
| Planning topics | Search Console candidates go into the backlog (topics) |
| Learning your site's look | Only while AutoPSEO hosts the articles (hosting) |
| Limit | Value |
|---|---|
| Sitemap files per scan | 400 |
| Addresses kept per scan | 60,000 |
| Entries per sitemap file | 50,000, and 50 MB |
| Pages read in a full scan | 5,000 |
| Pages read in a weekly refresh | 1,000, the ones never read and the ones whose sitemap lastmod moved |
| Pages read at the same time | 6 |
| Per page | 15 seconds and the first 2 MB |
| Search Console rows | 25,000 over three months |
A page that exists in several languages counts once. A first path segment such as /de/ is read as a language when your sitemap's hreflang annotations say so, or when at least three addresses use it and at least 30% of them also exist without the prefix or under another language code.
The scan itself spends no research credits. Only the steps for typed pages look up search volumes, and those draw on research credits.
Let AutoPSEO-Bot read your site
The scanner's user agent starts with AutoPSEO-Bot/1.0.
- robots.txt rules for
User-agent: AutoPSEO-Botapply, falling back to the*group. Pages it disallows are not read. A site without robots.txt is read in full. - Only your own host is read. A redirect that only adds a trailing slash or switches to https is followed; any other redirect is recorded and not followed.
- 404 and 410 mark a page gone. 401, 403, 429 and 503 mark it blocked, and 429 and 503 get one more try after three seconds. Bot-check pages ("Just a moment" and similar) also count as blocked.
If your firewall or bot protection challenges unknown crawlers, allow AutoPSEO-Bot. Google's robots.txt documentation explains the group syntax if you want rules for this crawler alone.
The scan reports what got in its way:
| Note | Meaning |
|---|---|
| The site has no robots.txt. | Everything was allowed |
| robots.txt blocks AutoPSEO-Bot; pages it disallows are not read. | Your rules shut the crawler out of some pages |
| No sitemap was found; pages came from Search Console only. | Add a sitemap so new pages are found (sitemaps) |
| The sitemaps list more addresses than one scan keeps; the first ones were kept. | Over 60,000 addresses |
| The site has more pages than one scan reads; the most visited were read first and the next scans read more. | Over the per-scan page limit |
| Most pages refused AutoPSEO-Bot. | More than half were blocked: check your firewall |
| More than a fifth of the pages could not be read. | Failures or blocks above 20% |
| Fewer than three pages read as articles, so the format was not proposed from them. | Set the format by hand |
The proposal
What the scan proposes lists four parts, each with a checkbox: Languages, Content types (marked New or Changes), Article format (shared by every content type) and Topic rules. Untick what does not fit and choose Apply what is ticked. Applied parts leave the list; when nothing differs from your settings it says "Everything the scan found matches the settings already."
If your site's webhook exports a profile, what it declares wins over what the scan detected (delivery).
Languages
The primary language is the one your unprefixed addresses use (from hreflang), else the home page's lang, else the language with the most pages. Another language is proposed when it has at least three pages and at least a fifth as many as the primary. The engine writes 27 languages; others are listed as not written. Applying replaces the language list. See translation.
Content types
Sections of the site with at least five pages, up to eight, each with the kinds of search its slugs show: questions, how-tos, definitions, comparisons, alternatives and best-of lists. A kind counts when at least a quarter of the section's slugs show it. Applying creates the new types and updates the rules and counts of existing ones; a type's path never changes. Later, Find the sections your site already has on the Content page offers sections as types too.
Article format
Measured on at least three pages read as articles:
| Field | Proposed from |
|---|---|
| Title suffix | An ending at least 60% of titles share |
| Title width | The 90th percentile of title widths, rounded up to 5, between 50 and 70 |
| Sections | The 10th to 90th percentile of section counts |
| FAQ, key takeaways | The same range, when at least 30% of articles have them; otherwise none |
| Direct answer | The 90th percentile of its length in words, when at least half the articles open with one |
| Meta description | The 10th to 90th percentile of lengths, from at least three |
| Sources | The range, when at least 40% of articles cite sources |
| Question field | On when at least 60% of H1s are questions |
| Brand name | Your site's name, when at least 60% of articles mention it |
| Categories | Two or more shared categories covering half the articles, when one section holds 80% of them |
| Link whitelist | Non-article pages at least 20% of articles link to, up to 30 |
Applying merges these values into your format. The fields themselves are explained in writing and review.
Topic rules
Brand terms (your domain's name and your site's name) and noise words (words in at least 40% of your titles and 30% of your top 200 searches, up to 10). Applying only adds to your rules, never removes. See topics.
After anything is applied, every topic waiting in the backlog is checked again against the new languages, types and rules.
Ready to write
The Ready to write checklist on the Your site page shows what the engine still needs:
- The site was scanned
- Content types match the site's sections
- The article format is set
- The site's languages are set
- Brand terms are set, so searches for the brand never become topics
- The property context says what the business does
- Articles have somewhere to go: AutoPSEO hosting or your site's webhook
- A writer is ready: AutoPSEO, or your own AI connected through MCP
- Topics are waiting, in the queue or the backlog
- The engine is on (articles per week above zero)
Later scans
The daily run at 03:17 UTC starts a refresh scan for properties whose last finished scan is more than seven days old. On the Your site page you can also choose Look for changes (a refresh) or Scan the whole site again (a full scan), and Stop a running one; what it found so far is kept. Only a full scan that read every sitemap marks pages that disappeared as gone. A scan still running after six hours is marked failed.
Pages on your site lists every page the scan found, the most visited first, and Earlier scans keeps the history. Next: where topics come from.