AutoPSEO-Bot

Identifying it

Requests carry the user agent AutoPSEO-Bot/1.0 (On-Page SEO Checker), which the on-page checker extends with a link back to this page. The crawler runs on Cloudflare Workers, so source addresses come from Cloudflare ranges and change without notice: match on the user agent string, not on an IP allowlist.

What it requests

One GET per URL, and nothing else. It does not execute JavaScript, so it sees exactly the HTML your server returns.

• GET only, never POST or any other method
• Accepts text/html, follows up to 5 redirects, and gives up after 8 seconds
• At most 5 requests in flight at once
• Stops reading after roughly 1 MB of HTML
• Public http and https hosts only, so private and loopback addresses are refused

When it runs

Only while a check is running: it fetches robots.txt, any sitemaps it needs, and the pages under test. Results are cached for 24 hours, so re-opening a report costs your server nothing. A full site crawl is bounded at 100 pages by default with a hard cap of 500, a depth of 3, and one request per page.

Allowing it

If robots.txt allows all agents there is nothing to do. If you block unknown agents by default, add an explicit block for AutoPSEO-Bot with the paths you want checked; a staging site behind HTTP auth simply returns 401 and the report says so rather than guessing.

Blocking it

A User-agent block naming AutoPSEO-Bot with Disallow: / stops the crawler on the next run, and Allow rules within that block are honoured as usual. Blocked URLs are reported in the run as blocked by robots.txt rather than silently dropped, so a check that suddenly returns nothing is easy to explain.