robots.txt tester: is this URL allowed for Googlebot or GPTBot?

How Google decides whether a URL is blocked

Google's rules are now standardized in RFC 9309 (the Robots Exclusion Protocol, published in 2022) plus a few documented Google behaviors. The tester implements them in three steps:

1. 1.Group selection. The file is split into groups — one or more User-agent lines followed by rules. The crawler picks the group whose user-agent token matches it most specifically; groups naming the same token are merged. Google's sub-crawlers fall back to their parent, so Googlebot-Image uses a Googlebot group if there's no Googlebot-Image group. If nothing matches, the * group applies; if there's no * group either, everything is allowed.
2. 2.Rule matching. Each Allow and Disallow pattern is compared with the URL's path and query string from the start. * matches any sequence of characters, $ at the end means "the URL ends here", and characters are compared after percent-encoding normalization.
3. 3.Precedence. The matching rule with the longest pattern wins. On a tie between Allow and Disallow, Allow wins. No matching rule means allowed. /robots.txt itself is always allowed.

Worked examples

The most surprising rule: groups don't inherit from *

If your file has a User-agent: * group that disallows /admin and a separate User-agent: Googlebot group that disallows /staging, Googlebot may crawl /admin. It only reads its own group. This catches many sites that add a Googlebot or GPTBot section and accidentally open up everything they had blocked for everyone else. Copy the shared rules into each named group.

How to test your robots.txt

1. 1.Open https://yourdomain.com/robots.txt in a browser and paste the whole file into the tester. Each hostname (www, a subdomain, http vs https) has its own robots.txt.
2. 2.Enter a URL or path. Include the query string if your rules use ? or *, because patterns match against path + query.
3. 3.Enter a crawler. Use a token like Googlebot, Bingbot, GPTBot or ClaudeBot, or paste a full user-agent string from your logs — the tester finds the group whose token appears in it.
4. 4.Read the verdict, the group it obeyed and the deciding rule (shaded in the parsed file). If the result isn't what you intended, add a longer, more specific rule rather than reordering lines — order doesn't matter to Google.
5. 5.Fix the syntax warnings: rules before any User-agent line, paths not starting with /, and directives Google ignores such as crawl-delay and noindex.

Common robots.txt mistakes

• Blocking a page to remove it from Google. Disallow stops crawling, so Google can't see a noindex tag on the page and may keep the URL indexed from links. Allow crawling and use noindex instead.
• Blocking CSS and JavaScript. Google renders pages; blocking /assets/ or /_next/ can make it see a broken page.
• Relying on order. Google ignores the order of rules within a group — only length and type matter. (Some other crawlers use first-match, which is why specific rules are safest.)
• Using noindex or crawl-delay in robots.txt for Google. Google dropped noindex support in 2019 and has never honored crawl-delay; Bing and some others do honor crawl-delay.
• Forgetting trailing slashes. Disallow: /blog also blocks /blog-post-ideas and /blogroll.
• Assuming every bot complies. Well-behaved crawlers do; scrapers don't. robots.txt is a public request, not access control.

Testing rules for AI crawlers

The same semantics apply to GPTBot, ClaudeBot, PerplexityBot and other AI crawlers that honor robots.txt (RFC 9309 is the standard they cite). Build AI-specific groups with the AI crawler robots.txt generator, then paste the result here to confirm each bot gets the access you intended. To see which crawlers really visit and what they fetch, use server-side AI crawler tracking — robots.txt only states your intent.