robots.txt rules
RankNova fetches your public robots.txt and evaluates the applicable group and directive for each supported crawler, using longest-match precedence between Allow and Disallow.
Free AI crawler access test
AI Crawler Checker – Test AI Bot Access
Check whether AI search, retrieval and training crawlers can access your website.
RankNova analyzes robots.txt rules and supported page-level signals to show which AI and search crawlers are allowed, blocked or restricted — and why.
robots.txt rulesPer-path testingMatched rule shownSearch + AI crawlers
Supported checks
What Does RankNova's AI Crawler Checker Check?
Every item below is read from your live website. Nothing here is inferred or assumed.
Specific URL and path access
Crawler permission is path-specific. You can test any path on the same host, because a crawler allowed on your homepage may be blocked on /pricing or /docs.
HTTP response
The homepage and each tested path are fetched so you can see the returned status. A robots.txt permission does not guarantee a successful retrieval.
Indexing directives
Fetched HTML is checked for a noindex directive in a meta robots tag or an X-Robots-Tag response header. These are indexing signals and are separate from robots.txt crawl rules.
Sitemap discovery
RankNova reports whether a usable sitemap was found from robots.txt declarations or the conventional sitemap location.
RankNova reads the raw HTML response and does not execute JavaScript, so it cannot tell you what a JavaScript-rendering agent would ultimately see.
Purpose matters
Not All AI Crawlers Do the Same Job
AI-related crawlers serve different purposes. Some support search or user-requested retrieval, while others are associated with model training, model improvement or other AI processing. A robots.txt policy should therefore be based on each crawler's documented purpose rather than treating every "AI bot" identically.
AI Search & Retrieval
Agents an operator documents as surfacing or linking pages inside an AI search experience.
OAI-SearchBot
OpenAI · ChatGPT search results
OpenAI states this agent is used to surface websites in search results in ChatGPT's search features.
Claude-SearchBot
Anthropic · Claude search quality
Anthropic states this agent navigates the web to improve search result quality for users.
PerplexityBot
Perplexity · Perplexity search index
Perplexity states this agent surfaces and links websites in search results, and is not used to crawl content for AI foundation models.
User-Initiated AI Fetch
Agents that fetch a page because a person asked a question, rather than crawling routinely.
ChatGPT-User
OpenAI · User-initiated fetch
OpenAI states this agent is used for certain user actions in ChatGPT and Custom GPTs, rather than for routine crawling.
Claude-User
Anthropic · User-initiated fetch
Anthropic states this agent supports Claude users, accessing websites when an individual asks Claude a question.
Perplexity-User
Perplexity · User-initiated fetch
Perplexity states this agent supports user actions, visiting a page to help answer a question, and is not used for crawling or training.
AI Training & Data Use
Agents and robots.txt tokens an operator documents as relating to model training or other generative AI use.
GPTBot
OpenAI · Model training
OpenAI states this agent crawls content that may be used in training its generative AI foundation models. It is not the ChatGPT search crawler.
ClaudeBot
Anthropic · Model training
Anthropic states this agent collects web content that could potentially contribute to training its generative AI models.
Google-Extended
Google · Generative AI use control
Google states this has no separate request user agent. It is a robots.txt token used in a control capacity over whether content informs Google's generative AI models.
Traditional Search
Conventional search engine crawlers, included so you can compare AI rules against search rules.
Googlebot
Google · Google Search index
Google's main search crawler. Preferences addressed to Googlebot affect Google Search and related products.
Bingbot
Microsoft · Bing Search index
Microsoft's search crawler for the Bing index.
Coverage
AI Crawlers You Can Check
This list is generated from the same registry the checker evaluates, so it always reflects what the tool actually supports.
| Crawler | Operator | Purpose | Category | Check available |
|---|---|---|---|---|
| OAI-SearchBot | OpenAI | ChatGPT search results | AI Search & Retrieval | Yes |
| Claude-SearchBot | Anthropic | Claude search quality | AI Search & Retrieval | Yes |
| PerplexityBot | Perplexity | Perplexity search index | AI Search & Retrieval | Yes |
| ChatGPT-User | OpenAI | User-initiated fetch | User-Initiated AI Fetch | Yes |
| Claude-User | Anthropic | User-initiated fetch | User-Initiated AI Fetch | Yes |
| Perplexity-User | Perplexity | User-initiated fetch | User-Initiated AI Fetch | Yes |
| GPTBot | OpenAI | Model training | AI Training & Data Use | Yes |
| ClaudeBot | Anthropic | Model training | AI Training & Data Use | Yes |
| Google-Extended | Generative AI use control | AI Training & Data Use | Yes | |
| Googlebot | Google Search index | Traditional Search | Yes | |
| Bingbot | Microsoft | Bing Search index | Traditional Search | Yes |
Google-Extended has no separate request user agent. Google documents it as a robots.txt token used in a control capacity, so RankNova evaluates the rules written against it rather than reporting a crawler fetch.
Workflow
How to Check AI Crawler Access
Enter your website domain or sub-domain.
RankNova retrieves the public robots.txt and homepage response.
Rules are evaluated for every supported crawler.
Review which crawlers are allowed or blocked.
Inspect the matched user-agent group and directive behind each result.
Test important same-host paths separately.
Review technical blockers such as HTTP status or indexing directives.
Adjust your crawler policy if appropriate, then run the check again.
Configuration
How to Control AI Crawlers with robots.txt
robots.txt rules are written per user-agent group. These examples use GPTBot, but the same shape applies to any agent in the table above.
Express a crawl restriction
User-agent: GPTBot Disallow: /
This expresses a crawl restriction for requests matching GPTBot. It is a stated preference that a well-behaved crawler is expected to honour, not a technical access control.
Express no restriction
User-agent: GPTBot Disallow:
An empty Disallow places no restriction on that group. Because each agent has its own documented purpose, policies should be chosen intentionally per crawler based on the operator's current documentation and your own goals.
RankNova does not recommend a single correct policy. Allowing every AI crawler and blocking every training crawler are both legitimate choices, and the right one depends on your business.
Troubleshooting
Why Is an AI Crawler Blocked?
A blocked or failed result can come from your crawler policy or from the request never succeeding. These are the common causes.
An explicit robots.txt rule
A Disallow written against that crawler's own user-agent group. RankNova shows the exact directive it matched.
A wildcard robots.txt rule
A rule under User-agent: * applies to any crawler without its own group, so a broad Disallow can block AI agents you never named.
A path-specific restriction
The crawler may be allowed at the root but blocked deeper in the site. Test the specific paths that matter.
HTTP 401 or 403
The page requires authentication or the server refused the request, regardless of what robots.txt permits.
HTTP 429
The server rate-limited the request. Aggressive throttling can look like a block to a crawler.
CDN, WAF or bot management
Edge security rules can challenge or refuse a crawler before the request ever reaches your application.
A noindex directive
A meta robots tag or X-Robots-Tag header asks that a page not be indexed. This does not prevent crawling, and operators differ in how they treat it.
JavaScript-dependent content
If the meaningful content is rendered client-side, an agent that does not execute JavaScript may receive very little. RankNova reads the raw HTML response and does not render JavaScript, so it cannot test this for you.
Read results carefully
"Allowed" Does Not Always Mean "Reachable"
An Allowed robots.txt result means RankNova did not find an applicable robots rule blocking that supported crawler for the tested path. It does not guarantee that the crawler can successfully fetch, render, index, use or cite the page.
- robots.txt
- Crawler allowed?
- HTTP reachable?
- Content available?
- Indexing signals
- Platform processing
- Potential retrieval or use
Only the first two stages are something a robots.txt check can answer. The final stage is never guaranteed by any configuration.
What each status means
- AllowedNo applicable robots.txt rule blocks that crawler from the tested path.
- BlockedA robots.txt rule in the matched user-agent group disallows the tested path.
- Not fetchablerobots.txt permits the request, but the page could not be retrieved as HTML, so indexing signals were not evaluated.
- UnknownThe check could not be completed, for example because robots.txt could not be retrieved.
Related but different
AI Crawler Checker vs Robots.txt Checker
Robots.txt checker
Validates that your directives are syntactically correct and reports how rules apply, usually against generic or search user agents.
AI Crawler Checker
Applies those same rules to specific AI-related user agents, and separates search, user-initiated and training purposes so you can see exactly which AI use you are permitting or declining.
A direct answer
Does Allowing AI Crawlers Improve AI Visibility?
Allowing a supported AI crawler means your robots.txt configuration is not intentionally blocking that crawler from the tested path. It does not guarantee that your website will appear, rank, be recommended or be cited in an AI-generated answer.
AI visibility can also depend on retrieval systems, indexing, content relevance, accessibility, authority, entity understanding, third-party sources and each platform's own systems and policies. These are not a published ranking formula, and no configuration guarantees an outcome.
Crawler access is a prerequisite you can control. Treat it as removing a blocker, not as a growth tactic.
Direct answers
AI Crawler Checker FAQ
What is an AI crawler checker?
An AI crawler checker reads your public robots.txt and evaluates the rules that apply to specific AI-related user agents. RankNova shows which supported crawlers are allowed or blocked for a given path, and which rule produced that result.
What is an AI crawler?
An AI crawler is an automated agent operated by an AI company that requests web pages. Different agents serve different purposes: some surface pages in an AI search product, some fetch a page because a user asked a question, and some collect content that may be used for model training.
Which AI crawlers does RankNova check?
RankNova checks the AI search agents OAI-SearchBot, Claude-SearchBot and PerplexityBot; the user-initiated agents ChatGPT-User, Claude-User and Perplexity-User; the training-related tokens GPTBot, ClaudeBot and Google-Extended; and the search crawlers Googlebot and Bingbot. The supported list on this page is generated from the same registry the checker uses.
What is the difference between GPTBot and OAI-SearchBot?
They do different jobs. OpenAI documents GPTBot as crawling content that may be used in training its generative AI foundation models, and OAI-SearchBot as surfacing websites in ChatGPT's search features. Blocking GPTBot does not block OAI-SearchBot, so a site can allow ChatGPT search while declining training use.
What is the difference between AI search crawlers and AI training crawlers?
AI search and retrieval agents fetch pages so they can be surfaced or linked in an AI answer. Training agents collect content that may contribute to building or improving a model. Because the purposes differ, a robots.txt policy should be set per agent rather than applied to every AI bot at once.
Does robots.txt block AI crawlers?
robots.txt expresses a crawl preference that a well-behaved crawler is expected to honour. The operators covered here document that they respect it. It is not a technical access control, so it cannot enforce a block the way authentication or a firewall rule can.
Can a crawler be allowed in robots.txt but still fail to access my website?
Yes, and this is common. A page can be permitted by robots.txt and still return 401, 403, 429 or a server error, be withheld by a CDN or WAF, or depend on JavaScript for its content. RankNova reports the HTTP status alongside the robots.txt result so you can tell these cases apart.
What does Allowed mean in RankNova?
Allowed means RankNova found no applicable robots.txt rule blocking that crawler from the tested path. It is a statement about your robots.txt configuration only, not a guarantee that the crawler fetched, rendered, stored, used or cited the page.
Does blocking AI crawlers affect Google rankings?
Blocking an AI-specific agent such as GPTBot or Google-Extended is separate from Googlebot, which is what Google Search uses. The practical risk is an overly broad rule that catches Googlebot too, which is why RankNova checks search crawlers alongside AI agents so you can spot that mistake.
Does allowing an AI crawler guarantee AI visibility?
No. Allowing a crawler only means your robots.txt is not blocking it from the tested path. Whether your content appears in an AI answer also depends on retrieval, content relevance, accessibility, authority, entity understanding, third-party sources and each platform's own systems and policies.
Can I test a specific page URL?
Yes. After the host-level check completes, you can test any path on the same host against any supported crawler. This matters because robots.txt rules are path-specific, so a crawler allowed on your homepage may be blocked on /pricing or /docs.
How often should I check AI crawler access?
Check after any change to robots.txt, after a CDN, WAF or hosting migration, and after a site rebuild or CMS change. These are the moments when crawler rules are most often altered unintentionally.
Next step
Find Out Which AI Crawlers Can Reach Your Website
Check your robots.txt policy against AI search, user-initiated and training agents in one pass.