Robots.txt access rules
Reads the host's robots.txt file and evaluates whether a selected path is allowed or blocked for a supported crawler.
RankNova Website Crawl Test
Free Website Crawl Test – Check Google & AI Crawler Access
Check whether search engines and AI crawlers can access your website. RankNova's free Website Crawl Test helps identify crawlability issues, robots.txt restrictions, sitemap problems and indexing signals that may prevent your pages from being discovered.
Free crawlability check for Google, Bing and major AI crawlers. No technical knowledge required.
Robots.txt accessSitemap discoveryGooglebot and BingbotAI crawler path checks
Start with the definition
What Is a Website Crawl Test?
A website crawl test checks whether search engine crawlers can access and understand your website.
Before a page can appear in search results, crawlers such as Googlebot need to discover and access it. Technical configurations including robots.txt rules, indexing directives, redirects and sitemap availability can affect this process.
RankNova helps you identify these crawlability signals so you can find potential problems before they affect your website's search visibility.
Supported checks
What Does RankNova's Website Crawl Test Check?
The host check and its report use the existing RankNova crawl API. The cards below describe what the scanner can actually evaluate; it does not claim to perform a full site crawl.
Sitemap discovery
Checks sitemap references from robots.txt and the host's standard sitemap.xml location, then reports whether a usable sitemap was found.
Homepage crawlability
Reports the homepage access result from the host-level crawl and highlights when robots.txt blocks the root path.
Indexability signals
Checks fetched HTML for noindex meta or X-Robots-Tag signals on the homepage and paths you submit in the report.
HTTP status on path checks
Shows the HTTP response status when a selected path can be fetched for a supported crawler check.
Search and AI crawler access
Lets you compare Googlebot, Bingbot, and supported AI crawler rules on individual paths from the same host.
Simple workflow
How to Run a Website Crawl Test
Enter your website URL
Start the crawl test
Review crawler access
Find crawlability issues
Fix issues and test again
Search visibility foundation
Why Website Crawlability Matters for SEO
Search engines generally need to discover and crawl a page before they can evaluate it for search. If robots.txt blocks a valuable path, the server cannot be reached, or a page returns an error, that content may be harder to process.
Crawlability is also a foundation for AI and GEO work. This test checks access signals; it does not promise rankings, citations, or placement in an AI answer.
Two different signals
Website Crawling vs Indexing
Crawling means discovering and accessing a webpage. Indexing means evaluating and potentially storing eligible webpage information in a search engine's index.
A crawlable page is not automatically guaranteed to be indexed. RankNova keeps these concepts separate: the report evaluates robots.txt access and, when HTML can be fetched, checks noindex meta and X-Robots-Tag signals.
Crawling
Can the selected crawler reach and fetch the path under the host's access rules?
Indexing
Does fetched HTML expose a noindex signal that tells search systems not to index the page?
Search and AI crawler access
Check AI Crawler Access
RankNova checks supported crawler configurations through robots.txt rules. After the host-level report completes, use the path checker to choose a path and compare access for Googlebot, Bingbot, GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, or Google-Extended.
The result is an access and indexability signal for the selected path. It is not a claim that an AI platform will cite, rank, or display the page.
GO
Googlebot
Search crawler access for the paths you choose in the report.
BI
Bingbot
Bing search crawler access based on the host's robots.txt rules.
GP
GPTBot
OpenAI crawler access for public paths you want to review.
CH
ChatGPT-User
Interactive retrieval access for the selected path.
CL
ClaudeBot
Anthropic crawler access based on robots.txt directives.
PE
PerplexityBot
Perplexity crawler access for a path on the tested host.
GO
Google-Extended
Google AI usage access controls for the selected path.
Troubleshooting guide
Common Website Crawlability Problems
Some problems are directly surfaced by the report; others need a broader site audit. These distinctions help you choose the next check instead of assuming the host-level result explains everything.
robots.txt blocks important pages
An overly broad Disallow rule can keep search or AI crawlers away from public pages. Review the matched directive shown in the report.
Accidental noindex directives
A noindex meta tag or X-Robots-Tag can make a fetched page ineligible for indexing. Crawlability and indexability are separate checks.
Missing or incorrect sitemap
A sitemap helps crawlers discover important URLs. Confirm that robots.txt references the right sitemap and that the sitemap responds with valid XML.
Broken or inaccessible URLs
A path can be allowed by robots.txt and still return an error or fail to provide fetchable HTML. Use the report's path check to investigate a URL.
Redirects and orphan pages
Redirect chains and pages with no useful internal links can reduce discovery. These are broader site checks to review after this host-level test.
Built for practical checks
Who Should Use the Website Crawl Test?
Anyone responsible for a public website can use the check before launching a page, changing robots.txt, or investigating a drop in discoverability.
Related RankNova tools
Continue your technical SEO and GEO checks
Use descriptive next steps when the crawl report points to a wider issue. Each link below leads to an existing RankNova route.
AI Crawler CheckerRun a focused check for supported AI and search crawler access.Meta Tag CheckerReview page-level metadata after you resolve crawl barriers.Website Speed TestCheck performance signals that affect the experience after a page is fetched.SEO AuditContinue with a wider technical SEO review of your website.Technical SEOLearn how crawlability fits into broader technical site health.GEO ScoreConnect crawl access with RankNova's AI search readiness workflow.
Direct answers
Website Crawl Test FAQ
What is a website crawl test?
A website crawl test checks whether crawlers can access a host and selected paths through robots.txt rules. RankNova also checks sitemap discovery and available indexability signals.
How do I check if Google can crawl my website?
Enter your domain in RankNova's Website Crawl Test. After the host check finishes, use the path checker and select Googlebot to review a specific URL.
Can robots.txt stop Google from crawling my website?
Yes. A Disallow rule can block a path for Googlebot. RankNova shows the matched robots.txt directive when you check a supported path.
Why is my website not being crawled?
Possible causes include restrictive robots.txt rules, an unavailable server, error responses, missing sitemap signals, or pages that are difficult to discover. The report helps separate access signals from broader site issues.
What is the difference between crawling and indexing?
Crawling is discovering and accessing a page. Indexing is evaluating and potentially storing information about that page. A crawlable page is not automatically guaranteed to be indexed.
Can I check AI crawler access?
Yes. The report path checker supports GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, and Google-Extended, as well as Googlebot and Bingbot.
Is the RankNova Website Crawl Test free?
Yes. You can start a public host-level crawl test without an account. The report is available through the browser session that started the test.
Next step
Make Sure Search Engines Can Access Your Website
Find crawlability problems before they affect your search visibility.