See which search and AI crawlers your robots.txt allows.

Inspect the robots.txt attached to a public site and interpret its rules for Googlebot, Bingbot, GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot and other common agents.

Report includesEvidence based

Crawler-by-crawler access evidence

File status

Whether robots.txt is available and how it responded.

Agent rules

Allow or disallow interpretation for each supported user agent.

Evidence

The rule behind each access result, rather than a bare badge.

Sitemaps

Sitemap locations declared in robots.txt.

Results describe the submitted URL and separate measured evidence from anything still unverified.

What the check covers

Useful answers, with the limits made clear.

Search crawlers

Review access for Googlebot and Bingbot.

AI crawlers

Review access for GPTBot, ClaudeBot and PerplexityBot.

User-triggered agents

Keep ChatGPT-User separate from model-training crawlers where rules differ.

Rule evidence

Show the matching directive or default behavior for each agent.

Why this matters

Allowed does not mean visited, indexed or cited

Robots.txt communicates crawl permissions. It cannot prove that a crawler visited the page, that a search engine indexed it, or that an AI answer cited it. The report labels this distinction clearly so access is not confused with activity.

Rules can vary by user agent and path. This free check evaluates the submitted URL against common crawler identities and preserves the evidence used for the result.

Questions answered

Robots.txt Checker FAQ

Does robots.txt block indexing?+

Robots.txt primarily controls crawling, not guaranteed indexing. A blocked URL can sometimes still appear in search if discovered elsewhere, usually with limited information.

Does Allowed mean the bot has visited?+

No. It only means the evaluated rules permit access. Real visits require server or edge logs.

Should I allow every AI crawler?+

That is a business and content-policy choice. The checker shows current access so you can make an informed decision; it does not assume every bot should be allowed.

What happens if robots.txt is missing?+

Crawlers generally treat a confirmed missing file as no robots.txt restrictions, but temporary server errors can be handled differently. The HTTP status is useful context.

Continue checking

Related free tools

View all tools →