Website AuditorWebsite Auditor

robots.txt Checker

Fetch and validate a site's robots.txt: syntax, blocked paths, disallowed crawlers and sitemap references.

Free, unlimited, no signup.

About robots.txt Checker

A single stray Disallow line in robots.txt can quietly de-index your entire product, and a leaky one can hand attackers a directory of admin paths to probe. This checker validates the syntax, resolves every rule against a set of common URLs, and confirms your sitemap reference actually resolves.

What this tool checks

robots.txt Checker focuses on the following signals from your site's live response:

  • robotsmodule in the full audit

Why it matters

Signals covered by robots.txt Checker are the ones attackers, browsers, and search engines look at first. A weak result here means real users are exposed — via downgrade attacks, broken trust warnings, indexing problems, or leaked data — long before anything obvious breaks. Fixing them is usually a config change, not a rewrite, and the impact is immediate.

How to interpret your result

  • Pass — the signal meets modern best practice. Keep it monitored; regressions happen after deploys.
  • Warn — functional but weaker than recommended. Usually a quick header, DNS, or config tweak away from a full pass.
  • Fail — a real risk to users or search visibility. Follow the recommendation shown next to the finding — it's copy-pasteable.

Best practices

  • • Re-run after every deploy — configuration drift is the #1 cause of regressions.
  • • Compare against a competitor's report to spot easy wins.
  • • Wire the public API into CI to gate merges on the score.
  • • Fix warns before failures — they're the cheapest to close and pay off compounding gains.

Frequently asked questions

Not directly, but exposing admin paths there gives attackers a map. We flag it.

Related guides

Guide

Website Security Tips

A pragmatic checklist of the security wins that give the biggest reduction in risk for the least effort.

Read

Related articles

headers

Why Your Website Needs HSTS in 2026

One header, one config line, and downgrade attacks stop reaching your users. If your site doesn't ship HSTS in 2026, you're leaving free security on the table.

Read

headers

How to Roll Out CSP in Report-Only Mode

CSP is one of the most effective XSS defenses ever shipped — and one of the most feared to deploy. Report-only mode turns a scary launch into a boring one.

Read

dns

DMARC in Plain English

SPF, DKIM, DMARC. Three DNS records, one goal: stop other people from sending email as you. Here is what each one actually does.

Read

More tools

Tool

Sitemap Checker

Confirm your sitemap is discoverable and valid.

Read