robots.txt Checker
Fetch and validate a site's robots.txt: syntax, blocked paths, disallowed crawlers and sitemap references.
A single stray Disallow line in robots.txt can quietly de-index your entire product, and a leaky one can hand attackers a directory of admin paths to probe. This checker validates the syntax, resolves every rule against a set of common URLs, and confirms your sitemap reference actually resolves.
robots.txt Checker evaluates the following signals from your target host:
Frequently asked questions
- Not directly, but exposing admin paths there gives attackers a map. We flag it.
Related guides & resources
headers
Why Your Website Needs HSTS in 2026
One header, one config line, and downgrade attacks stop reaching your users. If your site doesn't ship HSTS in 2026, you're leaving free security on the table.
Read
headers
How to Roll Out CSP in Report-Only Mode
CSP is one of the most effective XSS defenses ever shipped — and one of the most feared to deploy. Report-only mode turns a scary launch into a boring one.
Read
dns
DMARC in Plain English
SPF, DKIM, DMARC. Three DNS records, one goal: stop other people from sending email as you. Here is what each one actually does — and how to turn them on without losing legitimate mail.
Read