What the validator checks

Crawlers skip lines they cannot understand, often without warning, so a single typo can quietly undo a rule. The validator reads your file the way a crawler would and lists every problem with its line number and a suggested fix.

How crawlers choose which rule applies

• A crawler first looks for the group whose User-agent name matches it most specifically. If none matches, it uses the * group.
• Within that group, the rule with the longest matching path wins.
• If an Allow and a Disallow rule match with the same length, Allow wins.
• * matches any run of characters and $ marks the end of an address, so “Disallow: /*.pdf$” blocks PDF files.
• Paths are case-sensitive: /Blog/ and /blog/ are different.

Testing a specific page

Enter a full address or a path such as /checkout/, choose a crawler and press Test. Crawllyt shows whether the page is allowed or blocked and quotes the exact line that decided it, which is the quickest way to debug an unexpected block.

When there is no robots.txt

If your site returns “not found” for robots.txt, crawlers assume they may visit everything. That is fine for many small sites. Server errors are different: Google treats them as a temporary block, so make sure the file loads reliably.