Site Health
Your site · Applicable SEO checks passed
View issues and fixesFree Robots.txt Generator
Write a robots.txt file without learning the syntax. Pick a default rule, list folders to keep crawlers out of, add your sitemap and choose exactly which search engines and AI crawlers may visit, with a live preview as you go.
What it checks
- A default rule for every crawler: allow or block
- Folders to keep out of search, and an optional crawl delay
- Links to one or more sitemaps
- Separate choices for Google, Bing, DuckDuckGo, Apple, Yandex and Baidu
- Separate choices for 14 AI crawlers from OpenAI, Anthropic, Google, Apple, Perplexity, Meta and others
- Quick presets, including “block AI training, allow AI search”
How to use it
- Log in to your free Crawllyt account.
- Set your rules. The preview updates instantly and you can edit it by hand.
- Download robots.txt, upload it to the root of your site, then confirm it with the Robots.txt Validator.
robots.txt is followed voluntarily by reputable crawlers. Protect private content with passwords, not robots.txt.
What robots.txt does
robots.txt is a small text file at the root of a website, such as example.com/robots.txt. Before a well-behaved crawler fetches your pages, it reads this file to learn which parts of the site you would rather it skipped.
Each set of rules starts with a User-agent line naming a crawler, or * for all of them, followed by Disallow and Allow lines listing paths. A Sitemap line can point crawlers to your sitemap.
Choosing rules for AI crawlers
AI companies run different crawlers for different jobs. Some collect pages to train future models, while others fetch pages live to answer a user’s question or to build an AI search index, often with a link back to you.
If you want to stay visible in AI answers but keep your content out of training data, block the training crawlers and leave the search and user-requested ones alone. The “Block AI training, allow AI search” preset does exactly this.
- Training crawlers include GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent and Bytespider. Google-Extended and Applebot-Extended are control tokens that opt you out of AI training without affecting normal search.
- Search and user-requested crawlers include OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User.
Example robots.txt
User-agent: *
Disallow: /admin/
Disallow: /checkout/
User-agent: GPTBot
User-agent: CCBot
Disallow: /
Sitemap: https://example.com/sitemap.xmlMistakes to avoid
- Leaving “Disallow: /” in place after moving a site from staging to live, which asks every crawler to stay away.
- Blocking CSS, JavaScript or image folders that search engines need to render your pages.
- Using robots.txt to hide a page from Google. A blocked page can still appear in results if other sites link to it; use a noindex tag and let it be crawled instead.
- Forgetting that each subdomain needs its own file: shop.example.com does not use example.com/robots.txt.
Frequently asked questions
Where do I upload robots.txt?
Upload it to the top folder of your website so it opens at https://yourdomain.com/robots.txt. Crawlers ignore robots.txt files placed anywhere else.
Will blocking a page remove it from Google?
Not always. Blocking stops Google from reading the page, but the address can still be indexed if it is linked from elsewhere. To remove a page, allow crawling and add a noindex tag.
Should I block AI crawlers?
It depends on your goals. Blocking training crawlers keeps your content out of future model training. Blocking AI search crawlers may also stop your pages appearing in AI answers with links back to you.
Does Google follow Crawl-delay?
No. Google ignores Crawl-delay. Bing and Yandex respect it. Use it only if your server struggles with crawl traffic.
Why does an allowed crawler repeat my blocked folders?
A crawler that has its own group ignores the general * rules, so Crawllyt copies your blocked folders into that group to keep them protected.
Is the generator free?
Yes. It is included with a free Crawllyt account.
Home › SEO › Site Audit
Site Audit
Real page checks. Affected URLs. Clear next steps.
Start with a URL
Your report will show every check, including passed checks and items requiring manual verification. No scores are shown until a website has been fetched successfully.
On-page SEOTitles, descriptions and lengths, duplicate titles across the sample, H1–H6 headings, image alt text.
Technical & trackingRobots.txt, GA4 and GTM IDs, Search Console meta tag, broken links and redirect destinations.
AI search & GEOSchema types, llms.txt, visible-content signals and clearly labelled heuristic scores.
Crawled Pages
AI Search Health Estimate
Observed page signals, not measured AI rankings
How it worksAI Crawler Restrictions
Analysed pages blocked by robots.txt
View robots.txt rulesGoogle-Extended controls AI usage, not Google Search indexing.
Errors
Affected occurrences across checked pages
Top Issues
View all checks and fix instructionsGEO signals*
Error occurrences
Warning occurrences
Passed checks
All checks, issues & fix instructions
Expand any row for affected items and instructions. “Review” means the result needs context or cannot be confirmed by a public scan.
Page titles, descriptions & content
Pagination also applies to the tracking, schema and checked-link tables below. Internal links count all occurrences; unique destinations are shown separately.
| Analysed URL | Title / length | Meta description / length | Headings & images | Internal links |
|---|
Tracking, verification & public files
View robots.txt code
Sitemap URLs checked
| Page | Google Analytics | Google Tag Manager | Search Console meta |
|---|
Detected IDs do not confirm that tracking fires correctly. No meta tag does not mean Search Console is unverified: DNS, files and other methods may be used.
AI readiness & GEO review
How these scores work — important limitations
AI readiness v2 weights ten observed signals: indexability (15), main heading (10), section structure (10), title (10), summary (10), valid structured data (10), visible content (15), language (5), descriptive internal links (10) and image alt attributes (5). Content and link checks allow partial credit. Thresholds are editorial tool guidelines, not search requirements. A score of 100 means all checklist points were met, not guaranteed AI citations.
GEO averages three observed signals: a question/FAQ heading, a recognised entity schema type, and a descriptive internal link. Scores do not measure AI rankings, citation probability, factual accuracy or content authority. llms.txt and tracking codes do not affect these scores.
SEO score is the percentage of applicable on-page/technical check categories passed. Manual-review checks, contextual “few links” checks and redirects are excluded. It is not a SEMrush or Google score. Character targets are your configured editorial guidelines.
Google AI features guidance · Search Console ownership verification
Checked internal links & redirects
| Requested URL | Status | Final destination / detail |
|---|
Bounded sample audit, not an exhaustive site crawl. External scripts, consent flows and JavaScript-rendered content are not executed. Blocked sites are reported without invented scores.