Skip to content
Get started

AI crawlers in robots.txt

Do4U builds the rules for AI crawlers in your robots.txt. You choose per group or per crawler: model training and datasets, search and answer index, and fetches on a user's request. Each crawler in the list comes with its operator, its purpose and a link to the operator's documentation. If you enter your website, Do4U reads your current robots.txt, shows which AI crawlers it allows or blocks today, and adds a clearly marked AI section at the end. Your other groups and sitemap lines stay as they are. The code then reads the new file again and checks that every crawler has the rules you chose. Everything happens in code, without AI, and the file is made in your browser.

Try it now Pricing

Free and without an account: AI crawler check for robots.txt →

Who it is for

For website owners, agencies and developers who want to decide which AI crawlers may use their content, without breaking the rules they already have.

Typical uses

  • Block AI training crawlers and still be found in AI search
  • Check which AI crawlers your robots.txt allows today
  • Add an AI section to an existing robots.txt without losing rules
  • Allow AI crawlers only for selected paths

Example

Input

Website: https://example.com Current robots.txt: User-agent: * Disallow: /admin/ User-agent: GPTBot Disallow: / Sitemap: https://example.com/sitemap.xml Choice: block training, allow search, allow user fetches

Result

GPTBot: blocked (own rules) → blocked ClaudeBot: partly blocked (rules for all) → blocked OAI-SearchBot: partly blocked (rules for all) → partly blocked (own rules, /admin/ stays blocked) The rule for all crawlers and the sitemap line stay unchanged. The old GPTBot group stays in the file as a comment, and the new AI section is added at the end.

A fixed example, shortened. Real results are AI-generated: please check them.

Price and limits

Price
free, without sign-up
Available with
Free, With package
Input limit
Up to 60,000 characters per file

Privacy

Processed on our own servers in the EU (Poland). Inputs and results are deleted after 7 days (Free) or 30 days (packages), or right after you see the result if you choose not to store them. Images are deleted right after processing.

In the API

Tool name: ai_robots_txt. Create a job with POST /api/v1/jobs and fetch the result with GET /api/v1/jobs/{id}. You find your API key in your account.

curl -X POST https://do4u.eu/api/v1/jobs \
  -H "Authorization: Bearer d4u_…" -H "Content-Type: application/json" \
  -d '{"tool": "ai_robots_txt", "site_url": "https://example.com", "groups": {"training": "block", "search": "allow", "user": "allow"}}'

Full API documentation →

Questions and answers

Does robots.txt really stop AI crawlers?

No file can force a crawler. Following robots.txt is voluntary, and each crawler decides itself whether it obeys the rules. Some fetchers that act on a user's request may ignore robots.txt, as their operators say. Do4U marks these in the list.

Will my other rules and my sitemap be kept?

Yes. Do4U keeps all other groups and all sitemap lines. Only lines that name a crawler you now decide about become comments, so that old and new rules do not mix. The AI section is clearly marked at the end of the file, and a second run replaces it.

Does blocking Google-Extended remove my site from Google?

No. Google-Extended is not a crawler but a token in robots.txt. According to Google, it controls the use of your content for Gemini models and does not affect Google Search. Applebot-Extended works in a similar way for Apple.

Is my text stored?

Only as long as needed: 7 days on Free, 30 days with a package – or not at all if you choose not to store it. Processing happens on our own servers in the EU (Poland).

Can I use the result right away?

Results are AI-generated. Please read them before you publish or send them.

Related tools

  • llms.txt generator

    Read your website and get a ready llms.txt file for AI assistants: the name of your website, a short summary and your most important pages, each with one sentence. The code builds and checks the file; you download or copy it.

  • AI visibility check

    Find out how well a page answers the questions people typically ask about its topic. The AI proposes the questions and rates each answer with an exact quote from your page; the code checks the quotes and calculates the coverage. You also see missing answers and hints on FAQ markup, llms.txt and robots.txt. No ranking promise.

All tools →