How Do4U fetches web pages

Our user agents

Do4U-Reader/1.0 (+https://do4u.eu)

Reads a single page whose address someone entered into a tool, for example for the website check, the accessibility check or title and description. Also reads sitemaps for internal links.

Do4U-Check/1.0 (+https://do4u.eu/bot)

Reads robots.txt, the sitemap and further pages of the same website for the llms.txt generator (limit per task: 8 pages).

Do4U-Webhook/1.0 (+https://do4u.eu/bot)

Tells a customer that a task is finished, at an address the customer entered.

How often

• One page per task. Only the llms.txt generator reads several pages, and then at most one request per second.
• Sitemaps for internal links: at most one request per second per website.
• Regular checks (website watch) only for pages a customer entered, every 30 days.
• We follow at most three redirects and never fetch addresses in internal networks.

robots.txt

The llms.txt generator follows robots.txt: it does not read pages that are blocked for Do4U-Check or for all crawlers. It always reads the start page whose address the customer entered.

A single page whose address someone enters directly is read the way a browser reads it, without asking robots.txt.

To block the llms.txt generator for your whole website:

User-agent: Do4U-Check
Disallow: /