Skip to content
KuponGuru
Back

Robots.txt Generator & Tester

Build a robots.txt with presets, then test any URL against it using Google's own matching rules.

FreeRuns in your browser

About the Robots.txt Generator & Tester

Create a robots.txt file without memorising its syntax: start from a preset, block AI training crawlers with one switch, add rules per crawler and list your sitemaps. Then test any URL against the file, yours or any site's, using the rules Google documents: the most specific crawler group, the longest matching path, and Allow winning a tie.

How to use it

  1. 1Choose a preset, or add crawler groups and Allow or Disallow rules.
  2. 2Add your sitemap URL and download the file for your site's root.
  3. 3Switch to Test and check which URLs each crawler may fetch.

Questions

Does Disallow keep a page out of Google?

No. robots.txt controls crawling, not indexing. A blocked page can still appear in results if other pages link to it. To keep a page out of search, allow crawling and add a noindex robots meta tag or header.

Which rules does Google read?

Only user-agent, allow, disallow and sitemap. Google ignores crawl-delay and noindex lines in robots.txt, reads at most 500 KiB of the file, and caches it for up to a day.

What does blocking Google-Extended do?

It stops your content being used for Google's Gemini models. It does not affect crawling or ranking in Google Search, which uses Googlebot.

Related tools

Free

Meta Tag Generator

Title, description, canonical, Open Graph and X (Twitter) tags in clean HTML, with a Google result and social card preview.

SEOUse tool
Free

Schema Markup Generator

JSON-LD for articles, organizations, local businesses, products, events, breadcrumbs and FAQs, checked against Google's rules.

SEOUse tool
Free

URL Slug Generator

Turn titles into clean, readable URL slugs, one at a time or a whole list, with Bangla kept if you want it.

SEOUse tool

Looking for something else? Browse every tool.