Home / Web Tools / Robots.txt Generator

Robots.txt Generator

Build a correct robots.txt - with optional AI-crawler blocking.

How to use

  1. Choose what crawlers may access and list blocked paths.
  2. Optionally block AI training bots.
  3. Copy and save as /robots.txt on your site.

Frequently asked questions

Does robots.txt actually block bots?

Well-behaved crawlers respect it; it's a convention, not a lock. Sensitive content needs real authentication.

Related tools

  • Open Graph Generator — Generate OG + Twitter card tags so links preview beautifully.
  • DNS Lookup — Look up A, AAAA, MX, TXT, NS and other DNS records for any domain.
  • IP Lookup — Find the location, ISP, organisation and network details behind any IP address.
  • WHOIS Lookup — Check a domain's registrar, registration and expiry dates, status and nameservers.
  • Sitemap Generator — Turn a list of URLs into a valid XML sitemap for Google and Bing.
  • Open Graph Preview — Paste your meta tags and preview how your link looks on Facebook, LinkedIn and X.

See all web tools