Robots.txt Generator

# robots.txt generated by raatools.net

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

What is robots.txt?

Robots.txt is a text file placed at the root of a website (example.com/robots.txt) that tells search engine crawlers which pages or sections they are allowed or not allowed to crawl. It is part of the Robots Exclusion Protocol, a voluntary standard that well-behaved crawlers (like Googlebot and Bingbot) follow to respect website owners' crawling preferences.

A robots.txt file consists of rules specifying User-agent (which crawler the rule applies to) and Disallow or Allow directives (which paths to block or permit). You can also specify the location of your XML sitemap. While robots.txt cannot prevent determined scrapers from accessing your content, all major search engines honor it.

Robots.txt syntax

• User-agent: * — applies to all crawlers. Use specific names like Googlebot for targeted rules.
• Disallow: /admin/ — blocks crawling of the /admin/ directory and everything in it.
• Allow: /admin/public/ — overrides a broader Disallow to permit specific paths.
• Sitemap: https://example.com/sitemap.xml — tells crawlers where to find your sitemap.

How to use this tool

Select your desired settings — which crawlers to target, which directories to block, and whether to include a sitemap reference. The tool generates a valid robots.txt file that you can download or copy. It also validates existing robots.txt files for syntax errors and conflicting rules.

Common use cases

Block admin panels and login pages from indexing. Prevent crawling of duplicate content or thin pages. Block specific bots (like AI training crawlers). Prevent indexing of staging or development sites. Hide private directories that should not appear in search results. Rate-limit aggressive crawlers with Crawl-delay directives.

Frequently asked questions

Does robots.txt protect private content?

No. Robots.txt is a suggestion, not a security measure. Any person or bot can read your robots.txt file and deliberately ignore it. In fact, malicious bots often check robots.txt to find interesting directories to target. For truly private content, use authentication (passwords), server-side access controls, or the noindex meta tag.

What happens if I do not have a robots.txt file?

Without a robots.txt file, crawlers assume they are allowed to access everything on your site. This is fine for most sites. If a crawler requests robots.txt and gets a 404 (not found), it treats this as permission to crawl everything. Only create a robots.txt if you need to restrict crawling of specific sections.