Robots.txt Generator

# robots.txt generated by raatools.net

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

What is robots.txt?

Robots.txt is a text file placed at the root of a website (example.com/robots.txt) that tells search engine crawlers which pages or sections they are allowed or not allowed to crawl. It is part of the Robots Exclusion Protocol, a voluntary standard that well-behaved crawlers (like Googlebot and Bingbot) follow to respect website owners' crawling preferences.

A robots.txt file consists of rules specifying User-agent (which crawler the rule applies to) and Disallow or Allow directives (which paths to block or permit). You can also specify the location of your XML sitemap. While robots.txt cannot prevent determined scrapers from accessing your content, all major search engines honor it.

Robots.txt syntax

• User-agent: * — applies to all crawlers. Use specific names like Googlebot for targeted rules.
• Disallow: /admin/ — blocks crawling of the /admin/ directory and everything in it.
• Allow: /admin/public/ — overrides a broader Disallow to permit specific paths.
• Sitemap: https://example.com/sitemap.xml — tells crawlers where to find your sitemap.

How to use this tool

Select your desired settings — which crawlers to target, which directories to block, and whether to include a sitemap reference. The tool generates a valid robots.txt file that you can download or copy. It also validates existing robots.txt files for syntax errors and conflicting rules.

Common use cases

Block admin panels and login pages from indexing. Prevent crawling of duplicate content or thin pages. Block specific bots (like AI training crawlers). Prevent indexing of staging or development sites. Hide private directories that should not appear in search results. Rate-limit aggressive crawlers with Crawl-delay directives.

Example in practice

Suppose you want Google to index everything except your staging folder and internal search results. Add one group with User-agent: *, an Allow: / line, and two Disallow lines for /staging/ and /search. Append Sitemap: https://example.com/sitemap.xml. The generated file tells every crawler it may fetch the whole site apart from those two path prefixes, and points it to your sitemap for efficient discovery.

Common mistakes

A frequent error is using robots.txt to hide sensitive pages. Disallow only stops crawling, not indexing — a blocked URL can still appear in results if other sites link to it, so use a noindex meta tag or authentication instead. Another trap is blocking /css/ or /js/; Google needs those assets to render and score your pages. Finally, matching is case-sensitive, so Disallow: /Admin will not block /admin.

Frequently asked questions

Does robots.txt protect private content?

No. Robots.txt is a suggestion, not a security measure. Any person or bot can read your robots.txt file and deliberately ignore it. In fact, malicious bots often check robots.txt to find interesting directories to target. For truly private content, use authentication (passwords), server-side access controls, or the noindex meta tag.

What happens if I do not have a robots.txt file?

Without a robots.txt file, crawlers assume they are allowed to access everything on your site. This is fine for most sites. If a crawler requests robots.txt and gets a 404 (not found), it treats this as permission to crawl everything. Only create a robots.txt if you need to restrict crawling of specific sections.

Does robots.txt guarantee a page stays out of Google?

No. It only requests that compliant crawlers skip fetching the URL. To reliably keep a page out of search results, allow crawling and add a noindex directive, or protect it behind a login. Malicious bots ignore robots.txt entirely.