Robots.txt Generator — Create SEO Robots Files Online, Free
Generate a robots.txt file for your website. Configure user-agent, allow/disallow rules, and sitemap URL. Perfect for SEO — controls how search engines crawl your site. Free, no signup.
Last updated:
CommentsConfigure allow/disallow rules, user-agent, and sitemap URL to generate a complete robots.txt — copy and drop into your website root.
User-agent: *
Disallow: /admin
Disallow: /api
Allow: /
Sitemap: https://example.com/sitemap.xml
# Generated by devkits ToolsWhat is Robots.txt Generator?
Robots.txt is a file placed at the root of your domain that instructs search engine crawlers which paths they may and may not access. It uses the Robots Exclusion Protocol — a simple text format with User-agent, Allow, Disallow, and Sitemap directives. It's public (anyone can read it), it's a request (not an enforcement), and it's essential for every website.
How to generate a robots.txt file
- 1Set the User-agent (* for all bots, or a specific one like Googlebot).
- 2Add Disallow rules for paths to block (/admin, /api).
- 3Add Allow rules for paths to explicitly permit.
- 4Set your Sitemap URL.
- 5Copy the output to public/robots.txt in your project root.
Frequently Asked Questions
What is robots.txt used for?
Robots.txt tells search engine crawlers which pages they can and cannot access on your site. It's a text file placed at the root of your domain (yoursite.com/robots.txt). It can block specific paths from being indexed, but it's not a security measure — malicious bots can ignore it.
What's the difference between Disallow and Noindex?
Disallow in robots.txt prevents pages from being CRAWLED (the bot skips visiting them). Noindex in a meta tag or HTTP header prevents pages from being INDEXED (they appear in search results). A disallowed page can still be indexed if discovered through external links.
Where do I put robots.txt?
At the root of your website: https://yoursite.com/robots.txt. In Next.js, place it in the public/ folder. In static hosting (Vercel, Netlify), it's served automatically from public/robots.txt.
Try Next
Meta Tag Preview
Preview how your page will appear in Google search results and social share cards on Facebook, Twitter/X, and LinkedIn. Parses <title>, meta description, canonical, and full Open Graph / Twitter Card tags with diagnostics. 100% local.
Related Tools
HTTP Status Codes
Complete reference of HTTP status codes with meanings, use cases, and RFC citations. Search 60+ codes from 100 Continue to 511 Network Authentication Required.
cURL Converter
Convert cURL commands to Python (requests), JavaScript (fetch), Node.js, Go (net/http), PHP, or Java (HttpClient). Parses -H, -d, --data-urlencode, -F, -u, --json and more. 100% local.
URL Parser
Parse any URL into scheme, host, port, path, and query parameters — or build a URL from scratch with a form. Great for debugging redirect chains, tracking pixels, and OAuth flows. 100% local.
CIDR Calculator
Calculate IPv4 subnets from CIDR notation online. Get network address, broadcast, netmask, wildcard, usable host range, IP class, and binary representation. 100% local — nothing leaves your browser.
Color Converter
Convert colors between HEX, RGB, HSL, and HSV instantly with a live color preview and copy-ready CSS strings. Free, no signup, runs entirely in your browser.
User Agent Parser
Parse User-Agent strings to identify browser, engine, operating system, and device. Instantly see what your own browser sends.