robots.txt tells crawlers which paths they may or may not crawl. Set a user-agent, add disallow paths (one per line), and a sitemap URL — then copy the generated file to your site root.
What is Robots.txt Generator?
A robots.txt generator builds the file that tells search engine crawlers which parts of your site they may and may not access. Getting it right controls crawl budget and keeps private or duplicate areas out of the index.
How it works
robots.txt sits at your domain root and uses simple directives — User-agent (which bot), Allow, Disallow, and Sitemap. The generator turns your choices into correctly-formatted rules you can paste straight into the file.
How to use this tool
Choose which paths to allow or block and add your sitemap URL. The tool outputs a ready-to-use robots.txt. Upload it to your site root and verify it in Search Console.
Why it matters
A misconfigured robots.txt can accidentally block your entire site from Google (a single wrong line) or waste crawl budget on junk pages. Generating it correctly avoids both extremes and points crawlers at your sitemap.
Frequently asked questions
What does robots.txt do?
It instructs search engine crawlers which URLs they can and can't access, helping manage crawl budget and keep certain areas out of search.
Can robots.txt hide a page from Google?
Not reliably — Disallow stops crawling but a blocked page can still be indexed if linked elsewhere. Use a noindex meta tag to truly keep a page out of search.
Should I link my sitemap in robots.txt?
Yes — adding a Sitemap directive helps crawlers discover all your important URLs efficiently.