A robots.txt generator builds the small text file that tells search engines and AI crawlers which parts of your site they may visit. This free custom robots.txt generator for Blogger (and any website) creates a valid robots.txt in seconds: pick a Blogger, WordPress or Shopify preset, add your sitemap, set allow and disallow rules, and optionally block AI training bots like GPTBot, ClaudeBot and Google-Extended in one click. On Blogger, paste the result under Settings, Crawlers and indexing, Enable custom robots.txt.
Looking for a custom robots.txt generator for Blogger? You are in the right place. Your robots.txt is the first file a crawler reads before it touches your blog or site. It lists simple rules that decide which pages a bot may crawl and which to skip. On Blogger, the default file blocks your archive and search pages in a way that is not always ideal, so a clean custom file helps Google crawl your posts and find your sitemap faster.
This free robots.txt generator for Blogger and websites does the syntax for you. Pick the Blogger preset (or WordPress, Shopify, custom), drop in your sitemap, add any paths to block, and, if you want, tick the AI crawlers you would rather keep out. You get a valid file to copy or paste straight into Blogger. Nothing installs, and your inputs never leave your browser.
Key takeaways
- On Blogger, use the Blogger preset, then paste the file under Settings, Crawlers and indexing, Enable custom robots.txt.
- A robots.txt file controls crawler access. It does not hide a page from Google on its own, it only asks bots not to crawl it.
- It must sit at your domain root, reachable at
yourdomain.com/robots.txt. Blogger uses Settings, Crawlers and indexing instead. - A single wrong
Disallow: /can block your entire site, so always preview before you publish. - You can now block AI crawlers (GPTBot, ClaudeBot, Google-Extended and more) to stop AI models training on your content, while still letting Google and Bing rank you.
Why use this robots.txt generator?
Valid every time
Correct directive order and syntax, so Google, Bing and other crawlers actually understand the rules.
Presets for every platform
Tested starting points for WordPress, WooCommerce, Shopify, Blogger and custom sites.
AI crawler control
Block GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others with a single tick.
Copy or download
Grab the file as text or download a ready robots.txt. Everything runs privately in your browser.
How to use it
- Pick your platform preset (WordPress, WooCommerce, Shopify, Blogger, or Custom).
- Add your full sitemap URL, for example https://yourdomain.com/sitemap.xml.
- List any folders or pages to block under Disallow, one per line. Add Allow rules for exceptions.
- Optionally tick the AI crawlers you want to keep out, and set a crawl-delay for aggressive bots.
- Click Generate, then Copy or Download, and place the file at your site root (or paste it into Blogger, Settings, Crawlers and indexing).
How to add a custom robots.txt in Blogger
Blogger lets you replace its default file with your own, so this is the exact flow to use the custom robots.txt generator for Blogger above:
- In the tool, choose the Blogger blog preset. It pre-fills
Disallow: /search, which stops Google wasting crawl budget on search and label pages. - Paste your blog sitemap in the Sitemap field:
https://yourblog.blogspot.com/sitemap.xml(or your custom domain). - Optionally tick any AI crawlers you want to block, then click Generate and Copy.
- In Blogger, open Settings, Crawlers and indexing, turn on Enable custom robots.txt, and paste the file.
- Save. Check it loads at
yourblog.blogspot.com/robots.txt.
A typical custom robots.txt for a Blogger blog looks like this:
User-agent: *
Disallow: /search
Allow: /
Sitemap: https://yourblog.blogspot.com/sitemap.xmlThe Disallow: /search line keeps search and label URLs out of Google, while Allow: / makes sure your posts and pages stay crawlable.
What the main directives mean
A robots.txt file is just a list of rules grouped under one or more crawlers. Here is what each line does.
| Directive | What it does | Example |
|---|---|---|
User-agent | Names the crawler the rules apply to. * means every bot. | User-agent: * |
Disallow | Asks the bot not to crawl a path. Empty value means block nothing. | Disallow: /cart/ |
Allow | Carves out an exception inside a disallowed folder. | Allow: /wp-admin/admin-ajax.php |
Sitemap | Points crawlers to your XML sitemap so pages get found faster. | Sitemap: https://site.com/sitemap.xml |
Crawl-delay | Asks some bots to wait between requests. Google ignores it; Bing honours it. | Crawl-delay: 10 |
Should you block AI crawlers?
AI companies send their own bots to collect text for training and for answer engines like ChatGPT and Perplexity. You can allow them (helpful if you want to be cited in AI answers) or block them (if you would rather your content is not used for model training). This tool lets you choose per bot. A few common ones:
| Crawler | Who runs it | What it feeds |
|---|---|---|
GPTBot | OpenAI | ChatGPT model training |
OAI-SearchBot | OpenAI | ChatGPT Search results |
ClaudeBot | Anthropic | Claude model training |
Google-Extended | Gemini training (separate from Search) | |
PerplexityBot | Perplexity | Perplexity answer engine |
CCBot | Common Crawl | Open dataset used by many AI models |
Blocking Google-Extended stops Gemini training but does not affect your normal Google Search ranking, so it is a safe way to opt out of AI training while staying visible in search.
A safe starter robots.txt
If you just want a clean, safe file that lets search engines in and points them to your sitemap, this is it:
User-agent: *
Disallow:
Sitemap: https://yourdomain.com/sitemap.xmlThe empty Disallow: line means nothing is blocked. Swap in your real domain and you are done.
Popular use cases
- Bloggers who want Google to crawl posts but skip search and label pages.
- WordPress and WooCommerce owners blocking admin, cart and checkout URLs.
- Publishers who want to opt out of AI training while staying in Google Search.
- Agencies preparing a correct robots.txt for client sites in seconds.
- Anyone who accidentally added Disallow: / and needs a clean file fast.
Frequently asked questions
How do I create a custom robots.txt for Blogger?
Pick the Blogger preset in the generator above, add your blog sitemap, then click Generate and Copy. In Blogger, go to Settings, Crawlers and indexing, turn on Enable custom robots.txt, and paste the file. It becomes live at yourblog.blogspot.com/robots.txt.
Is a custom robots.txt better than the default one on Blogger?
For most blogs, yes. Blogger default settings can block your archive and search pages, and beginners often disable the wrong thing. A clean custom robots.txt lets you keep posts crawlable, block only search and label URLs, and point Google to your sitemap.
Where do I put the robots.txt file?
On a normal website it goes in the root folder so it is reachable at yourdomain.com/robots.txt. On Blogger, paste it under Settings, Crawlers and indexing, Enable custom robots.txt.
Does robots.txt hide a page from Google?
No. It asks crawlers not to visit a path, but a blocked URL can still appear in search if other sites link to it. To keep a page out of Google entirely, use a noindex meta tag or password protection instead.
How do I block AI crawlers like ChatGPT?
Tick the AI bots in the tool (GPTBot, ClaudeBot, Google-Extended and others). It adds a separate rule block that disallows each one, so AI models stop crawling your content while Google and Bing keep ranking you.
Will blocking Google-Extended hurt my rankings?
No. Google-Extended only controls Gemini AI training and is separate from Googlebot, which handles Search. Blocking it opts you out of AI training without touching your search visibility.
What is the difference between Disallow and noindex?
Disallow stops a bot from crawling a URL. Noindex lets it crawl but tells it not to list the page. For reliable removal from search, use noindex, because a disallowed page can never read its own noindex tag.
Do I need a robots.txt at all?
Small sites can run fine without one, but a proper file helps you block low-value URLs, save crawl budget, point bots to your sitemap and control AI access, so it is worth having.
Does the crawl-delay line work for Google?
Google ignores Crawl-delay and manages its own crawl rate in Search Console. Bing and some other crawlers do honour it, so it is still useful for them.
Is this robots.txt generator free?
Yes, completely free with no signup and no limits. Everything runs in your browser, so your paths and sitemap URL never leave your device.
Want this done for you?
We build fast, SEO-ready websites and run digital marketing for businesses across Noida & Delhi NCR.