AM A robots.txt file sits at the root of your domain and tells crawlers which URLs they may request. One wrong line can stop search engines from crawling your whole store, so this generator builds the file from a tested preset and checks it for the mistakes that cause real damage.
How to use the robots.txt generator
- Pick a preset: WordPress and WooCommerce, Shopify, or static site.
- Add rules. Enter any extra Allow or Disallow paths, such as a staging folder or internal search results.
- Toggle AI crawlers. Switch on any of GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, Bytespider, Applebot-Extended and meta-externalagent to block them.
- Add your sitemap URL so the file includes a Sitemap line.
- Read the warnings. The tool flags rules that block everything or block CSS and JavaScript files.
- Download robots.txt and publish it at the root of your domain, for example https://example.com/robots.txt.
Platform presets
| Platform | How robots.txt works there | How to publish |
|---|---|---|
| WordPress and WooCommerce | WordPress serves a virtual robots.txt unless a real file exists in the site root | Upload the file to the root folder, or paste it into your SEO plugin's robots.txt editor |
| Shopify | Shopify generates robots.txt automatically from the robots.txt.liquid theme template | Add rules through robots.txt.liquid in your theme code |
| Static sites | The server returns whatever file sits at /robots.txt | Put the file in the folder your host publishes as the site root |
A WooCommerce store usually has no reason to let crawlers spend time on cart, checkout and account pages, which is why stores commonly disallow them. Product, collection and blog pages should stay crawlable.
Editing robots.txt on Shopify
You cannot upload a robots.txt file to Shopify. Its default file already blocks paths such as /admin, /cart, /checkout and /search, and includes your sitemap. To change it, Shopify's instructions are:
- In Shopify admin, go to Online Store > Themes, open the menu on your current theme and click Edit code.
- Click Add a new template, choose robots, then Create template.
- Add your extra rules (for example AI crawler blocks) while keeping Shopify's default Liquid output, and save.
Shopify calls this an unsupported customization that its support team cannot help with, and warns that incorrect edits can cost you all your traffic. Use the generator output as the rules to add, not as a replacement for the whole template.
Blocking AI crawlers
Each toggle adds a User-agent group with Disallow: / for that crawler. Know what each one does before you block it:
| User agent | Operator | What it is used for |
|---|---|---|
| GPTBot | OpenAI | Collecting content that may be used to train its models |
| ClaudeBot | Anthropic | Collecting content for model training |
| CCBot | Common Crawl | A public web archive that many AI datasets are built from |
| Google-Extended | A control token for Gemini training and grounding, not a separate crawler | |
| PerplexityBot | Perplexity | Indexing pages for Perplexity search answers |
| Bytespider | ByteDance | ByteDance's web crawler |
| Applebot-Extended | Apple | A control token for training Apple's generative models |
| meta-externalagent | Meta | Crawling for AI training and related uses |
There is a trade-off. Blocking crawlers that power AI search, such as PerplexityBot, can reduce how often your pages are cited in those answers. Also, robots.txt is voluntary: well-behaved crawlers follow it, but it cannot force a bot to stop.
Google-Extended is not Googlebot
Googlebot crawls pages for Google Search. Google-Extended is a product token that controls whether content Google has crawled can be used to train future Gemini models and for grounding. Google states it does not affect inclusion in Google Search and is not a ranking signal. Blocking Google-Extended is safe for SEO. Blocking Googlebot removes you from Google Search crawling.
What robots.txt does not do
- It controls crawling, not indexing. Google says robots.txt is not a way to keep a page out of search results. A blocked URL can still appear, without a description, if other pages link to it. Use a noindex tag or password protection instead, and do not block the page, or Google never sees the noindex.
- Google ignores crawl-delay. Google does not support the crawl-delay field. Some other crawlers honor it.
- It is not security. The file is public, so listing a secret path only advertises it.
How crawlers read the rules
A few details decide whether your rules do what you expect:
- Location: the file only works at the root of a host. A blog on a subdomain needs its own robots.txt.
- Matching: Google applies the most specific (longest) matching rule, so Allow: /wp-admin/admin-ajax.php wins over Disallow: /wp-admin/.
- Case: paths are case-sensitive. Disallow: /Search does not block /search.
- Wildcards: * matches any characters and $ marks the end of a URL, as in Disallow: /*?add-to-cart=.
- Size: Google reads the first 500 KiB of the file and ignores anything after that.
An AI crawler block in the finished file looks like this:
User-agent: GPTBot
Disallow: /
Mistakes the checker catches
- Disallow: / under User-agent: *. This blocks every compliant crawler from your whole site. It often ships by accident when a staging file goes live.
- Blocking CSS and JavaScript. Google renders pages like a browser. If it cannot load your theme files, it may misread your layout and mobile friendliness.
- A missing or wrong sitemap line. The Sitemap URL must be absolute, including https://.
Tip: After publishing, open yoursite.com/robots.txt in a browser to confirm the live file, then check the robots.txt report in Google Search Console.
Related tasks
Robots.txt is one part of technical SEO. Run through the Shopify SEO audit for the rest, see Shopify SEO basics for how crawling fits with titles and collections, and use the collections SEO guide before you block any filtered collection URLs. If you build niche sites, the AI Site Builder creates complete sites including an llms.txt file for AI tools.