Free tools

AI Crawler Checker

The AI Crawler Checker is a free tool that reads your site's robots.txt and shows which AI bots, like OpenAI's OAI-SearchBot and GPTBot, may read your pages, which blocks actually hide you from AI answers, and how to fix them on Shopify.

Free · No sign-up · Results in seconds

ExampleA made-up robots.txt that blocks two bots. Edit it to test your own rules.

ExampleThe result for the made-up robots.txt above.

Action needed

1 bot that affects AI answers is blocked

1 training bot is blocked. That only stops AI training.

  • Tip: Add a Sitemap line so bots can find all your pages.

Pages we test your rules on

  • your home page /
  • a product page /products/example
  • a collection page /collections/all

Can recommend or cite you

  • PerplexityBotPerplexity

    Blocked on 1 of 3 pages

    Decides whether your pages can show up in Perplexity's answers.

    Blocked on a product page. Allowed on your home page and a collection page.

    Blocked by line 10: Disallow: /products/ (in the User-agent: PerplexityBot group).

  • GooglebotGoogle

    Allowed

    Google Search's crawler. Google AI Overviews and AI Mode answer from what it reads.

  • OAI-SearchBotOpenAI

    Allowed

    Decides whether your pages can show up in ChatGPT search answers. OpenAI says changes take about 24 hours.

  • BingbotMicrosoft

    Allowed

    Bing's crawler. Microsoft Copilot answers come from Bing, and ChatGPT search also uses Microsoft as a search provider.

  • Claude-SearchBotAnthropic

    Allowed

    Reads pages for Claude's search results. Anthropic says blocking it may reduce your visibility in Claude.

  • Google-ExtendedGoogle

    Allowed

    A robots.txt setting, not a crawler. It lets Gemini use your pages for training and to ground answers in the Gemini app. It doesn't affect Google Search or AI Overviews.

  • ApplebotApple

    Allowed

    Apple's crawler for Siri, Spotlight and Safari, including Apple's AI answers with links. If your file doesn't name Applebot, Apple follows your Googlebot rules.

  • Meta-WebIndexerMeta

    Allowed

    Helps Meta AI cite and link to your pages in its answers.

  • Amzn-SearchBotAmazon

    Allowed

    Lets your pages appear in Amazon search experiences such as Alexa. Amazon says it isn't used for AI training.

  • DuckAssistBotDuckDuckGo

    Allowed

    Reads pages for DuckDuckGo's AI-assisted answers. It doesn't affect normal DuckDuckGo results.

  • Storebot-GoogleGoogle

    Allowed

    Google Shopping's crawler. It affects all Google Shopping surfaces, like the Shopping tab.

  • MistralAI-IndexMistral

    Allowed

    Indexes pages for Mistral's search, which answers questions in Mistral's assistant.

Fetch a page when someone asks

  • ChatGPT-UserOpenAI

    AllowedMay not follow robots.txt

    Fetches a page when someone asks ChatGPT about it. OpenAI says robots.txt rules may not apply to it.

  • Claude-UserAnthropic

    Allowed

    Fetches a page when someone asks Claude about it. Anthropic says it follows robots.txt, so a block stops Claude reading your page for that person.

  • Perplexity-UserPerplexity

    AllowedMay not follow robots.txt

    Fetches a page when someone asks Perplexity about it. Perplexity says it generally ignores robots.txt.

  • meta-externalfetcherMeta

    AllowedMay not follow robots.txt

    Fetches a page when someone asks Meta AI to. Meta says it may bypass robots.txt.

  • Amzn-UserAmazon

    AllowedMay not follow robots.txt

    Fetches a page live when someone asks Alexa a question. Amazon says it may not follow every robots.txt rule.

  • MistralAI-UserMistral

    Allowed

    Fetches a page when someone asks Mistral's assistant to.

Training only: blocking these doesn't hide you from answers

  • GPTBotOpenAI

    Blocked

    Collects pages to train OpenAI's models. Blocking it stops training use, not ChatGPT search.

    Blocked on your home page, a product page and a collection page.

    Blocked by line 7: Disallow: / (in the User-agent: GPTBot group).

  • ClaudeBotAnthropic

    Allowed

    Collects pages to train Anthropic's models. Blocking it stops training use, not Claude's answers.

  • Applebot-ExtendedApple

    Allowed

    A robots.txt setting for training Apple's AI models. Apple says blocking it doesn't remove you from its search results.

  • meta-externalagentMeta

    Allowed

    Collects pages to train Meta's AI models or improve its products. Meta doesn't say blocking it hides you from Meta AI.

  • AmazonbotAmazon

    Allowed

    Collects pages to improve Amazon's products and may train its AI models. Blocking it doesn't block Amzn-SearchBot.

  • CCBotCommon Crawl

    Allowed

    Builds Common Crawl's open web archive, which many AI training sets are built from.

  • MistralAI-TrainingMistral

    Allowed

    Collects pages to train Mistral's models. Mistral says it isn't used for search or live answers.

Fix it on Shopify

In Shopify admin, go to Online Store, then Themes. Next to your live theme, click the three dots and choose Edit code. In the Templates folder, open robots.txt.liquid. Delete the lines shown below, save, then run this check again.

Lines to delete

Line 9: User-agent: PerplexityBot
Line 10: Disallow: /products/

Lines that block training bots aren't listed. Blocking those is your choice.

No robots.txt.liquid file? Then your store uses Shopify's standard file, and the block comes from somewhere else: an app, a developer, or a service in front of your store such as Cloudflare. Ask them to remove it.

Want Shopify's standard rules back? Delete robots.txt.liquid (copy it somewhere first).

Source: Shopify Help Center

What robots.txt can't show

A firewall or bot protection, such as Cloudflare's setting that blocks AI bots, can turn bots away before they read robots.txt. Google also has a switch in Search Console (Settings, then Search generative AI) that can keep a site out of AI Overviews. And some assistants fetch a page when a person asks and say robots.txt may not apply to those visits, including Google-Agent.

Sources: Search Console Help, Google's user-triggered fetchers

GEO is a Shopify app that helps your products get recommended by AI, then counts the orders AI sends you.

Let GEO check this for you every week

What this tool checks

  • Your live robots.txt, read the way the robots.txt standard (RFC 9309) says bots read it
  • 25 bots from OpenAI, Google, Anthropic, Perplexity, Microsoft, Apple, Meta, Amazon, DuckDuckGo, Mistral and Common Crawl
  • Your home page, a product page and a collection page
  • The exact line that blocks each bot, and how to remove it on Shopify

What is GPTBot?

GPTBot is OpenAI's crawler that collects pages to train its AI models. Blocking it in robots.txt tells OpenAI not to use your pages for training. It doesn't change whether ChatGPT search can show your pages: that's OAI-SearchBot. OpenAI publishes GPTBot's addresses at openai.com/gptbot.json, so a firewall can tell the real bot from fakes.

Source: OpenAI's crawler page

What is OAI-SearchBot?

OAI-SearchBot is OpenAI's search crawler. It decides whether your pages can appear in ChatGPT search answers. OpenAI says robots.txt changes take about 24 hours to apply, and asks sites to allow its published addresses (openai.com/searchbot.json) at their firewall or CDN. For a store that wants ChatGPT to recommend it, this is the OpenAI bot that matters most.

Sources: OpenAI's crawler page, OpenAI Help: ChatGPT search

Every bot we check

From each company's own crawler pages, checked 11 October 2026
BotCompanyWhat it does
GooglebotGoogleGoogle Search's crawler. Google AI Overviews and AI Mode answer from what it reads.
OAI-SearchBotOpenAIDecides whether your pages can show up in ChatGPT search answers. OpenAI says changes take about 24 hours.
PerplexityBotPerplexityDecides whether your pages can show up in Perplexity's answers.
BingbotMicrosoftBing's crawler. Microsoft Copilot answers come from Bing, and ChatGPT search also uses Microsoft as a search provider.
Claude-SearchBotAnthropicReads pages for Claude's search results. Anthropic says blocking it may reduce your visibility in Claude.
Google-ExtendedGoogleA robots.txt setting, not a crawler. It lets Gemini use your pages for training and to ground answers in the Gemini app. It doesn't affect Google Search or AI Overviews.
ApplebotAppleApple's crawler for Siri, Spotlight and Safari, including Apple's AI answers with links. If your file doesn't name Applebot, Apple follows your Googlebot rules.
Meta-WebIndexerMetaHelps Meta AI cite and link to your pages in its answers.
Amzn-SearchBotAmazonLets your pages appear in Amazon search experiences such as Alexa. Amazon says it isn't used for AI training.
DuckAssistBotDuckDuckGoReads pages for DuckDuckGo's AI-assisted answers. It doesn't affect normal DuckDuckGo results.
Storebot-GoogleGoogleGoogle Shopping's crawler. It affects all Google Shopping surfaces, like the Shopping tab.
MistralAI-IndexMistralIndexes pages for Mistral's search, which answers questions in Mistral's assistant.
ChatGPT-UserOpenAIFetches a page when someone asks ChatGPT about it. OpenAI says robots.txt rules may not apply to it.
Claude-UserAnthropicFetches a page when someone asks Claude about it. Anthropic says it follows robots.txt, so a block stops Claude reading your page for that person.
Perplexity-UserPerplexityFetches a page when someone asks Perplexity about it. Perplexity says it generally ignores robots.txt.
meta-externalfetcherMetaFetches a page when someone asks Meta AI to. Meta says it may bypass robots.txt.
Amzn-UserAmazonFetches a page live when someone asks Alexa a question. Amazon says it may not follow every robots.txt rule.
MistralAI-UserMistralFetches a page when someone asks Mistral's assistant to.
GPTBotOpenAICollects pages to train OpenAI's models. Blocking it stops training use, not ChatGPT search.
ClaudeBotAnthropicCollects pages to train Anthropic's models. Blocking it stops training use, not Claude's answers.
Applebot-ExtendedAppleA robots.txt setting for training Apple's AI models. Apple says blocking it doesn't remove you from its search results.
meta-externalagentMetaCollects pages to train Meta's AI models or improve its products. Meta doesn't say blocking it hides you from Meta AI.
AmazonbotAmazonCollects pages to improve Amazon's products and may train its AI models. Blocking it doesn't block Amzn-SearchBot.
CCBotCommon CrawlBuilds Common Crawl's open web archive, which many AI training sets are built from.
MistralAI-TrainingMistralCollects pages to train Mistral's models. Mistral says it isn't used for search or live answers.

How this tool works

  • We find your home page first (yourstore.com often sends visitors to www.yourstore.com), then fetch /robots.txt from that address, following up to 4 redirects, and read up to 512 KB of it. Google reads the first 500 KB. Source: Google's robots.txt rules
  • We read it the way the robots.txt standard (RFC 9309) describes: a group that names a bot replaces the general * group for that bot, and the longest matching rule wins. Allow wins a tie. Source: RFC 9309
  • Apple says Applebot follows your Googlebot rules when your file doesn't name Applebot, so we do the same. Source: Apple's crawler page
  • We test your home page, one product page and one collection page we find on your home page, plus the page you entered.
  • We don't visit your site pretending to be each bot, so firewall or CDN blocks don't show here.
  • If your site answers 401, 403 or 429, we say it turned us away, and never call that all clear. If there's no robots.txt (404), every bot is allowed.
  • The bot list comes from each company's own crawler pages, checked 11 October 2026. We review it every 3 months.

Maintained by GEO. Updated .

Questions people ask

Does blocking GPTBot hide my store from ChatGPT?

No. GPTBot only collects pages to train OpenAI's models. Whether your pages can appear in ChatGPT search is decided by a different bot, OAI-SearchBot. OpenAI says sites that block OAI-SearchBot won't be shown in ChatGPT search answers, though they can still appear as plain navigational links.

Source: OpenAI's crawler page

Should I block AI training bots?

It's your choice. Blocking training bots like GPTBot or ClaudeBot stops those companies using your pages to train future models, and their docs say it doesn't remove you from their live answers. Google-Extended is different: it also controls whether the Gemini app can use your pages to ground its answers. If you want AI assistants to know your products, leaving them open is the simpler choice.

Sources: OpenAI's crawler page, Anthropic's crawler page, Google's crawler page

What does Shopify's robots.txt block by default?

Shopify's standard robots.txt blocks no AI bot. It keeps all bots out of pages like the cart, checkout and customer accounts, and lets them read products, collections, pages and blogs. Blocks usually come from edits to the robots.txt.liquid file, an app, or a service in front of the store.

Sources: Shopify Help Center, Shopify's demo store robots.txt

How do I edit robots.txt on Shopify?

In Shopify admin, go to Online Store, then Themes. Next to your live theme, click the three dots and choose Edit code. In the Templates folder, open robots.txt.liquid, delete the lines that block the bot, save, and run this check again. If there's no robots.txt.liquid, your store uses Shopify's standard file.

Sources: Shopify Help Center, Shopify's robots.txt.liquid docs

My robots.txt allows every bot. Can AI still be blocked?

Yes. A firewall or bot protection, such as Cloudflare's setting that blocks AI bots, can turn bots away before they read robots.txt, and that doesn't show here. Google also has a separate switch in Search Console, called Search generative AI, that can keep a site out of AI Overviews. OpenAI and Perplexity ask sites to allow their published IP addresses at the firewall.

Sources: Search Console Help, OpenAI Help: ChatGPT search, Perplexity's crawler page

Do AI bots follow robots.txt?

The crawlers that build search indexes say they do, including OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot and Applebot. Some fetchers that open a page because a person asked say robots.txt may not apply to them, such as ChatGPT-User and Perplexity-User. Anthropic says its Claude-User fetcher does follow robots.txt.

Sources: OpenAI's crawler page, Perplexity's crawler page, Anthropic's crawler page

More free tools

  • llms.txt Checker

    Check your llms.txt against the llmstxt.org format and copy a starter file.

  • Product Schema Checker

    See which product facts Google and AI shopping tools can read from your product page.

  • AI Visibility Checker

    See if ChatGPT, Gemini and Perplexity recommend your product, and who they pick instead.

See all free tools

Updated · Made by GEO