When Claude or ChatGPT can't reach your site
Why bot protection, a firewall or a security plugin can turn your AI away, how to let it through at each, and how to check it worked.
Last checked .
Why would my AI be turned away?
Say Dana asks Claude about her S'mores site, and Claude says it can't reach it. The site is up and loads fine for her. What's stopping Claude is the protection the site already has against bad bots: it can't tell a helpful assistant from a harmful one, so it turns both away.
Often it does this quietly. The assistant gets an empty answer or a "checking your browser" page. Nothing looks broken from your side, so it's easy to miss.
Why does an assistant look like a bot?
When you browse your site, your browser runs the page, keeps cookies, and can pass a "checking your browser" test. An assistant does none of that. It calls one address on your site straight from a server in a data center, with no cookies and no page code, and reads the answer as data. That's why a challenge page stops it cold.
Two kinds of caller can be turned away:
- A named assistant, like ChatGPT's agent or Claude, acting for a person right now. Your protection can recognize these by name.
- Your site manager, calling from its own servers. It asks for one address on your WordPress site: /wp-json/superintelligentfoo/. That's the address to let through for it.
Every step below lets that named assistant, or that one address, through. None of them turns your protection off for the whole site.
How do I let it through on Cloudflare?
Cloudflare has a setting made for named assistants. Start there.
- Let assistants in by name: go to "Security", then "Settings", filter by "Bot traffic", and open "AI Crawler and Agent access policies". Set "Agents" to "Allow (do not block)". Under "Personal Agents" you can set each one; ChatGPT's agent shows as "dots, OpenAI". (Older dashboards call this "Configure AI bot policies".)
- On Cloudflare's free plan, Bot Fight Mode can't be skipped for one address. If Cloudflare's "Security Events" shows Bot Fight Mode is what stops your site manager, use Talk to Foo at the top of this page and we'll work out the next step with you.
- On a paid plan, let your site manager through with a custom rule for its address and nothing wider: in "Security", then "WAF", then "Custom rules", match "URI Path starts with /wp-json/superintelligentfoo/", choose "Skip", and tick only "All Super Bot Fight Mode rules" and "All rate limiting rules". Leave "All managed rules" unticked, so Cloudflare's firewall still checks every request to that address.
- On Cloudflare's Enterprise plan, your Cloudflare contact can also set a rule for one assistant alone.
What if my host has its own bot protection?
Many hosts add their own layer, called something like bot protection, anti-bot or security challenge. You usually can't see it in your dashboard. Write to your host's support and say: "An AI tool I use calls https://yoursite.com/wp-json/superintelligentfoo/ from cloud servers, and it's being blocked or challenged. Please let that address through without a challenge." That one sentence is usually enough.
What about WordPress security plugins?
- Wordfence: in Firewall, then All Firewall Options, then Advanced Firewall Options, add /wp-json/superintelligentfoo/ to "Allowlisted URLs". Then open "Live Traffic" and check your site manager's calls show as allowed.
- Solid Security, All In One Security and others: look for a setting that blocks the REST API (every /wp-json/ address) for visitors who aren't logged in. Your site manager is never logged in that way, so it's blocked too. Most of these settings can only block all of it or none of it; if yours can allow one address, allow /wp-json/superintelligentfoo/.
- A plugin with a rate limit ("too many requests"): your site manager may ask several things in a row. Allow /wp-json/superintelligentfoo/, or raise the limit for it.
- Sucuri or any other firewall in front of your site: find its allow list and add /wp-json/superintelligentfoo/.
How do I check it worked?
- Ask your assistant the same question again. If it answers with your site's real details, you're through.
- Still failing? Look at what your protection logged when you asked: Cloudflare's Security Events, Wordfence's Live Traffic, or your host's support. Search for the address /wp-json/superintelligentfoo/, and for the caller's name: your site manager's own requests say "Super Intelligent Foo Connect" or "python-httpx"; assistants say ChatGPT-User or Claude-User when you asked them, and OAI-SearchBot, Claude-SearchBot or PerplexityBot when a search tool was reading. The block shows there with its reason, and that tells you which setting to change.
Your protection is doing its job. It just can't tell your assistant from a bad bot, so let your assistant, or your site manager's address, through and ask again.
