How businesses block AI by accident
One of the most common problems we find on local business websites is a setting that blocks AI assistants from reading the site. Usually nobody meant to turn it on.
Over the last two years, many web hosts, WordPress security plugins and Cloudflare added one-click "Block AI bots" switches. That can make sense for a news site protecting paid articles. For a local service business, it hides your services, service areas and phone number from every customer who asks ChatGPT or Perplexity who to hire.
The rules live in a small file on your website called robots.txt. In TUSKOS we call it your AI access settings.
The six AI assistants we check
Every TUSKOS audit reads your AI access settings and checks six AI visitors. Your web person may know them by the names in brackets:
- ChatGPT, learning (
GPTBot): how OpenAI reads the web to improve ChatGPT. - ChatGPT search (
OAI-SearchBot): how ChatGPT finds businesses and links when it answers. - Google Gemini (
Google-Extended): controls whether Gemini can use your site. It's separate from normal Google Search. - Perplexity (
PerplexityBot): how Perplexity finds the pages it links to. - Claude (
ClaudeBot): Anthropic's assistant. - Apple Intelligence (
Applebot-Extended): Apple's AI features.
Safe settings to hand your web person
For most local businesses that want to be recommended by AI, while keeping admin and checkout pages private, the settings look like this. You can copy it and send it to whoever looks after your website:
User-agent: *
Allow: /
Disallow: /wp-admin/
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Applebot-Extended
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml
Other things that can block AI
Even with the right settings, AI can still be turned away if your website shows it:
- A security check or "Just a moment..." page meant to stop bots.
- A cookie banner that covers everything until someone clicks Accept.
- A nearly empty page that only fills in once a browser runs extra code.
During a TUSKOS audit we visit your pages the way an AI assistant would and tell you if any of these got in the way, so you know straight away whether AI can actually read your site.
How to check the fix worked
- Open
https://yourdomain.com/robots.txtin a private browser window. Make sure none of the six names above hasDisallow: /under it. - Ask your web person to check your security settings (for example in Cloudflare, Sucuri or Wordfence) so trusted AI visitors are let through.
- Run another audit in TUSKOS. On Pro and Agency, TUSKOS also checks every day and warns you if AI gets blocked again.