User-agent: * Allow: / # --- AI / LLM crawler policy --- # OpenAI: ChatGPT Search indexing + user-initiated retrieval (required for citations) User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / # OpenAI training-only crawler; blocking does NOT affect ChatGPT Search citations User-agent: GPTBot Disallow: / # Perplexity: index + user-initiated retrieval User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / # Anthropic: user-initiated retrieval / general training crawler User-agent: Claude-Web Allow: / User-agent: Claude-SearchBot Allow: / User-agent: ClaudeBot Disallow: / # Google: search index feeds AI Overview RAG; MUST NOT be blocked User-agent: Googlebot Allow: / # Google-Extended: Gemini training/grounding consent only; does NOT control Googlebot / AIO User-agent: Google-Extended Allow: / # Microsoft Bing: primary index behind ChatGPT Search citations User-agent: Bingbot Allow: / # Common Crawl: open dataset / secondary redistribution, not a citation channel User-agent: CCBot Disallow: / # Apple: generative-training consent only; does not affect Applebot search indexing User-agent: Applebot-Extended Disallow: / # ByteDance: low overlap with this TH-language audience; re-evaluate on CN expansion User-agent: Bytespider Disallow: / Sitemap: https://asphat.com/sitemap.xml