# AI Syndicate · robots.txt # We are a Generative Engine Optimization agency. AI crawlers and training # bots are explicitly welcomed — being cited in AI answers is the goal. # # /api/ is disallowed for every agent below: those routes are backend # endpoints (POST-only handlers), not pages, so crawling them only wastes # crawl budget and returns 4xx. # # Machine-readable companions (read these for richer site signal): # • /llms.txt — brand summary engineered for AI-engine retrieval # • /llms-full.txt — full-text companion: the complete prose of every key page in one file # • /agents.md — action-oriented orientation for autonomous AI agents # • /sitemap.xml — full page inventory (Sitemap: directive below) # ============================================================ # OpenAI (ChatGPT) # ============================================================ User-agent: GPTBot Allow: / Disallow: /api/ User-agent: OAI-SearchBot Allow: / Disallow: /api/ User-agent: ChatGPT-User Allow: / Disallow: /api/ # ============================================================ # Anthropic (Claude) # ============================================================ User-agent: ClaudeBot Allow: / Disallow: /api/ User-agent: anthropic-ai Allow: / Disallow: /api/ User-agent: Claude-Web Allow: / Disallow: /api/ User-agent: Claude-SearchBot Allow: / Disallow: /api/ User-agent: Claude-User Allow: / Disallow: /api/ # ============================================================ # Google (Search, Gemini, AI Overviews) # ============================================================ User-agent: Googlebot Allow: / Disallow: /api/ User-agent: Googlebot-Image Allow: / Disallow: /api/ User-agent: Googlebot-Video Allow: / Disallow: /api/ User-agent: Googlebot-News Allow: / Disallow: /api/ User-agent: Google-Extended Allow: / Disallow: /api/ User-agent: GoogleOther Allow: / Disallow: /api/ User-agent: AdsBot-Google Allow: / Disallow: /api/ User-agent: Mediapartners-Google Allow: / Disallow: /api/ # ============================================================ # Perplexity # ============================================================ User-agent: PerplexityBot Allow: / Disallow: /api/ User-agent: Perplexity-User Allow: / Disallow: /api/ # ============================================================ # Microsoft (Copilot, Bing) # ============================================================ User-agent: bingbot Allow: / Disallow: /api/ User-agent: BingPreview Allow: / Disallow: /api/ User-agent: msnbot Allow: / Disallow: /api/ # ============================================================ # xAI (Grok) # ============================================================ User-agent: xAI-Bot Allow: / Disallow: /api/ User-agent: Grok Allow: / Disallow: /api/ # ============================================================ # Mistral (Le Chat) # ============================================================ User-agent: MistralAI-User Allow: / Disallow: /api/ # ============================================================ # DeepSeek # ============================================================ User-agent: DeepSeekBot Allow: / Disallow: /api/ # ============================================================ # Cohere # ============================================================ User-agent: cohere-ai Allow: / Disallow: /api/ User-agent: cohere-training-data-crawler Allow: / Disallow: /api/ # ============================================================ # Apple (Apple Intelligence) # ============================================================ User-agent: Applebot Allow: / Disallow: /api/ User-agent: Applebot-Extended Allow: / Disallow: /api/ # ============================================================ # Meta (Meta AI / Llama) # ============================================================ User-agent: meta-externalagent Allow: / Disallow: /api/ User-agent: meta-externalfetcher Allow: / Disallow: /api/ User-agent: FacebookBot Allow: / Disallow: /api/ User-agent: facebookexternalhit Allow: / Disallow: /api/ # ============================================================ # ByteDance (Doubao, TikTok) # ============================================================ User-agent: Bytespider Allow: / Disallow: /api/ # ============================================================ # Amazon (Alexa) # ============================================================ User-agent: Amazonbot Allow: / Disallow: /api/ # ============================================================ # DuckDuckGo (DuckAssist) # ============================================================ User-agent: DuckDuckBot Allow: / Disallow: /api/ User-agent: DuckAssistBot Allow: / Disallow: /api/ # ============================================================ # Yandex (Russian search + GenSearch) # ============================================================ User-agent: YandexBot Allow: / Disallow: /api/ User-agent: YandexImages Allow: / Disallow: /api/ # ============================================================ # Baidu (Chinese search + ERNIE) # ============================================================ User-agent: Baiduspider Allow: / Disallow: /api/ # ============================================================ # Naver (Korean) # ============================================================ User-agent: Yeti Allow: / Disallow: /api/ # ============================================================ # Huawei (Petal Search) # ============================================================ User-agent: PetalBot Allow: / Disallow: /api/ # ============================================================ # Kagi (privacy-first search) # ============================================================ User-agent: Kagibot Allow: / Disallow: /api/ # ============================================================ # You.com # ============================================================ User-agent: YouBot Allow: / Disallow: /api/ # ============================================================ # Allen Institute (open-source AI2 / Olmo / Dolma) # ============================================================ User-agent: AI2Bot Allow: / Disallow: /api/ User-agent: AI2Bot-Dolma Allow: / Disallow: /api/ # ============================================================ # Common Crawl (open dataset; many open models train on it) # ============================================================ User-agent: CCBot Allow: / Disallow: /api/ # ============================================================ # Knowledge / structured-data crawlers # ============================================================ User-agent: Diffbot Allow: / Disallow: /api/ User-agent: SemrushBot-OCOB Allow: / Disallow: /api/ # ============================================================ # Wayback Machine # ============================================================ User-agent: ia_archiver Allow: / Disallow: /api/ User-agent: archive.org_bot Allow: / Disallow: /api/ # ============================================================ # Other AI / answer engines # ============================================================ User-agent: Omgilibot Allow: / Disallow: /api/ User-agent: Omgili Allow: / Disallow: /api/ User-agent: Timpibot Allow: / Disallow: /api/ User-agent: ImagesiftBot Allow: / Disallow: /api/ User-agent: iaskspider/2.0 Allow: / Disallow: /api/ # ============================================================ # Default — allow everyone else # ============================================================ User-agent: * Allow: / Disallow: /api/ Sitemap: https://www.aisyndicate.com/sitemap.xml