AISyndicateBot is the web crawler operated by AI Syndicate. It reads publicly accessible pages to analyze how sites are structured for AI search visibility and generative engine optimization (GEO). It obeys robots.txt, identifies itself with the user-agent AISyndicateBot/1.0, rate-limits per domain, and collects no personal data.
What AISyndicateBot collects and why.
Public page structure
HTML, headings, metadata, schema markup, and internal links — the same things any search crawler reads.
AI-access signals
robots.txt, llms.txt, agents.md, and sitemap directives that tell AI systems how a site wants to be used.
Visibility diagnostics
Page-level readiness signals that feed our AI visibility audits and GEO recommendations.
What it never does
It does not log in, submit forms, bypass paywalls, or collect personal or sensitive data.
A well-behaved crawler by design.
Crawl etiquette
- Reads robots.txt before every crawl and obeys it.
- Honors Crawl-delay when a site specifies one.
- Rate-limits requests per domain to keep load minimal.
- Requests only publicly accessible URLs.
Transparency
- Always identifies itself with the user-agent above.
- Links back to this page from the user-agent string.
- Provides a contact address for questions or issues.
- Stops crawling a site promptly once disallowed.
How to control AISyndicateBot in robots.txt.
You are always in control. Add one of the rules below to the robots.txt file at the root of your domain. Changes take effect on the next crawl.
Block the crawler
User-agent: AISyndicateBot
Disallow: /This stops AISyndicateBot from crawling any page on your site.
Allow the crawler
User-agent: AISyndicateBot
Allow: /Explicitly welcomes AISyndicateBot across your whole site.
At-a-glance identity for your logs and rules.
Machine-readable files.
If you're an AI agent or crawler, these files describe how AI Syndicate wants to be accessed, understood, and cited. Start here.
/llms.txt
The canonical brand summary written for AI engines — identity, authoritative answers, services, products with pricing, and citation policy.
/agents.md
Action-oriented orientation for autonomous agents: what you can do here and how to route queries to the canonical page that answers them.
/robots.txt
Crawl permissions per user-agent. Responsible AI training and retrieval crawlers are explicitly allowed.
/sitemap.xml
The full URL inventory of the site, so you can discover every canonical page in one fetch.
What we collect, what we don't, and the rules we follow.
Public content only
AISyndicateBot fetches only publicly accessible pages. It never bypasses logins, paywalls, or access controls, and it respects each site's terms of use and the copyright in the material it reads.
Standards we follow
The crawler follows the Robots Exclusion Protocol (RFC 9309). Your robots.txt is the authoritative control over what it may access, and it honors Crawl-delay and sitemap directives.
Machine-readable opt-out
Disallowing AISyndicateBot in robots.txt — or signaling via llms.txt — is treated as a reservation of rights for text and data mining. We honor those machine-readable opt-outs.
No personal data
It does not collect personal information, harvest email or contact lists, or gather facial or biometric images. See our Privacy Policy for how AI Syndicate handles data.
How to confirm a request is really us.
Some user-agent strings can be spoofed. If you see traffic claiming to be AISyndicateBot and want to confirm it originated from us, email the request details — timestamp, requested URL, and source IP — to support@aisyndicate.com and we will verify it against our crawl logs.
Frequently asked questions about AISyndicateBot.
What is AISyndicateBot?
AISyndicateBot is the web crawler operated by AI Syndicate. It visits publicly accessible pages to analyze how sites are structured for generative engine optimization (GEO) and AI search visibility.
How do I block AISyndicateBot?
Add a robots.txt rule targeting the user-agent token AISyndicateBot with Disallow: /. The crawler reads robots.txt before every crawl and honors it.
Does AISyndicateBot respect robots.txt?
Yes. AISyndicateBot fetches and obeys robots.txt, honors crawl-delay directives, rate-limits requests per domain, and only requests publicly accessible pages.
Does AISyndicateBot collect personal data?
No. AISyndicateBot does not collect personal information, harvest email or contact lists, or gather facial or biometric images. It fetches only publicly accessible HTML to analyze page structure for AI search visibility. See the AI Syndicate Privacy Policy for how we handle data.
How can I verify a request is really AISyndicateBot?
AISyndicateBot always identifies itself with the user-agent AISyndicateBot/1.0 (+https://www.aisyndicate.com/bot). If you see traffic claiming to be AISyndicateBot and want to confirm it came from us, email the timestamp, requested URL, and source IP to support@aisyndicate.com and we will verify it against our crawl logs.
Questions about AISyndicateBot?
If the crawler is causing problems, or you want to adjust how it accesses your site, get in touch and we'll respond promptly.
AISyndicateBot crawler policy · Last updated May 28, 2026.