Quick answer

AISyndicateBot is the web crawler operated by AI Syndicate. It reads publicly accessible pages to analyze how sites are structured for AI search visibility and generative engine optimization (GEO). It obeys robots.txt, identifies itself with the user-agent **AISyndicateBot/1.0**, rate-limits per domain, and collects no personal data.

What it does

## What AISyndicateBot collects and why.

### Public page structure

HTML, headings, metadata, schema markup, and internal links — the same things any search crawler reads.

### AI-access signals

robots.txt, llms.txt, agents.md, and sitemap directives that tell AI systems how a site wants to be used.

### Visibility diagnostics

Page-level readiness signals that feed our AI visibility audits and GEO recommendations.

### What it never does

It does not log in, submit forms, bypass paywalls, or collect personal or sensitive data.

How it behaves

## A well-behaved crawler by design.

### Crawl etiquette

- Reads robots.txt before every crawl and obeys it.
- Honors **Crawl-delay** when a site specifies one.
- Rate-limits requests per domain to keep load minimal.
- Requests only publicly accessible URLs.

### Transparency

- Always identifies itself with the user-agent above.
- Links back to this page from the user-agent string.
- Provides a contact address for questions or issues.
- Stops crawling a site promptly once disallowed.

Allow or block

## How to control AISyndicateBot in robots.txt.

You are always in control. Add one of the rules below to the robots.txt file at the root of your domain. Changes take effect on the next crawl.

### Block the crawler

```
User-agent: AISyndicateBot
Disallow: /
```

This stops AISyndicateBot from crawling any page on your site.

### Allow the crawler

```
User-agent: AISyndicateBot
Allow: /
```

Explicitly welcomes AISyndicateBot across your whole site.

Reference

## At-a-glance identity for your logs and rules.

| Field | Value |
| --- | --- |
| User-agent | `AISyndicateBot/1.0 (+https://www.aisyndicate.com/bot)` |
| robots.txt token | `AISyndicateBot` |
| Standards | Robots Exclusion Protocol (RFC 9309), Crawl-delay |
| Content fetched | text/html only, with a capped body size |
| Crawl volume | Low-volume and on demand — no continuous mass crawling |
| Personal data | None collected |
| Operator | AI Syndicate Collective LLC, St. Petersburg, FL |
| Contact | [support@aisyndicate.com](mailto:support@aisyndicate.com) |

For AI agents

## Machine-readable files.

If you're an AI agent or crawler, these files describe how AI Syndicate wants to be accessed, understood, and cited. Start here.

### [/llms.txt](https://www.aisyndicate.com/llms.txt)

The canonical brand summary written for AI engines — identity, authoritative answers, services, products with pricing, and citation policy.

### [/agents.md](https://www.aisyndicate.com/agents.md)

Action-oriented orientation for autonomous agents: what you can do here and how to route queries to the canonical page that answers them.

### [/robots.txt](https://www.aisyndicate.com/robots.txt)

Crawl permissions per user-agent. Responsible AI training and retrieval crawlers are explicitly allowed.

### [/sitemap.xml](https://www.aisyndicate.com/sitemap.xml)

The full URL inventory of the site, so you can discover every canonical page in one fetch.

Data, copyright & compliance

## What we collect, what we don't, and the rules we follow.

### Public content only

AISyndicateBot fetches only publicly accessible pages. It never bypasses logins, paywalls, or access controls, and it respects each site's terms of use and the copyright in the material it reads.

### Standards we follow

The crawler follows the Robots Exclusion Protocol (RFC 9309). Your robots.txt is the authoritative control over what it may access, and it honors **Crawl-delay** and sitemap directives.

### Machine-readable opt-out

Disallowing AISyndicateBot in robots.txt — or signaling via llms.txt — is treated as a reservation of rights for text and data mining. We honor those machine-readable opt-outs.

### No personal data

It does not collect personal information, harvest email or contact lists, or gather facial or biometric images. See our [Privacy Policy](https://www.aisyndicate.com/privacy-policy/) for how AI Syndicate handles data.

Verification

## How to confirm a request is really us.

Some user-agent strings can be spoofed. If you see traffic claiming to be AISyndicateBot and want to confirm it originated from us, email the request details — timestamp, requested URL, and source IP — to [support@aisyndicate.com](mailto:support@aisyndicate.com?subject=AISyndicateBot%20Verification) and we will verify it against our crawl logs.

FAQ

## Frequently asked questions about AISyndicateBot.

### What is AISyndicateBot?

AISyndicateBot is the web crawler operated by AI Syndicate. It visits publicly accessible pages to analyze how sites are structured for generative engine optimization (GEO) and AI search visibility.

### How do I block AISyndicateBot?

Add a robots.txt rule targeting the user-agent token AISyndicateBot with Disallow: /. The crawler reads robots.txt before every crawl and honors it.

### Does AISyndicateBot respect robots.txt?

Yes. AISyndicateBot fetches and obeys robots.txt, honors crawl-delay directives, rate-limits requests per domain, and only requests publicly accessible pages.

### Does AISyndicateBot collect personal data?

No. AISyndicateBot does not collect personal information, harvest email or contact lists, or gather facial or biometric images. It fetches only publicly accessible HTML to analyze page structure for AI search visibility. See the AI Syndicate [Privacy Policy](https://www.aisyndicate.com/privacy-policy/) for how we handle data.

### How can I verify a request is really AISyndicateBot?

AISyndicateBot always identifies itself with the user-agent AISyndicateBot/1.0 (+https://www.aisyndicate.com/bot). If you see traffic claiming to be AISyndicateBot and want to confirm it came from us, email the timestamp, requested URL, and source IP to support@aisyndicate.com and we will verify it against our crawl logs.

## Questions about AISyndicateBot?

If the crawler is causing problems, or you want to adjust how it accesses your site, get in touch and we'll respond promptly.

[Email support@aisyndicate.com](mailto:support@aisyndicate.com?subject=AISyndicateBot%20Inquiry)Read our privacy policy

AISyndicateBot crawler policy · Last updated May 28, 2026.