# TruvonBot

TruvonBot is Truvon's crawler. Truvon keeps a dated record of U.S. businesses for AI assistants, and TruvonBot reads a business's own website so the record says what the business says, not only what directories copied.

## How to recognize it

Every request carries this user agent:
```
Mozilla/5.0 (compatible; TruvonBot/1.0; +https://truvon.ai/bot)
```

In robots.txt its name is `TruvonBot`. It never fills in forms, signs in, buys or books, and it doesn't run a page's scripts.

Anyone can copy a user agent, so TruvonBot also signs every request (Web Bot Auth: HTTP Message Signatures, RFC 9421, Ed25519). Its `Signature-Agent` is `"https://truvon.ai"` and its public key is at [https://truvon.ai/.well-known/http-message-signatures-directory](https://truvon.ai/.well-known/http-message-signatures-directory). A host that checks signatures can let in TruvonBot and nothing that only copies its name.

## What it reads

- The home page, `robots.txt` and the sitemap.
- Files written for AI and agents, if the site has them: `llms.txt`, an agent card, an MCP or OpenAPI description, a UCP or API catalog, and `security.txt`.
- Up to four of the site's own pages about what it offers (a menu, services, prices) that the home page links to, and up to two of their PDFs; and when the home page states no hours, the page its links say holds them (hours, location or contact).

It reads at most 600 KB of a page and waits at most 12 seconds for one. A visit is about 17 requests, most of them for small files many sites don't have, and up to seven more for those pages.

## How often

When a business is first listed, about once a month after that, and when its owner asks for a check again.

## It obeys robots.txt

Truvon's crawl follows robots.txt as RFC 9309 sets out: a group that names TruvonBot is the one it reads, and otherwise the group for every crawler (`User-agent: *`). A missing robots.txt lets it read everything; one that fails (a server error, or no answer) keeps it out until the next visit. A read of one site that a person asks for, such as an owner listing their own business, is a visit on their behalf, not a crawl.

## Let TruvonBot in

These two lines in robots.txt let TruvonBot in, even when the rest of the file keeps other crawlers out:
```
User-agent: TruvonBot
Allow: /
```

Where robots.txt lives:
- **WordPress.** If the site has a robots.txt file of its own, edit it with your host's file manager. Otherwise WordPress writes one, and SEO plugins can change it: Yoast SEO (Tools, then File editor), Rank Math (General Settings, then Edit robots.txt), or All in One SEO (Tools, then Robots.txt Editor).
- **Wix.** Wix has a robots.txt editor in its SEO settings.
- **Squarespace.** Squarespace writes robots.txt itself and doesn't let you add lines. If it keeps TruvonBot out, write to us.
- **Shopify.** Shopify builds robots.txt from a theme template, `robots.txt.liquid`, which you can add or edit in your theme's code.
- **Webflow.** Webflow has a robots.txt box in your site's SEO settings.
- **Anything else.** Your web host, or whoever built your site, can add the two lines to the robots.txt file at the top of your site.

## If your site turns bots away at the door

Some sites answer crawlers with an error (401 or 403) or a challenge page before robots.txt is ever read. What usually does it, and how to let TruvonBot through:
- **A password or "coming soon" page.** An answer of 401 means the site asks for a password. Turn off password protection or the coming-soon mode once the site is ready.
- **Cloudflare.** Cloudflare's bot protection challenges crawlers it doesn't know. On a paid plan, a custom rule can skip it for TruvonBot: in Security, then WAF, then Custom rules, match a User Agent that contains `TruvonBot` and choose Skip. On the free plan, Bot Fight Mode can't be skipped by a rule; turning it off lets TruvonBot in.
- **WordPress security plugins.** Wordfence, Sucuri, Solid Security and others can block crawlers they don't know. Add TruvonBot to the plugin's allowed bots, or turn off its setting for unknown bots.
- **Wix and Squarespace.** They run their own protection, which their customers can't change. If yours turns TruvonBot away, write to us.

A rule that trusts a user agent lets in anyone who copies it; one that checks TruvonBot's signature lets in only TruvonBot.

## Keep TruvonBot out

These two lines keep it out; the business stays listed from public sources, and nothing on its site reaches AI through Truvon:
```
User-agent: TruvonBot
Disallow: /
```

To take a business out of Truvon altogether, its owner can write to [support@truvon.ai](mailto:support@truvon.ai).

## Questions

Write to [support@truvon.ai](mailto:support@truvon.ai). Owners can check how AI sees their business at [truvon.ai/check](/check).