TruvonBot

TruvonBot is Truvon's crawler. Truvon keeps a dated record of U.S. businesses for AI assistants, and TruvonBot reads a business's own website so the record says what the business says, not only what directories copied.

How to recognize it

Every request carries this user agent:

Mozilla/5.0 (compatible; TruvonBot/1.0; +https://truvon.ai/bot)

In robots.txt its name is TruvonBot. It never fills in forms, signs in, buys or books, and it doesn't run a page's scripts.

Anyone can copy a user agent, so TruvonBot also signs every request (Web Bot Auth: HTTP Message Signatures, RFC 9421, Ed25519). Its Signature-Agent is "https://truvon.ai" and its public key is at https://truvon.ai/.well-known/http-message-signatures-directory. A host that checks signatures can let in TruvonBot and nothing that only copies its name.

What it reads

• The home page, robots.txt and the sitemap.
• Files written for AI and agents, if the site has them: llms.txt, an agent card, an MCP or OpenAPI description, a UCP or API catalog, and security.txt.
• Up to four of the site's own pages about what it offers (a menu, services, prices) that the home page links to, and up to two of their PDFs; and when the home page states no hours, the page its links say holds them (hours, location or contact).

It reads at most 600 KB of a page and waits at most 12 seconds for one. A visit is about 17 requests, most of them for small files many sites don't have, and up to seven more for those pages.

How often

When a business is first listed, about once a month after that, and when its owner asks for a check again.

It obeys robots.txt

Truvon's crawl follows robots.txt as RFC 9309 sets out: a group that names TruvonBot is the one it reads, and otherwise the group for every crawler (User-agent: *). A missing robots.txt lets it read everything; one that fails (a server error, or no answer) keeps it out until the next visit. A read of one site that a person asks for, such as an owner listing their own business, is a visit on their behalf, not a crawl.

Let TruvonBot in

These two lines in robots.txt let TruvonBot in, even when the rest of the file keeps other crawlers out:

User-agent: TruvonBot
Allow: /

Where robots.txt lives:

• WordPress. If the site has a robots.txt file of its own, edit it with your host's file manager. Otherwise WordPress writes one, and SEO plugins can change it: Yoast SEO (Tools, then File editor), Rank Math (General Settings, then Edit robots.txt), or All in One SEO (Tools, then Robots.txt Editor).
• Wix. Wix has a robots.txt editor in its SEO settings.
• Squarespace. Squarespace writes robots.txt itself and doesn't let you add lines. If it keeps TruvonBot out, write to us.
• Shopify. Shopify builds robots.txt from a theme template, robots.txt.liquid, which you can add or edit in your theme's code.
• Webflow. Webflow has a robots.txt box in your site's SEO settings.
• Anything else. Your web host, or whoever built your site, can add the two lines to the robots.txt file at the top of your site.

If your site turns bots away at the door

Some sites answer crawlers with an error (401 or 403) or a challenge page before robots.txt is ever read. What usually does it, and how to let TruvonBot through:

• A password or "coming soon" page. An answer of 401 means the site asks for a password. Turn off password protection or the coming-soon mode once the site is ready.
• Cloudflare. Cloudflare's bot protection challenges crawlers it doesn't know. On a paid plan, a custom rule can skip it for TruvonBot: in Security, then WAF, then Custom rules, match a User Agent that contains TruvonBot and choose Skip. On the free plan, Bot Fight Mode can't be skipped by a rule; turning it off lets TruvonBot in.
• WordPress security plugins. Wordfence, Sucuri, Solid Security and others can block crawlers they don't know. Add TruvonBot to the plugin's allowed bots, or turn off its setting for unknown bots.
• Wix and Squarespace. They run their own protection, which their customers can't change. If yours turns TruvonBot away, write to us.

A rule that trusts a user agent lets in anyone who copies it; one that checks TruvonBot's signature lets in only TruvonBot.

Keep TruvonBot out

These two lines keep it out; the business stays listed from public sources, and nothing on its site reaches AI through Truvon:

User-agent: TruvonBot
Disallow: /

To take a business out of Truvon altogether, its owner can write to support@truvon.ai.

Questions

Write to support@truvon.ai. Owners can check how AI sees their business at truvon.ai/check.