Crawlers
How Vorne reads websites
VorneBot and VorneAuditBot visit a site only when someone asks Vorne to check it. This page explains what they read and how to stop them.
When they visit
VorneBot: website setup
When a Vorne customer sets up or updates their workspace from their own website, VorneBot reads the homepage and up to eight pages that describe the business, such as about, pricing, delivery and FAQ pages.
VorneBot: pages added by address
When a customer adds a web page to their evidence library by its address, VorneBot reads that one page.
VorneBot: Shopify product pages
When a store owner has connected their Shopify store, VorneBot reads up to three of its product pages to check their structured data.
VorneAuditBot: audits
When someone runs Vorne's free audit on a website, VorneAuditBot reads its homepage. When a customer checks their own site's eligibility for Google's AI features, it reads the homepage and a few pages it links to.
How they behave
robots.txt comes first
Both crawlers read robots.txt before anything else and follow RFC 9309. They skip what it disallows and check the rules again after a redirect. If robots.txt cannot be read because of a server error or timeout, they read nothing.
Crawl-delay is honoured
Website setup waits out Crawl-delay between requests and reads fewer pages rather than reading faster. The other checks make only a few requests, one after another.
Public pages only
No logins, no forms, and no JavaScript. Private and internal network addresses are refused.
Small by design
Website setup reads at most nine pages, two at a time, and each Vorne workspace can run it at most six times an hour.
What happens to the text
Text read for a customer is shown to that customer to review, and may be processed by the AI providers listed in our subprocessors. Vorne does not use it to train AI models.
How to recognise and block them
They identify themselves with these user agents:
VorneBot/1.0 (+https://vorne.ai/bot) VorneAuditBot/1.0 (+https://vorne.ai/bot)
To stop either one, add these lines to your robots.txt. The change applies from their next visit.
User-agent: VorneBot Disallow: / User-agent: VorneAuditBot Disallow: /
We do not publish a fixed list of IP addresses. To reach us about our crawlers, contact us. See also our subprocessors and security practices.