SEOBandwagonBot

What it is

SEOBandwagonBot is the name SEO Bandwagon's servers use whenever they read a page on another site. Every such request carries this User-Agent:

Mozilla/5.0 (compatible; SEOBandwagonBot/1.0; +https://seobandwagon.com/bot)

It only fetches pages (GET, and HEAD for image checks). It does not sign in to anything, submit forms or run scripts, and it does not follow more than 4 redirects.

What it fetches, and why

• Link-building research. Our pipeline reads publication pages (articles, author and staff pages, and pages that explain how to contribute) to find the right editor or writer for a story. Before we suggest newer data to the author of a page that cites an older study, the bot re-reads that page once to check it still cites the study.
• Prospect reports. When a signed-in account first asks our API for a business's prospect report, the bot reads that business's homepage once for its structured data (JSON-LD). The result is stored, so the page is not read again. A homepage the bot could not read is shown as “not checked”.
• Checking a search result. A script run by hand reads a page that appears in a business's own search results, once, to decide whether it is the business's website (the listing's name and phone number are looked for on it).
• Maintenance and research. Occasional scripts run by hand, such as checking that the images our blog posts embed still exist, or what the old links a study cites serve today.

How to block it

The bot reads your robots.txt before its first request to your site, and its robots.txt product token is SEOBandwagonBot. To keep it out entirely:

User-agent: SEOBandwagonBot
Disallow: /

A group named for SEOBandwagonBot takes precedence over a group for *; the longest matching rule wins, and Allow wins a tie. The answer is kept for up to 24 hours, so a change to your robots.txt can take that long to reach the bot. A robots.txt that answers 404 (or any other 4xx except 429) counts as no rules. When your robots.txt answers 429 or a server error, times out, or cannot be reached, the bot treats your whole site as off limits until it next reads the file.

How fast it goes

• At least 1 second between any two requests to one host, robots.txt included.
• A Crawl-delay in the group that applies sets the gap between two page requests to your host. The bot waits up to 10 seconds for it; when the gap you ask for would need a longer wait, it skips the request rather than sending it early, and records the page as not checked.
• Each redirect it follows is treated as its own request to that host.

How to verify it is us

Every request to another site is signed with HTTP Message Signatures (RFC 9421) in the Web Bot Auth profile: the signature covers @authority and signature-agent, carries tag="web-bot-auth" and an Ed25519 key id, and expires 60 seconds after it is made. The Signature-Agent header is "https://seobandwagon.com", and the public keys are published at https://seobandwagon.com/.well-known/http-message-signatures-directory.

Verify by signature, not by IP address: the requests come from shared cloud infrastructure whose addresses we do not control, so we publish no IP list. A request with this User-Agent and no valid signature is either not from us or from a development machine, and should be treated as unverified. Requests to seobandwagon.com itself are not signed.

Questions

Email info@seobandwagon.com. Include the URL you saw the bot fetch and the time.