About BlasterBot
Blaster looks across a sea of vendors and finds the lowest prices for specific products, like ammunition or graphics cards. The system will allow its users to set up a watch for a given product or class of products and will notify them if it falls below their trigger price. Users are sent directly to a vendor's product pages ready to purchase. Everyone is happy. :)
What it fetches
Product pages, the listings and sitemaps that lead to them, and robots.txt. It reads each product's name, price, stock and identifiers. It doesn't log in, fill in forms, add to carts or check out.
How it identifies itself
Its user agent is BlasterBot/0.1.0 (+https://blaster.bot/about-the-bot).
Its requests can be signed with
Web Bot Auth
(HTTP Message Signatures, RFC 9421, Ed25519), covering the
request's host and path. The Signature-Agent
header names https://blaster.bot, and
its public keys are at
https://blaster.bot/.well-known/http-message-signatures-directory
.
A signed request that doesn't verify against that key isn't ours.
How it behaves
-
It follows your robots.txt: the groups for
User-agent: BlasterBot, or with none, those forUser-agent: *. It doesn't fetch what they disallow, and it reads the sitemaps listed. -
It starts requests to a site at least a second apart, and no closer
than your
Crawl-delay, with at most two in flight. -
A 429 or 5xx holds that page back for as long as
Retry-Afterasks, or a growing wait without one; more than a few slow the whole site down. - A run of 403s or refusals stops it fetching from the site for a while.
-
It asks again with
If-None-MatchandIf-Modified-Since, so an unchanged page costs you a 304.
Contact
To ask it to slow down, stop, or fetch from a feed instead, write to bot@blaster.bot.