Our crawler, GetTheJerseyBot
GetTheJerseyBot reads the public listings of shops that sell vintage football shirts, so Get the Jersey can compare their item prices. This page explains what it fetches, how fast, and how to stop it.
It names itself in every request with this user agent:GetTheJerseyBot/0.2 (+https://getthejersey.com/bot; hhtb18@gmail.com)
What it collects
From public product pages and shop feeds: title, item price, size, availability and the link to the product page. Season, kit and condition are read from the title and description by machine. We do not show your photos or your description texts.
It never logs in, never gets past a paywall, never puts anything in a cart and never reads customer data.
Rate limit
At most one request per second per shop, and slower for shops on a shared shop platform. A Crawl-delay for GetTheJerseyBot in your robots.txt slows it down further. On HTTP 429 or a server error it waits and tries again (honouring Retry-After); if your shop keeps answering 429, the run stops.
robots.txt
Before it fetches anything, it reads your robots.txt and follows the rules for GetTheJerseyBot, or for all user agents (*) if there are none for it. If your robots.txt cannot be read, or answers 401, 403, 429 or a server error, it fetches nothing from your shop. To block it completely:
User-agent: GetTheJerseyBot
Disallow: /The block applies from the next run. To take your shop off the site straight away, use the form Remove your shop.
Contact
Questions about the crawler, a problem it caused, or a request to slow it down: write to us through the contact form. We read every message.