The Datestone crawler

Datestone keeps track of the legal dates on a rented home in England, so a landlord is told what is due before it lapses. One of those things is property licensing, which every council sets for itself, so we read what councils publish about it.

We are a small operation. The crawler runs on one computer in the UK, one page at a time.

Published pages on council websites about property licensing for private landlords, which usually means selective licensing, additional licensing, and mandatory HMO licensing.

Finding that page is a ladder, and it is worth setting out in order, because most of it is not a walk through your navigation. We ask for your robots.txt. Then your sitemap, if you publish one, and we may read a page straight out of it. Then a short fixed list of 7 paths that councils commonly use for this — most councils do not have most of them, so most of those come back 404, and a 404 tells us nothing about you that we record. Then your A-Z index, if you have one. Only then do we load your homepage and follow your own links, 3 clicks in at the most. We stop at whichever rung answers first, so on a council that publishes a good sitemap we never reach the later ones at all.

We stay on your own domain throughout. We read nothing that needs a login, fill in no forms, and submit nothing.

So that a landlord who tells us where a property is can be told which licences that council requires, with a link to the council's own page as the source.

We do not republish your pages. We keep the page we read, so a person here can check an answer against what the page actually said, and everything we show a landlord links back to your page rather than replacing it. A person, not the crawler, decides what we publish about a council.

• It identifies itself on every request: Mozilla/5.0 (compatible; DatestoneBot/1.0; +https://www.datestone.io/crawler)
• It fetches at most 18 pages from one council in a visit, plus your robots.txt, and usually far fewer. It waits 45 to 120 seconds between pages it reads, and 15 to 35 seconds between cheap requests such as your sitemap.
• Those are PAGES, not lines in your access log. It reads them in a real browser, so each page also loads that page's own stylesheets, scripts, fonts and images, exactly as a visitor's browser would — and because the name is set on the browser itself, you will see those under the same name too. So the honest expectation for one visit is a few hundred log lines, not 18. We say so because the arithmetic is the first thing anyone checks, and a webmaster who does it should not conclude we are lying.
• When we find what we came for, we leave that council alone for at least 14 days. A visit that finds nothing is a different case and we will not pretend otherwise: we may come back sooner to try again, though never within 6 hours, and a council whose licensing page we simply cannot find steps out of the queue for 30 days. If you turn us away, or your robots.txt tells us not to read a path, those have their own longer pauses below.
• Never two requests at once, to your site or to anyone else's.
• The nightly check described below is quicker than the reader, because it fetches one page it already knows about rather than looking for anything: it is never faster than one request every 1.5 seconds to your site, however many of your pages we have cited, and it takes your pages in turn with everybody else's rather than in a run.
• Once we have found the page we were looking for, we come back to check it for changes, not to crawl the site again.

We fetch your robots.txt before we read anything, we keep it for a day, and we obey it. If it disallows a path, we do not fetch that path. If you set a Crawl-delay, we wait that long between requests.

That one line covers everything we run, not just the reader described above. The nightly check that tells us whether a page we have cited is still where we cited it, the change monitor that notices when you republish a policy, and the fetch that confirms a quotation still appears on your page all identify themselves as DatestoneBot and all ask your robots.txt first.

To keep us off your site entirely, this is enough:

User-agent: DatestoneBot
Disallow: /

One thing worth knowing before you do. We are not indexing you and we are not republishing your pages: what we do with them is notice when they change, so that a landlord reading a rule on our site is reading what your page says today rather than what it said last year. If you block us, we stop checking, and we say on the page that we cannot confirm the position for your area rather than guessing at it. Nothing is held against you, and you can tell us to start again whenever you like.

• It never tries to get past a refusal. A 403, a 429, or a challenge page ends the visit, and repeated refusals stop us coming back at all.
• It never attempts a CAPTCHA.
• It never signs in, and it only reads pages any member of the public can open.
• It is not looking for personal data, and it does not go looking for people.
• It does not use search engines to find your pages, and never has.

Write to us through the contact form, say which council you are from, and we will take you out. A person reads it and replies by email.

The same goes if we are reading the wrong page. If there is a better page for landlords than the one we found, tell us and we will use that one instead. That helps your residents more than it helps us.