Scraping 241 UK council planning portals – 2.6M decisions so far
Source: Hacker News
The scraping problem
Most UK councils run one of a handful of portal systems, with Idox being the most common. In theory this makes things easy, but in practice every council has configured theirs differently. Some block non‑browser requests via TLS fingerprinting, some have rate limits that will get you banned inside 10 minutes, and a handful are running bespoke ASP.NET portals.
I ended up writing several scrapers:
- a standard
requests‑based scraper - a Playwright‑based scraper for councils that block anything that doesn’t look like a real browser
- a
curl_cffiscraper for TLS fingerprinting
Some councils I still can’t get. Liverpool’s portal sits behind AWS WAF with a JavaScript challenge. I have a working Playwright‑based scraper that solves the challenge once and reuses cookies, but the WAF rate‑limits the IP after about 10 requests and then blocks me for a day. So I have 60 k Liverpool decisions from an old scrape and no easy way to add more.
What I found
The approval‑rate statistics are what most people look for. Nationally it’s around 88 %, but it varies wildly by ward within a council, not just between councils.
A more interesting finding came from the time‑to‑decision data. Across 119 English and Welsh councils:
- 36.5 % of home‑extension applications missed the statutory 8‑week target in 2025, up from 27.9 % in 2019.
- Guildford is the worst at scale: 66 % of decisions were over target, averaging 13.3 weeks.
What it is now
- A free postcode checker
- Paid PDF reports (£19 / £79)
Zero paying customers so far, which is fine—I’m focused on data quality and coverage.
The site is planninglens.co.uk if you want to poke around.
AMA on the scraping side – that’s where the interesting problems are.
Comments: https://news.ycombinator.com/item?id=47931470 (Points: 29)