Using a direct (non scraped) source for North Herts and others?

[robberwick]

Hi

Before I go ahead and start doing work to raise a PR or PRs, I thought I'd check something first..

I recently reverse engineered the new mobile app for North Herts Council, which exposes endpoints for querying bin schedules directly via a single GET request. Previously I had been jumping through the same webscraping based hoops as you have to here, so being able to switch to a direct request was not only quicker, but also had fewer moving parts to fail, which is nice...

The only difference in workflow is that the mobile app requires a UPRN as part of the URL rather than an address, but that's easy to find using free online tools.

Would switching the current North Herts council implementation to use that mobile app API endpoint be something that you'd be interested in?

Looking at the listing of other apps on the google play store from the same publisher (https://play.google.com/store/apps/developer?id=Cloud+9+Technologies), it looks like the same approach would be valid for the other councils who use them for their mobile apps, as they are just reskinned with different logos. Indeed, having done some speculative testing by guessing the urls for other Councils, it works just fine.

The list of councils currently using that Cloud 9 app is:
West Northamptonshire Council
Conwy County Borough Council
Arun District Council
Chicester District Council
Armagh City Banbridge & Craigavon Council
Southend-on-sea City Council
Easy Riding Council
Horsham District Council
Rugby Borough Council
Adur & Worthing Councils
South Norfolk Council
Mid Sussex District Council
East Devon District Council
Warwick District Council
North Somerset Council
North Herts Council
East Herts Council

[robbrad]

Think that's absolutely fine as long as it doesn't need or use tokens that will expire or has some complex logging, secrets etc

Reliability is a driver.

[robberwick]

I don't think that should be a problem - the endpoint requires a few specific headers, but other than that, it's just a plain old HTTP GET:

GET https://apps.cloud9technologies.com/northherts/citizenmobile/mobileapi/wastecollections/<UPRN>/
Accept: application/json
Authorization: Basic Y2xvdWQ5OmlkQmNWNGJvcjU=
x-api-version: 2
x-app-version: 3.0.56
x-installation-id: a5cec49a-1a05-4c74-8f6c-4617b07117f5
x-platform: android

The x-installation-id is not checked, and can be any valid UUID4, and that basic auth will presumably not be changed, as it's common for all the council endpoint variants I've tested.

[robbrad]

Give it a go but I do see Authorisation: Basic in there.

[robberwick]

True, but that Auth header is hard coded in to the mobile app itself though, rather than being a per user or per session value. As such, I can't imagine that it's likely to change frequently, if at all, as it would break legacy versions of the app. In fact, I'd imagine that it's far less likely to change than the DOM structure of the web page content, causing a web scraping based flow to break.

I'll throw something together and raise a PR, and let you kick the tyres on it.