Get list of new developments/estates built between 2000 and 2010

[Robinlovelace]

Related to this: #56

[joeytalbot]

You can find a list of planning applications from PlanIt having app_size = large, app_state = permitted and end_date = 2002-01-01 under acton/releases/large_old_apps.csv.

@aspeakman I realise most of these applications are not from planning authorities that you have re-scraped data from. But what impact does that have on the app_size classifications? The list contains various applications that don't seem very large at first glance, such as "Extend and raise the roof to create a large dormer"; "Certificate of Lawfulness for the proposed replacement of windows"; "Consultation by Cornwall County Council for the construction of extension to kitchen.", etc. How would the re-scraping change this?

[Robinlovelace]

Great progress, simpler than initially thought it seems, and sounds like we have enough to get going on creating a model and writing-up as a plan of action (or should I say a plan of acton heheh ; )

[aspeakman]

Sorry but many of these applications were misclassified during my first pass at this when I thought that keywords 'prior approval' or 'prior notification' in the application_type field signified a large application. Now the decision is mainly based on the n_statutory_days and n_documents fields (where available).

For around 60% of authorities having a particular type (Idox) these fields are being collected and updated BUT applications before 2005 have not yet been included, because this is a cut-off date for arbitrary processing time reasons (especially noticeable in Cornwall). I hope to target these pre-2005 applications for update at some point .

Note across the board there is a long tail of other scraper types that each require an update to the source code to collect a value in the 'n_documents' field which is one of the key identifiers for a large application. In the mean time these other types will make the 'app_size' classification based on the 'n_statutory_days' field (>60)

[Robinlovelace]

Thanks for the updated @aspeakman.

I hope to target these pre-2005 applications for update at some point .

Is there any way you can fast track this? The project ends soon and getting a good handle on pre 2011 developments will be most useful in the next 24 hours if at all possible.

[aspeakman]

For various reasons there are quite major lags built in to the system. There are around 265 Idox scrapers and they are mostly quite slow. In 2016 I was scraping a few of the early examples back to year 2000 (hence the old data in that range), but as the number expanded I had to restrict the current data scraping to 2006 onwards. I recently moved the target scraping date back by a year to 2005 and it took more than a month for the system to catch up across the board.

So I can offer some targeted updates (but not within 24 hours). Alternatively you could restrict your analysis to 2005 to 2011 to provide better results. Or I can provide the list of target dates for each scraper so you know how far back you can trust the data in each case.

[Robinlovelace]

Thanks the quick reply @aspeakman, will defer to @joeytalbot on best option.

[mvl22]

Note that the PlanIt API now has a new 'Conditions' application type, which has been split out from Amendment.

You can see this in action at:
https://www.streetfocus.uk/map/

[Robinlovelace]

Another potential source of data for future developments: