Api calls failed (getting rate limited?) - use stored values where possible

[markgdev]

Api calls have recently started failing every now and then, it looks like they’re maybe getting rate limited. There are a lot of places where an api call is made where it doesn’t need to be, where possible data should be taken from stored values, most likely the rates entity in most cases.

Where it’s not a quick fix, for example run_devices the error should be caught so as not to cause further code not to be ignored (timers are executed after run_devices update so are not triggering.)

[create-issue-branch]

Branch issue-72-Api_calls_failed_getting_rate_limited_use_stored_values_where_possible created!

[markgdev]

Currently in testing:

• Added new entity “octopusagile.all_rates” to store the whole days rates which can then be used by sensors. This entity will not get updated in the same way as the rates entity does each 30 minutes.
• Previous rate, current rate and next rate updated to pull rates from the new entity
• Added try/except for run_devices updated until a better solution is implemented.

Need to force all_rates to update from the previous 30 min slot if the entity doesn’t already exist to avoid unknown values in the sensors on first startup.

[markgdev]

• Rates and all_rates will update if they’re not in octopusagile.json (file must currently exist though)

• New configuration option “consumption”, set to false to stop pulling historic consumption data, which involves many api calls.

[d1964b]

Hi, I installed the 0.1.1-beta.1code @ 22:43:44 and so far the following has been logged:

2021-04-30 22:43:44 WARNING (MainThread) [homeassistant.loader] You are using a custom integration sonoff which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2021-04-30 22:43:48 ERROR (SyncWorker_0) [custom_components.octopusagile.sensor] Error updating sensor.octopus_agile_previous_rate, check that octopusagile.all_rates populated
2021-04-30 22:53:26 WARNING (SyncWorker_2) [custom_components.octopusagile] Timer didn't exist
2021-04-30 23:00:00 ERROR (SyncWorker_0) [custom_components.octopusagile] Failed to update run_devices for dishwasher
2021-04-30 23:03:21 ERROR (SyncWorker_3) [OctopusAgile] Unmatched consumption 2021-03-14T11:30:01Z / cost 2021-03-14T11:30:00Z
2021-05-01 00:00:01 ERROR (SyncWorker_1) [custom_components.octopusagile] Failed to update run_devices for dishwasher
2021-05-01 00:30:04 WARNING (MainThread) [homeassistant.helpers.service] Unable to find referenced entities switch.sonoff_10006c0d87
2021-05-01 01:00:00 ERROR (SyncWorker_4) [custom_components.octopusagile] Failed to update run_devices for dishwasher
2021-05-01 02:00:00 ERROR (SyncWorker_6) [custom_components.octopusagile] Failed to update run_devices for dishwasher
2021-05-01 02:00:00 ERROR (SyncWorker_6) [custom_components.octopusagile] Failed to update run_devices for clotheswasher
2021-05-01 03:30:00 ERROR (SyncWorker_1) [custom_components.octopusagile] Failed to update run_devices for dishwasher
2021-05-01 04:00:00 ERROR (SyncWorker_4) [custom_components.octopusagile] Failed to update run_devices for dishwasher
2021-05-01 04:00:00 ERROR (SyncWorker_4) [custom_components.octopusagile] Failed to update run_devices for clotheswasher
2021-05-01 08:00:00 ERROR (SyncWorker_5) [custom_components.octopusagile] Failed to update run_devices for dishwasher
2021-05-01 08:00:00 ERROR (SyncWorker_5) [custom_components.octopusagile] Failed to update run_devices for clotheswasher
2021-05-01 08:30:05 ERROR (SyncWorker_3) [custom_components.octopusagile] Failed to update run_devices for clothesdryer

`

Overall summary
a) All of the timers which were calculated were successfully processed ( by toggling the values of my input_booleans ) , so if my understanding of the 'failed to update run_devices ..' is that this is the new , more managed error, arising from problems accessing the api servers- then that problem is no longer stopping known timers from being processed. 👍
b) I've been running the beta script with consumption set to false.

Queries
a) I don't understand why , if the integration cannot access the api servers , it should only fail to update a subset of the devices though - I would have thought it would have been unable to update any of them ( there are 3 devices - in my case - defined )
b) If I access Guy Lipman's page then I can recover consumption figures for the unmatched 30 mins , so the data is there to be retrieved - I'm wondering if you can't match the consumption because of the trailling 1 second in the consumption timestamp ?? - Even though I set 'consumption: false' in config.yaml , I ran the service manually to see what would happen hence consumption values were calculated.

Suggestions
Perhaps in addition to logging errors in the core log , a boolean value could be toggled at the end of each service run. Value at initialisation should be false. That way should a call to half_hour or update_consumption fail then a 'backup' automation to rerun that service could be created ( such as that suggested by iMiMx ) e.g half_hour_last_run = false . This automation could then be triggerd but only when it was actually neccessary and not just when a particular time pattern was matched. ( A time pattern in the automation is still useful , since it in combination withe the boolean switch would allow the load distribution of the call to the octopus api servers )

[d1964b]

The json errors have not entirely disapeared
`2021-05-01 20:00:00 ERROR (SyncWorker_5) [custom_components.octopusagile] Expecting value: line 1 column 1 (char 0)
2021-05-01 20:30:00 ERROR (SyncWorker_4) [custom_components.octopusagile] Expecting value: line 1 column 1 (char 0)
2021-05-01 20:41:15 WARNING (SyncWorker_2) [custom_components.octopusagile] Timer didn't exist

2021-05-01 21:00:00 ERROR (SyncWorker_3) [custom_components.octopusagile] Expecting value: line 1 column 1 (char 0)`
I assume this relates to the updating of the timers , as I was expecting to see that happen at 2000, but when that had not happened by 2041 I ran it myself manually. At that point the timers did successfuly update, byt the next_update for timers is still beig scheduled every 30 mins (?) and json erros have resumed

[markgdev]

Glad it worked as expected, good spot on the timer updates @ 2000, I hadn’t noticed that before, they retry every 30mins until they manage to update, it looks like mine succeeded at 2030 tonight.
I find the same with the run_devices updates only failing for certain entities, each one queries individually so it kind of makes sense.
If I’m honest, I’m not happy with the way I have the consumption at the moment, I’m going to revisit. My current plan is that I store everything historic in a file(s) as I think it’d be too much to store in an entity and then summaries in entities and keep track of any missing periods in a separate entity, each time I try to retrieve data I’ll then look for anything from the last update along with anything from the missing list.
I’m going to focus on the updates of run_devices, I’ll see how much work it’ll be to stop them calling the api each time, if that is going to take some time I’ll add either the flag you mention or a last update time stamp attribute that can be used to trigger an automation if it’s more than 30ish minutes ago.
I’ll then move onto the updating of the timers, I think it makes sense to do these second as they do eventually update from what I can see (also, the answer will probably be the same/similar as the run_devices)

Thanks for helping fault find and test this, it’s appreciated!

[d1964b]

It's not a problem particularly , just a curiosity, but I've noticed that although the timers next update may be scheduled for 1900Z it is actualy running at 1900BST.Which I think that is actually the best time for it , but maybe it would be worth advertising the time as 1900 only then that would cover both UTC and BST. Just a thought.