Commands to list and trim the Ops deferred queue

[tonyandrewmeyer]

Migrating from canonical/operator#2101, where there is a request for:

• the ability for users to see the size of event queue (for deferred events). It will allow to provide customer estimates and plan hardware usage better, as differed event execution takes CPU)
• the ability to clean unnecessary events somehow. At the moment users have to go into sqlite on unit to clean the queue => the dangerous and risky way. Otherwise wait for hours/days wasting CPU/time.

For the former, perhaps something like jhack defer-queue list that prettily lists the deferred notices (the 4 fields in the notices table), with a total count somewhere. Potentially with the snapshot, although that makes it more expensive and noisy (if anyone wanted it, there could be a future jhack defer-queue get x that showed details for one row including the snapshot, maybe?).

For the latter, perhaps something like jhack defer-queue clear [--duplicate] that wipes the notices and snapshot tables, or if --duplicate is used, then only removes duplicates. If people wanted there could be future filtering ability like limiting it to specific events.

--duplicate is a bit odd in that modern Ops won't store duplicates, so you shouldn't really have any, but it is useful if you're using an old version of Ops and have duplicates to carefully remove.

@PietroPasotti if you're on board with adding this functionality and we can agree on what it should look like, I can contribute a PR for it.

[PietroPasotti]

Hi! I think this would be a good addition to the dev toolkit.
I was playing last week with jhack script and some source contained in https://github.com/PietroPasotti/deferral_queue_editor/ to view and edit the deferral queue and that worked well for my use case, but perhaps offering something a bit more 'baked in' would be useful.
I'm happy to review PRs or work on it myself.
The approach I was using so far was: juju scp a python script that does the thing, juju ssh to run it, then remove the script
Another possible approach would be: juju scp to download the unit-state DB; then do the thing, then re-upload the edited version.

Caveats: controller storage behaves somewhat differently (see a related PR also in ops about a bug I found in _ops_main_mock while I was working on this).

[tonyandrewmeyer]

I was playing last week with jhack script and some source contained in https://github.com/PietroPasotti/deferral_queue_editor/ to view and edit the deferral queue and that worked well for my use case, but perhaps offering something a bit more 'baked in' would be useful.

Ah, I was sure I'd seen something along these lines from you before, but didn't remember that it was outside of jack.

I'm happy to review PRs or work on it myself.

Same - if you're keen to do this then leaving it to you would be great, but I didn't want to push work on to you :)

The approach I was using so far was: juju scp a python script that does the thing, juju ssh to run it, then remove the script Another possible approach would be: juju scp to download the unit-state DB; then do the thing, then re-upload the edited version.

@benhoyt what do you think about just using a script rather than having it baked in? This does seem like something that should be pretty rare, and the use-case in the original ticket mostly goes away by just using recent Ops.

[PietroPasotti]

yeah I'm also wondering if this is broad enough to be useful to many people. I think only the data team have defer-calypse scenarios

[benhoyt]

Yeah, I agree that using a script is best for cleaning things up. However, I still think listing the defer queue would be a good addition. jhack defer-queue or jhack deferred-events maybe?

[PietroPasotti]

sounds good! will hack it together soon :) (friday is hackday)

[PietroPasotti]

Finding out there's already a few ways to get what we want in common use cases:

• jhack tail supports a -d flag to include a listing of currently deferred events:

... But that could be unwieldy if there's thousands of them

• there's jhack scenario snapshot, intended to generate unittest-ready Scenario.States from live units, which can be bent to the purpose as well (pending a fix to work with charms using controller storage...)

This is much more flexible and supports various output formats, I think this is good enough for debugging, I don't see a point in exposing a separate path to an already rather simple command

[PietroPasotti]

side-note: haven't tested this in a long time, but you should be able to pipe the output of jhack scenario snapshot to file, edit the json, and then run jhack scenario state-apply to push your edited deferred-events queue to the unit. Probably broken with controller-storage-using charms as well ATM, but kinda fancy for other charms.

edit: NVM, I never implemented that for stored state/deferred events. could do though, if there's interest.