Not clear how to lock a thing?

[julianstirling]

In the first LabThings it was my understanding that a core concept was being able to lock a Thing so that it cannot be interacted with. It is not clear how to do this, or if it is possible with labthings-fastapi.

This is a problem for the OpenFlexure Microscope as it is still possible to tell the stage to move while a scan is running, or to interact with the camera.

Is it possible to lock? If not, how should this be implemented.

[rwb27]

python-labthings (AKA LabThings 1) had a StrictLock class, but it was basically just a threading.RLock with some additional timeout logic I think.

What you describe (the ability to lock a Thing so nobody else can use it while you do something) would be very useful, and should, I think, be supported at framework level. Exactly how to implement that will take some thought, but I agree it's worth doing.

Previous code (with labthings 1) made heavy use of threading.RLock reentrant locks. This meant that long-running actions on the server could lock out other actions, simply by acquiring a lock that was used by them. For example, if I pseudo-code a stage and a scanning routine,

class Stage(Thing):
    def move(self, x, y, z):
        with self.action_lock:
       # do some stuff

This meant that a scanning action could do:

with stage.action_lock:
    stage.move(0,0,2)

This all worked, because the actions were being called in the thread that had already acquired the lock, and RLock objects can be acquired many times by the same thread.

[rwb27]

I think there are a few places where locks would be useful:

• protecting hardware resources should already be done using threading locking primitives - but these generally shouldn't be used outside of the Thing.
• Adding a blanket lock on all the Actions of a Thing would (I think) do what you are suggesting - it would be simple to implement server-side but doesn't currently play nicely with the "things-should-work-identically-on-the-server-and-client" structure I'm so fond of. It would also enforce a one-action-per-thing policy, which I don't think is always desirable - though in practice many things do this already (e.g. the stage can only run one query at a time, which pretty much means only one action at a time).
• Figuring out a way to lock things that works remotely could be really handy - e.g. I could lock the microscope while my experiment runs in a notebook. Obviously this would be far trickier to implement, because you'd need to worry about timeouts, keep-alive packets, and probably security.

I'm not precisely sure what the minimum useful functionality would be here: a way to stop any actions being called on a thing would for sure be handy. That might want to be more than a simple lock though.

[julianstirling]

Can we have a custom lock ID that is passed in by dependency injection the same way the invocation_logger is passed in. So this can then be passed to actions that are called by that action using the same ID.

This way if we have some sort of lock where when you acquire it you give it your action ID. So if the ID matches then it gives you access, if it doesn't match then it doesn't.

Even better, would be if we had some sort of injected locker class so that the action could start with:

action_locker.lock(cam)
action_locker.lock(background_detect)
action_locker.lock(stage)

And then when this locker is cleaned up at the end of the action it unlocks everything automatically. would this be possible?

[julianstirling]

• Adding a blanket lock on all the Actions of a Thing would (I think) do what you are suggesting - it would be simple to implement server-side but doesn't currently play nicely with the "things-should-work-identically-on-the-server-and-client" structure I'm so fond of. It would also enforce a one-action-per-thing policy, which I don't think is always desirable - though in practice many things do this already (e.g. the stage can only run one query at a time, which pretty much means only one action at a time).

I am suggesting a re-entrant lock, so that when you start an action, then that action thread acquires the lock for the Thing's it is using. This is essential for things like scanning on the OpenFlexure, because the current Sangaboard lock is to stop two processes talking to the motors at the same time. But I can use the arrow keys to move the motors in view as long because the stage itself isn't locked, and it releases the lock every time a move finishes.

What about if the default logic was: When an action is called by a client, all Thing dependencies are locked for the duration of that action. This would ensure it works the same on the client or on the server. On the microscope it would have the expected outcome of not letting the stage move while a scan was running.

I think the vast majority of the time we do want actions for the same thing to be run in parallel. We also don't want settings/properties being updated while an action is running. I think there are some notable exceptions.

Actions that don't talk to hardware.

On the Microscope, SmartScanThing has actions to delete scans or download a zip. These can safely run while the microscope is scanning. We could decorate these actions that don't require the lock specifically.

@thing_action(aquire_lock=False):
    def download_zip(self, scan_name: str):

This way it doesn't aquire the Thing's lock, meaning other a scan could be started while this action was running. And that the action could be done during a scan.

Things that never lock

There may be some software interfaces that are conveniently written as a Thing, but actually never talk to hardware and therefore never need locking. There probably wants to be a way to turn off locking entirely for certain things. This could be done with a property, or a subclass.

class NonHardwareThing(Thing):
    lockable = False

class NonHardwareThing(NonLockingThing):
    ...

[rwb27]

I think we're pushing in the same direction here, and probably we need a way to play with some locking before figuring out how to put it into LabThings. To that end, I've tested using decorators for locks on actions. It's absolutely not a full solution, but it might give us enough to play with different options before implementing them properly. Code is in #177.

[rwb27]

I'm keen to get some sort of locking in for 0.0.12, even if it's fairly experimental. I think even a minimal implementation is not trivial, but this is repeatedly highlighted as a big issue for OFM and it would make sense to get this in early in the release cycle there, if we can.

[julianstirling]

I disagree.

I think 0.0.12 is already going to be such an enormous change for the server I would prefer to decouple that from locking. Or the upgrade path becomes very hard. We have new version of the server hopefully coming out in about 2 weeks. So I would try to get v0.0.12 out in the next week or so, so that we are ready at the start of the dev cycle.

We still need to do a lot of actual testing of new dependencies and how they work in anger. We also need to cull all of the old raw_thing_dependency, direct_thing_client code once we can move over.

Once we have new dependencies set up we can directly acess things. So we can actually prototype a locking structure in a branch of the microscope (in a way that is never merged to see what does and doesn't work before we formalise it).

We need to talk and think about locking in a lot of detail before we implement it. And we shouldn't delay an enormous release for even more major changes

[bprobert97]

I think adding is a lock is definitely a priority, but I agree with Julian - we need to do one thing at a time. The other changes for v0.0.12 are huge and complex, so I'd prefer we got those merged in so we can do integration testing with OFM and spot any problems. I worry that if we add locking in too, which is a non-trivial addition, that debugging how the new LabThings affects OFM is going to be extra complicated.

Could this be priority number 1 for v0.0.13?