Minimization step performed in the middle of simulations within the adaptive protocol

[silviamenin]

Hi!
I've noticed that before every simulation with somd, a little minimization step is performed, to avoid that simulations blow up for lambda windows far away from the end states where they were equilibrated.

The problem is that, when using the adaptive protocol, where a single simulation is made of more than one trajectory cycles, the minimization is performed not only at the very start of the trajectory, but also at the start of every subsequent simulation cycles.
In this way the simulations are not consistent, since there are several minimisation steps during the productive trajectory.

The software, after the first simulation cycle, should recognize it and turn off minimmisation from the somd.cfg

[fjclark]

Thanks @silviamenin!

As mentioned in our chat, a quick hack to fix this would be to add something like

for sim in win.sims:
    sim.set_simfile_option("minimise", "false")

between lines 508 and 509 here

a3fe/a3fe/run/stage.py

Line 509 in 4e25894

win.run(run_nos=run_nos, runtime=resubmit_time)

. However, this isn't an ideal solution since if you then ran calc.clean() and restarted you'd miss the initial minimisation since the somd.cfg files would remain set to minimise=false. I think this should be easier to fix once we merge in @Roy-Haolin-Du's changes from #40 (my fault this hasn't been merged yet as I need to find time to get round to testing in detail, sorry Roy.).

Maybe we could add a context manager to temporarily change the option?

[Roy-Haolin-Du]

Thanks Silvia, and thanks Finlay!
I’ll take a closer look at this issue. I can add a context manager to temporarily change the option if needed.

[fjclark]

Thanks Roy!

[Roy-Haolin-Du]

Hi @silviamenin and @fjclark , just had a quick look.

We could add a check to set this to False if the simulation has already been run :

if actual_run_time > 0:  # make sure the simulation has been run
    for sim in win.sims:
        sim.set_simfile_option("minimise", "false")
win.run(run_nos=run_nos, runtime=resubmit_time)

between lines 508 and 509 here

a3fe/a3fe/run/stage.py

Line 509 in 4e25894

win.run(run_nos=run_nos, runtime=resubmit_time)

if that sounds good to you?

[silviamenin]

Yes, thank you <3

[fjclark]

Thanks Roy! The one thing I think this is missing is that if you then ran calc.clean() and restarted you'd miss the initial minimisation since the somd.cfg files would remain set to minimise=false, which would likely be more of a problem than running some extra minimisations. It would be ideal if we could reset it to the original value. We could likely do this by reading the minimise option before we start, then resetting this once the calculations finish. However, to if there's an error while the simulations are being run it wouldn't be reverted, hence a context manager (or try/ finally statement) would be ideal to make sure that we always reset the option to the original value.

[Roy-Haolin-Du]

Thanks, Finlay, really appreciate your consideration. I think your reasoning makes a lot of sense.

Two types of errors: general software exceptions and hardware/device failures.

1. For general Python exceptions, the try/finally will execute, and minimise will be restored to True, which handles the simple case where the run is interrupted in a controlled way.

    @contextmanager
    def _temporarily_disable_minimisation(self, simulations):
        """Temporarily disable minimisation for a list of simulations."""
        try:
            for sim in simulations:
                sim.set_simfile_option("minimise", "false")
            yield
        finally:
            # would always be executed
            for sim in simulations:
                sim.set_simfile_option("minimise", "True")

Then,

if actual_run_time > 0:  # Make sure that we don't minimise again
    with self._temporarily_disable_minimisation(win.sims):
        win.run(run_nos=run_nos, runtime=resubmit_time)
else:
    win.run(run_nos=run_nos, runtime=resubmit_time)

2. However, in cases of hard interruptions like SLURM job termination or machine failure, the finally block doesn’t execute, and minimise remains stuck at False.
If we then run.clean(), it won’t reset the somd.cfg files, which meaning minimise is still False in the next run. This creates a corrupted state where minimisation is unintentionally skipped.

Can we address this by checking whether actual_run_time == 0 at the start of each run. When this condition is met, we force minimise = True to ensure proper minimisation.

def _run_without_threading(...):
...
            for win in self.lam_windows:
                actual_run_time = win.get_tot_simtime(run_nos=run_nos)
                if actual_run_time == 0:   # If this is the initial run, ensure minimisation is enabled
                    for sim in win.sims:
                        sim.set_simfile_option("minimise", "True")
...

This can handle both normal exceptions (via try/finally) and hard failures I think.
Let me know if you think this approach is reasonable, or if you’d suggest something else.

Thanks~

[fjclark]

Thanks Roy!

1. This looks good, but maybe store the initial value rather than assuming it's true (even though it should pretty much always be...) then reset to this later? Just in case minimisation is disabled for some reason.
2. Surely the try/ except would still work in the case of slurm job termination? As long as the process running the a3fe Python isn't terminated? I agree that 2 is more robust if we assume that the user always sets minimise to True, but ignores the user-set value of minimisation. Maybe it would be good to keep the flexibility to disable minimisation (even though I'm not sure when you'd do this in reality)?

I'd suggest either:

• a) Treat minimise as user-settable and respect what they set, e.g. read the initial value of minimise and reset this after running the simulations which carry on from .s3 files. Use the context manager approach.
• b) Treat minimise as something controlled by a3fe as ignore what the user has set. Decide whether to minimise based on whether we're starting from 0 or not. Make sure this approach is applied conisistently, e.g. running non-adaptively should always result in minimisation no matter what the user specifies. Document this change in behaviour.

I'm probably leaning towards a), but please go with whichever you think is best!

[Roy-Haolin-Du]

Thanks, Finlay. After our discussion, I think this is the most effective solution for now.

Use context manager approch to reset it to the original value.

    @contextmanager
    def _temporarily_disable_minimisation(self, simulations):
        """Temporarily disable minimisation for simulations, via user settings."""
        try:
            # Read original value from first simulation (assume all are the same)
            original_value = _read_simfile_option(simulations[0].simfile_path, "minimise")
        except (FileNotFoundError, ValueError):
            # Default to True if can't read the current setting
            original_value = "True"
        
        try:
            for sim in simulations:
                sim.set_simfile_option("minimise", "False")
            yield
            
        finally:
            for sim in simulations:
                sim.set_simfile_option("minimise", original_value)

and

if actual_run_time > 0:  # Make sure that we don't minimise again
    with self._temporarily_disable_minimisation(win.sims):
        win.run(run_nos=run_nos, runtime=resubmit_time)
else:
    win.run(run_nos=run_nos, runtime=resubmit_time)                 

Once the decoupled-somd is merged, I will fix this part.

Many thanks again to Silvia for spotting the issue and to Finlay for proposing the solution!
Have a good week~

[fjclark]

Looks good, thanks Roy!