Trajectory runtime optimization

[Ernst-Ellis]

Hi,

I am still experiencing high runtimes even with parallelization enabled. Before I enabled parallelization I was averaging between 3000 - 4000 seconds per step, but after parallelization I am still averaging around 3000 to 4000 seconds. For a total runtime of 200hours for a single trajectory on 8 cores, which well exceeds the cluster's 48 hour walltime for a single calculation. Now I can request a longer walltime, but I believe the longest walltime allowed is still only 100 hours. Do you have any suggestions.

I have attached my latest calculation using mpi. I stopped the calculation so please ignore any weird errors due to termination.

https://drive.google.com/drive/folders/1dTXhBwUt-A-snN_KAEeJP1FDjrTF9gGd?usp=sharing

Kind regards,
Ernst

[maisebastian]

Hi,
in our other thread, we discussed that your high run times might be due to SHARC4's new Molcas interface enforcing a super tight convergence. Did you try to reduce this to normal (Molcas-default) levels? Or is this now with a different system?

What kind of file system do you use as scratchdir? OpenMolcas benefits a lot from a fast scratch drive (like a local SSD on the node, rather than a network-mounted global file system).

One thing that I see in your MOLCAS.out files is that the RASSCF iterations do not take particularly long, but the RASSCF modules take pretty long. Maybe the printing of the Slater determinants is a bottleneck in your case. You can try to go to _write_rasscf() and remove the parts with PRWF and PRSD. To me it looks like this should not break anything for your case.

Best,
Sebastian

PS: By the way, the zip you sent is 7GB. The next time, ideally remove large files (everywhere, in the master, grad, nac, and tmp folders). For large, binary files, the content is not important, just the presence of the file is sometimes interesting (like in the mpi_parallel issues in the previous thread).

[Ernst-Ellis]

Hi,

How do I reduce the tight convergence to normal levels?

Also at the moment I am only running the code locally, which means that the scratchdir is on a local ssd, so I don't believe that to be the problem.

I will remove the PRWF and PRSD parts as a last resort and try to run again in the meantime.

Kind regards,
Ernst

[maisebastian]

Instead of

        if qmin.maps["gradmap"] and len(qmin.maps["gradmap"]) > 0:
            input_str += "THRS=1.0e-10 1.0e-06 1.0e-06\n"
        else:
            input_str += "THRS=" + " ".join(f"{i:14.12f}" for i in qmin.template["rasscf_thrs"]) + "\n"

simply use

input_str += "THRS=" + " ".join(f"{i:14.12f}" for i in qmin.template["rasscf_thrs"]) + "\n"

This increases a bit the chance that the gradient calculations experience convergence problems, but it should save quite a bit of time.

I now confirmed that the PRWF and PRSD keywords are not important for you.

Best,
Sebastian

[Ernst-Ellis]

I did both remove the PRWF and PRSD code lines as well as change the convergence code. I am down to ~1600 seconds a step. This is still too high. I think I am still baffled how with sharc3.2 one a single core i was able to get 300 seconds, and now I have 1600 seconds. It is the exact same setup.

Do you have any other ideas as to the increase?

Best,
Ernst

[maisebastian]

Well, the most interesting part would be to do exactly the same calculation (e.g., a single point with gradients and SOCs and maybe overlaps) with both SHARC3 and SHARC4 interfaces and then compare the Molcas output files. This would help identifying which parts take longer in SHARC4 and then one can see what could be a way to overcome this.

Best,
Sebastian