Make sure the xAOD science image open file failure causes a fall-back to the next replica

[gordonwatts]

Describe the bug

Test that the error message generated by Analysis Base when it can't open a file because it does not exist is properly interpreted. If there are multiple replicas, it should cause a fallback to the second replica.

To Reproduce
Steps to reproduce the behavior:

1. Give the AB science image two files, the first of which is non-existent file, which should cause an open failure.

Expected behavior

It should back off to the second file.

• It isn't well understood how this failover mechanism works.
• Evidence of this bug is only annecdotal

[ponyisi]

Hi @gordonwatts , @RogerJanusiak : it seems to me that the xAOD code relies on the following code:

# Run the job using the direct driver.
driver = ROOT.EL.DirectDriver()
driver.submit(job, options.submission_dir)

from https://github.com/iris-hep/func_adl_xAOD/blob/master/func_adl_xAOD/template/atlas/r21/ATestRun_eljob.py . Unless the submit method raises an exception I don't think this will give a non-zero error code if the requested files are not available, which is the requirement for the sidecar to attempt the next replica. Could you check?

[gordonwatts]

@ponyisi - good point. I don't know how that works! We should have tests.

Is there a particular exception it should raise for the infrastructure?

[ponyisi]

I think all it really needs to do is return a non-zero error code. A python script that exits on an exception will do this but sys.exit(1) should also work.