Make the photon a downloadable resource

[achiefa]

This PR addresses #2214

[achiefa]

Hi @scarlehoff , can you please start having a look? I'm pretty sure this needs to be tweaked a bit more, but it's a good starting point. I tried using a local photon set, generated using the new script fiatlux_exec.py, to check if I implemented the loading part correctly, and it works. Some major things are still missing:

• A proper remote folder, where the photon can be downloaded. Note that the code in download_photonQED is commented out because I'm waiting for instructions on this respect.
• Tests need to be updated. But I'd prefer to wait until a final version of this PR.

[scarlehoff]

A proper remote folder, where the photon can be downloaded. Note that the code in download_photonQED is commented out because I'm waiting for instructions on this respect.

Let's use the Milan server for this for now: /home/nnpdf/WEB/photons this is a sibling of e.g. /home/nnpdf/WEB/hyperscans

[scarlehoff]

I've only had a look at the download. The parallelization I need more time to be able to give meaningful comments.

Did you already tested and saw that 1) it was faster 2) it didn't make the memory explode 3) the results are correct (god knows whether fiatlux is threadsafe)

[scarlehoff]

I think the parallelization is ok as well (provided it runs, I didn't test it).

I think for now (before testing the download online) just make it so the download is a copy from some folder in your computer to the share folder. If that works the download from the server is a technicality and will work just the same.

I think this will already ease quite a bit the QED fits (e.g., all the positivity checks that are being done now could be done with QED for free thanks to this for the price of one single computation of the photon!)

[achiefa]

Please, have a look at this training report (link). In the new fit, I downloaded the photon previously computed with the script fiatlux-exec.

One thing that is left is whether we want to move fiatlux-exec into vp-setupfit fit. In general, we need to agree on how we want to compute and upload the photon PDF.

[scarlehoff]

Thanks for this. I've tried the light photons from the tests and seems to do ok. I'll try with some heavier photon, I'm curious about how heavy the eko will be in that case.

[scarlehoff]

I'm happy with the changes, just a few comments about the wording.

And a big comment about the docs! Could you please update the docs to include this functionality?

At the very least, explain it (and how to use it and where int he server are the photons) at the tutorial page https://docs.nnpdf.science/tutorials/run-qed-fit.html

Crucially it should mention:

1. What will happen when you use compute_in_setupfit
2. In which situations it makes sense to have it (best effort)
3. That the number of cores to use is the one in the runcard. FWIW, 64 cores is using 64 GB of RAM, perhaps this is also worth mentioning.

And any other info that you deem important.

I'm running a test 4.0-like QED fit with this branch. I don't expect to see any problems but probably will have finished by tomorrow so we can merge as soon as the docs are added if that doesn't crash smh

[achiefa]

I've added the documentation. Please, have a look and let me know.

[scarlehoff]

You went too specific :'( but ok for the rest

Look, I've submitted a job to create the photon for NNPDF40_nnlo_as_01180_qed I'll let you know when it is done.

[scarlehoff]

Suggested change

	luxset: 251127-jcm-lh-qed-001
	luxset: NNPDF40_nnlo_as_01180_qcd

I only added that for testing.

[scarlehoff]

shouldn't the compute in setupfit flag also feature here?

[achiefa]

Well, maybe. This part focuses only on the creation of the runcard. The compute_in_setupfit flag is needed if the user wants to compute the photon from scratch. But this is required only if the photon is not already stored in the remote. That's why I opted for the following reasoning order:

Setting up the runcard --> check and collect available resources --> Compute missing resources.

Maybe the wording is a bit misleading at this stage of the tutorial because the user is not yet aware of the possibility of either downloading/storing available resources or computing resources from scratch. Maybe something like that, which introduces the two possibilities, sounds better:

The code uses both the ``fiatlux`` block and the ``theoryid`` specified in the
runcard to identify the photon PDF set, which will be assigned the name
``photon_theoryID_<theoryid>_fit_<luxset>``. This name will be used to either
look for existing photon PDF sets or to store newly generated ones, as explained
below.

[scarlehoff]

But it is a flag for the dictionary fiatlux.

[scarlehoff]

If you want to be this specific, create a photon for NNPDF40_nnlo_as_01180_qed and theoryid 40000100

[achiefa]

In the context of a tutorial, I find this level of detail quite suited. If I were an external user, I'd appreciate this one-to-one approach. I agree that if we want to keep this approach, we need existing sets. What do you think?

[scarlehoff]

I'm happy with the level of detail. I am less happy about having to be the one creating the sets for the level of detail to apply 🤷‍♂️

[scarlehoff]

Same comments as before.

[scarlehoff]

A few comments here

1. you should mention explicitly the command to run for the .tar because it is not the same to do

tar -cf my_photon.tar my_photon_folder

or

cd my_photon_folder
tar -cf my_photon.tar *.npz

or even

cd my_photon_folder
tar -czf my_photon.tgz *.npz

2. In the profile you have the URL but you didn't add a photon_target_dir so it is impossible to know where in the server the photon lives!

[scarlehoff]

Btw, why is it not supported? You already have all the infrastructure so it should be easy.

We probably already talked about this but you can think about me like an LLM with about 30 words of memory.

[achiefa]

Btw, why is it not supported? You already have all the infrastructure, so it should be easy.

We probably already talked about this, but you can think about me like an LLM with about 30 words of memory.

I don't remember the reason, but I'm pretty sure that at some point we agreed on leaving this completely manual because it wasn't a priority :/ But with hindsight, I think at this stage we might want to allow for uploading photon resources using vp-upload, which eases the process by quite a lot.

[scarlehoff]

I'm pretty sure that at some point we agreed on leaving this completely manual because it wasn't a priority

I agree with my past self and you it isn't, we can do it at a later stage.

[scarlehoff]

The new more official and cooler photon is ready in the server: photon_theoryID_40000100_fit_NNPDF40_nnlo_as_01180_qed.tar

edit: FWIW, while I think it would be better to have it in vp-upload (and I'll be very happy if you do it next) I think with the changes to the docs I mentioned this can be merged and used. The functionality itself is quite nice, orange is this branch:

(and, not seen in the blue, you remove the possibility of being unlucky and having some replicas that won't converge)