ERROR IN FILELISTS

[MDallOsso]

a 'known' issue appeared again in filelists...
the names of 'ext' files are included in the filelist of the 'non ext' files too!

this is an important issue because, when we merge the 'non ext' and 'ext' versions of a sample we are double counting the extension.
From a quick check I see that QCD files are messed up starting from this commit a8402e4 .

can you please check the filelist of some MC that you usually use?
we need to:

• check if bug is present in script
• make a list of bugged filelists so that we know what needs to be re-processed.

martino

p.s. we are in a rush to get pre-approval documentation and this is extremely annoying.

[jpazzini]

I started checking the createlist code to isolate and fix the issue on the script generating the lists.
Lisa already spotted that while the 'ext' lists seems OK, the 'non-ext' ones contain both 'ext' and 'non-ext' file; she's now trying to evaluate the impact on her analysis (also going to pre-approval next week).
I suggest you start checking all the lists in the repo, and to isolate the latest working config of the createlist script.

PS:
This is an (unfortunately so far unseen) issue that is affecting us all, and is annoying for us all.
However, since nobody here is "developer-only" or "enduser-only", we have all to work to check and maintain the code working and properly functional.

[jpazzini]

It seems the issue was introduced because I tried to maintain full back-compatibility to all the exotic early 2016 samples (withHLT, reHLT, and so on). In doing this I introduced an error in the way the code greps the name of the files.
I can fix it quite easily, but I see two options here:

1. since we are now in 2017 with full reReco+reMiniAOD and PUMoriond17 samples, simply get rid of the back-compatibility with those (now useless) samples, and simplify the code
2. keep retro-compatibility by tricking the code a bit more

I strongly suggest 1, but I can prepare fix for both. Just make sure if anybody is still really using these obsolete samples.

[MDallOsso]

Thank you for the check.
From what I see, some QCD and the WJetsToLNu are affected by this.
I was pretty sure we checked that the createlist script worked after modification for Summer16 but probably it was not the case.

I vote for option 1 and I think we can go with that.
We are using only reMiniAOD.
Ugo could be on old MC still but he has surely filelists saved in local.
In the future, we can eventually retrieve back to previous version if needed.

[jpazzini]

Lisa and I just merged a new version of the createlist script (together with the newly generated filelists) on our development branch (REMINIAOD) with PR #235. To us, it looks OK.
I suggest everybody to take the script from that PR, check it carefully, and test it.
If everything is OK, I'd suggest you to pick the script and do the dedicated PR to the master (or your development branches) to update it, as well as the lists.