Adding the modern HO reference samples?

[nevrome]

So far this repository only includes packages for the 1240k part of the latest AADR release, not the 1240K+HO part. To my understanding this second compilation includes the same individuals as the first (pulled down to the HO SNP set, which is a perfect subset of the 1240k SNP set) plus 4114 modern reference samples, for which no 1240k data exists yet.

Of course it would be neat to have all these additional samples here as well, but I think most (if not all?) are already in various packages in the published_data repository.

What do you think, @AyGhal? Could we maybe compile a list of relevant packages to add it to our documentation and thus justify why the HO data is not in here? Or is it easier if I just prepare an additional package here in poseidon_aadr with the relevant individuals?

[stschiff]

Hmm... good catch. So, if we want to keep poseidon-aadr as self-contained as we intended it to be, we could indeed add those 4114 reference samples. Since Poseidon easily supports SNP-subsets, we can just add them to this repo and avoid having yet another poseidon-aadr+HO or so dataset. But I think adding them would be good eventually.

[nevrome]

For v54.1.p1 this was added in #2.

For v62.0 we can consider doing this as well eventually. Maybe this subset of the data did not change since v54.1.p1.

[stschiff]

OK, we should discuss this in the next joint meeting. I was also wondering what the arguments had been in the beginning to create parallel packages for new versions instead of using our Poseidon Versioning system. I'm sure there were good reasons.

[nevrome]

The AADR releases are snapshots that continue to be used even long after the release of a new version. By treating them as separate packages we can fix issues in our poseidonisation of these packages. If I discover a mistake in the poseidonisation of v54.1_p1 then I want to be able to go back to this package, fix this mistake, and add a new package version to our history. I'm almost certain that there are minor mistakes. Users who still work with the old release (for example because of a data freeze) should be able to report issues and get a more accurate version of the v54.1_p1.

For me this is a bit like LTS versions of large software projects. Old LTS releases still can get patches, even when they lack far behind the latest release. I consider each AADR release an LTS version that is worth preserving. Many research papers cite specific AADR releases. I think we provide a valuable service by keeping them around.

[stschiff]

Yes makes sense!

[nevrome]

I added a PR for the HO version of v62 now as well: #8