externalize adios2py and the xarray adios2 reader#16
Merged
germasch merged 243 commits intopsc-code:mainfrom Jan 15, 2025
Merged
externalize adios2py and the xarray adios2 reader#16germasch merged 243 commits intopsc-code:mainfrom
germasch merged 243 commits intopsc-code:mainfrom
Conversation
via lambda doesn't seem to make much of a difference. In both cases, when doing just a selection, it'll be kept lazy, but say rescaling by multiplying by a scalar leads to the data being loaded. Probably something where dask comes in.
Since the backend is registered by means of pyproject.toml now
No need to have a psc.RunInfo until later
This is all about gettings things more in line with other xarray backends
In particular this requires the subtlety of actually returning xr.Variable from open_store_variable(), not xr.DataArray
keep compatibility with files missing this information, though, even though there's not much we can do, other than special casing psc so that at least doesn't break
however, keep current behavior, ie always reverse
but also keep reversing for pfd* tfd* to maintain compatibility
This isn't consistent with xarray typing, and easy enough to avoid by sticking with open_dataset(Adios2Store.open(group))
Looks like just something left over from the template
We might not want to do this in the long run, since this leads to a transitive dependency on adios2, which is binary and potentially iffy to install, so if one were to want to use pscpy without the adios2 reader, that could cause some hassles.
Codecov ReportAll modified and coverable lines are covered by tests ✅
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The idea behind this is good, since adios2py and xarray-adios2 are independently useful. The execution is horrible, though,
in that this is a unreviewable mega PR.
Basic testing succeeds, though, and whatever rough edges there are, they shouldn't be too hard to fix up.
In existing code, it'll be necessary to replace direct reading of a psc adios2 file with something like