Merge production/mlglobal.v1 branch into main
#65
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Added scripts to run EAGLE ensemble on WCOSS2
Added the capability to generate ensemble mean and spread, update grib2 utils and run_graphcast to run both mlgfs and mlgefs
This PR: - updates pyproject.toml that allows a user to pip install the mlglobal package along with its dependencies (including -graphcast). pyproject.toml allows other installation methods as well. -updates the README.md to include the installation method - adds the CC0 license (same as the ufs-weather-model) - creates src/mlglobal and introduces ic_downloader entry point to download GFS and GEFS datasets - adds a github action for linter checks.
…40) This PR: - Separated `mlgfs` and `mlgefs` attributes into two json files to fix the header issues - Updated the job cards to use Rahul's virtual env to fix the `APCP` timestamp issue - Replaced method `pygrib` with `grib2io` method in the preprocessing
This PR: - removed "0p25" from the file names - changed "ml" to "ai", only applied to output directory and file names, and keep python scripts untouched
This PR: - Updated `run_graphcast.py` and `nc2grib.py` to save output at each step - Renamed `ml` to `ai` for scripts - Changed virtual env from `mlgfs` to `aigfs` For comparison between new method and old one (write until forecasts finish), here are outputs for `20250909/18` from two methods: `/lfs/h2/emc/nems/noscrub/linlin.cui/Tests/aigfs.20250909_old` `/lfs/h2/emc/nems/noscrub/linlin.cui/Tests/aigfs.20250909_new` Here is the difference for `t2m`: ``` Difference between old and new for t2m: mean 6.616234895773232e-05, max 0.3000030517578125, min -0.20001220703125 ```
This PR calls `set_event.sh` to trigger post jobs for aigfs (see NOAA-EMC/aigfs#4). Post jobs in in NOAA-EMC/aigfs#4 create grib2 .idx files and copy fiels to COM. This PR replaces #48. --------- Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com>
This PR is a minor cleanup and housekeeping. Specifically, - updates `gitignore` to include netcdf and grib2 files so they do not get committed to the repo - updates the graphcast submodule pointer to the NOAA-EMC/graphcast instead of google-deepmind. This does not change the virtual envs. - updates the `environment.yml` and `pyproject.toml` to be consistent.
This PR: - creates an issue template for reporting and capturing operational bug reports.
This PR includes the last changes that SPA team made to - `oper/gen_aigfs_ics.py` - `oper/gen_aigefs_ics.py` to operationalize the scripts. These changes are a band-aid fix that will need a coordinated fix in the next release. A PR such as #54 would likely resolve these issues. Some key areas we will need to improve on include: - `oper/gen_aigefs_ics.py` - Remove boto initialization (caused failures during testing) and calls to download data from s3 buckets - Output netCDF file is still has `mlgefs` prefix, which is consistent with [exaigefs_prep.sh](https://github.com/NOAA-EMC/aigefs/blob/62f3cb50438e13fce5d95ddd3a09301a68ca1214/scripts/exaigefs_prep.sh#L40) - Both scripts - Remove colon characters from intermediate file names - Explicitly locate files instead of looping and globbing (#36)
Contributor
Author
|
@LinlinCui-NOAA @junwang-noaa |
This PR made two changes to `run_graphcast.py`: - Added time log for each step - Removed conversion to Bfloat16. Bfloat16 should not be used for prediction. The reduced precision greatly increases run-to-run variance. --------- Co-authored-by: Russell Manser <russell.manser@noaa.gov>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
v1.0.0 tag has been released. This tag is used in the AIGFSv1.0.3 and AIGEFSv1.0.4.
This PR:
production/mlglobal.v1intomain