Skip to content

Virtual hackathon March 30th April 3rd 2020

Jump to bottom
m214089 edited this page Apr 3, 2020 · 25 revisions

The week planned

Participants: Katie (ETHZ), Jonas (ETHZ), Juergen (DWD), Uwe (MPIM), Luis (MPIM)

Next meeting: April 3rd, 2020 - 9:30

Remember to branch off develop!

git-lfs for input data

  1. Luis: Contact DKRZ about the howto taking into account the 2.5 TB space requirement:
    • have send email after the vc to get support, no answer yet
    • 10 TB are available and I know all the required howtos
    • There is a restriction of 5 GB max per file
    • DONE: Repo created and tested (Jonas, Luis)
  2. Jonas, Juergen: work on the directory hierarchy/layout, ASTER data need to be compressed/decompressed on the fly
    • Script to de/compress large files with xz is in repo, can be easily adapted to other compressors (gzip, etc)
  3. All: Review layout and get implemented at DKRZ
    • DONE: extpar-input-data transfered to DKRZ. Works fine now.
  4. Jonas: contact Burkhardt Rockel and inform about status of repository.
    • Added Burkhard to group extpar-data

Documentation:

  1. Add explanation of data repository in Extpar manual section 1.1
  2. Add mention of data repository in Extpar code repository README file

Functional based split of topography generation of GLOBE data

  1. Uwe, Jonas: Finalize split by transferring the pre-calculated data (reduced grid) into a file (read/write) to split the application into a pre-processing and remapping part
    • write of reduced grid is done
    • read routine for reduced grid done
  2. All: Validate

Documentation

  1. Update Extpar manual with description of new topography calculation workflow: Section 2.1, Figures 2-4

Get a more simple, single Python-script based handling of the trivial interpolation cases

  1. Jonas, Juergen: Pick up the albedo and/or emissivity case for introduction
    • extpar_alb_to_buffer is ready to review in branch albedo_python in directory src_python.
    • The requirements are:
      • python3.6
      • netCDF4
        All other modules are standards and should be available on all systems.
  2. All: Review
    • DONE
  3. All: Roadmap for removing the respective Fortran apps

Documentation:

  1. Update README with clearer explanation of python replacement of Fortran.
  2. Update README.compile_run with correct dependencies and recommended version for dependencies
  3. Update Extpar manual sections 2 and 4
  4. Add python/shell/cdo rules to Development guidelines

mmap based disk-cache to lower memory footprint of consistency check

  1. Luis: Implement with cases with traditional and new implementation:
    • provided all code up to the extpar programming interface and added to the extpar build process
    • wrote python tool replacing the old scheme by a switchable version with the new scheme
    • code compiles, logical switch introduced
    • updated/merged to new build environment, one more problem pending with Sergey
  2. All: Review and validate functionality and correctness

Documentation:

  1. Update Extpar manual sections 2 and 4

New building

  1. Luis: Contact Sergey for a review and potential implementation (outsourcing):
    • Sergey got a tar and is looking into it
    • Sergey started the implementation and is half way through
    • implementation finished
    • adapt option snippets to callable configure wrapper scripts
    • another update of install... is required, for installing libcdi
    • and again another update for the install... script required, done and tested
    • a problem is pending for Sergey
  2. Luis: Provide links to repos Sergey has been transferring:
  3. Come back to discussion on Wednesday

Documentation and reviewing

  1. Katie: Update and improve documentation with respect to all new developments
    • Provided detailed steps of documentation required for each development.
  2. Katie: What is the status of a hosting-transfer from github to DKRZ
    • Decision still to be discussed at COSMO scientific committee. However, we can proceed by setting up an exemplary repository including the Jenkins workflow.
    • had to answer a lot of questions triggered by Jean-Marie in supporting the case
    • a test repo (read only) is available under git@gitlab.dkrz.de:cosmo-sw/extpar.git - it is mirrored once per hour as long as my computer is running
  3. Katie: Gatekeeping

Managing Extpar dependencies

  1. Katie: Update testsuite to python 3
    • Branch created (testsuite-python3) with updated code. Does not currently run on Mistral, but it seems to make sense to wait until the new build system is in place before figuring it out.
  2. Katie/Jonas: Make install-extpar-swstack work for multiple machines (dependent on build system plans):
    • ** yet another update necessary: I (Luis) have to update and pass once more a new version**

ECCI landuse dataset

  1. Jürgen: provide code with new landuse categories
    • DONE
    • Question: have you been adding the raw data processing to the respective extpar directory?
  2. Katie: review code and merge into develop
  3. Jürgen, Jonas: put dataset into the Extpar data repository.

Interdependencies of above defined tasks:

  1. Sergey provides a new build system, including dependencies for all machines with Options files currently in repository.
  2. Then, testsuite-python3 branch can be fixed and merged, with proper python version set by build configuration.
  3. Then, feature-libcdi branch can be fixed and merged, with cdi library location set by build configuration.

Clone this wiki locally