Roocs for CWT - Questions and decisions

[agstephens]

We need to decide the following:

• how would an ESGF user discover the service?
  • do we use Intake catalogs / or just open up all the data?
    • links added to metadata records by the publisher?
• do we enable the fixing capability?
  • switch off completely?
  • need to double-check issues around xarray assumptions/features:
    • the _FillValue issue
    • the coordinates issue
• how do we track/monitor/report usage?
• which projects/activities should be available through the WPS?
  • CMIP6/CMIP5 - and any others that are open data?
    • would need more tests in the codebase
• what should the URL be: https://esgf-compute.ceda.ac.uk ?
• build support for access through an API KEY into the first release:
  • can we re-use other middleware tools to manage load/access/allocation?
• can we enable GET requests run by rook:
  • from a remote ESGF Data Node?
  • from a remote ESGF Compute Node?
    • might just return URLs that are passed to the user
  • would need a register of services
  • would need limits on how much can be cached/requested
• should we support URIs that include the service/host ID?

[agstephens]

@cehbrecht: we are starting to think about how to deploy the "rook" compute node for ESGF (at CEDA). What other decisions do you think we need to make? Please add to the list above. Thanks

[cehbrecht]

What other decisions do you think we need to make?

Just some thoughts:

• integrate the rook subsetting capability with STAC catalog?
• involve C4I as test user
• we need a distributed installation to cover the data pool: CEDA, DKRZ, IPSL, ...
  • ... that would also mean we do not have a common rook URL like in copernicus.
  • Datasets can be subsetted by 0-n different rook instances ...

[agstephens]

Thanks @cehbrecht, maybe I misheard you earlier. I think the STAC catalog would be a good place to start, but it is not ready yet.

We should definitely engage with C4I as a user.

And yes, we will need a distribution installation to cover the data pool! But we might try the CEDA version first as a proof-of-concept to show off to people. 👍