Skip to content

Add AppView support for new array format types and ndarray v1.1.0 annotations #30

@maxine-at-forecast

Description

@maxine-at-forecast

Context

The lexicon repo (forecast-bio/atdata-lexicon#21) has added new array format tokens and shim schemas:

New array format tokens

  • sparseBytes — Scipy sparse matrix format (CSR/CSC/COO)
  • structuredBytes — Numpy structured array format (compound dtypes)
  • arrowTensor — Arrow tensor IPC format
  • safetensors — HuggingFace safetensors format

New shim schemas

  • schemas/sparse_shim.json — sparse matrix bytes
  • schemas/structured_shim.json — structured array bytes
  • schemas/arrow_tensor_shim.json — Arrow tensor bytes
  • schemas/safetensors_shim.json — safetensors bytes
  • schemas/dataframe_shim.json — Parquet-encoded tabular data

NDArray shim v1.1.0

  • schemas/ndarray_shim_v1.1.0.json — adds optional dtype, shape, and dimensionNames annotation fields

Tasks

  • Update AppView schema validation to accept new arrayFormat known values
  • Handle new shim types in schema resolution (resolveSchema XRPC)
  • Support ndarray v1.1.0 annotation fields in schema validation/display
  • Update any UI components that display array format information
  • Add tests for AppView handling of new format types

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions