Skip to content

vortex metadata store #529

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
geoHeil wants to merge 1 commit into
base: 12-09-bump_minimal_python_version_to_3.11
Choose a base branch
from
Draft

vortex metadata store #529

geoHeil wants to merge 1 commit into from

Conversation

@geoHeil
Copy link
Collaborator

@geoHeil geoHeil commented Dec 10, 2025 •
edited
Loading

resolves: #274

limitations

  • no file format just table format
  • no remote writes in python api

@geoHeil Graphite App
Copy link
Collaborator Author

geoHeil commented Dec 10, 2025 •
edited
Loading

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

@geoHeil geoHeil mentioned this pull request Dec 10, 2025
Draft
geoHeil
geoHeil commented Dec 10, 2025
View reviewed changes
src/metaxy/metadata_store/vortex.py
**kwargs: Forwarded to [metaxy.metadata_store.base.MetadataStore][].

Raises:
NotImplementedError: If a remote path (s3://, gs://, az://) is provided.
Copy link
Collaborator Author

@geoHeil geoHeil Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No remote paths supported in python API

Copy link
Collaborator Author

@geoHeil geoHeil Dec 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See https://vortex-data.slack.com/archives/C07BV3GKAJ2/p1765471543845219 from the vortex slack

geoHeil
geoHeil commented Dec 10, 2025
View reviewed changes
src/metaxy/metadata_store/vortex.py

# ===== Storage operations =====

def write_metadata_to_store(
Copy link
Collaborator Author

@geoHeil geoHeil Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a file format - not a table format like ducklake, iceberg, delta. This means some extra complexity

Copy link
Collaborator Author

@geoHeil geoHeil Dec 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://spiceai.org/blog/releases/v1.9.0 Cayenne Data Accelerator could be interesting here

Copy link
Collaborator Author

@geoHeil geoHeil Dec 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and https://github.com/spiraldb/iceberg something in JVM land

@github-actions
Copy link
Contributor

github-actions bot commented Dec 10, 2025 •
edited
Loading

Test Results (Python 3.11)

1 820 tests   1 798 ✅  6m 40s ⏱️
    1 suites     22 💤
    1 files        0 ❌

Results for commit 9839ca9.

♻️ This comment has been updated with latest results.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 10, 2025 •
edited
Loading

Coverage

Coverage Report (Python 3.11) •
FileStmtsMissCoverMissing
metadata_store
   vortex.py132993%155, 157, 184–185, 250, 286, 310, 329, 334
TOTAL8458334060% 

Tests Skipped Failures Errors Time
1820 22 💤 0 ❌ 0 🔥 6m 40s ⏱️

@github-actions
Copy link
Contributor

github-actions bot commented Dec 10, 2025 •
edited
Loading

Coverage

Coverage Report (Python 3.12) •
FileStmtsMissCoverMissing
metadata_store
   vortex.py132993%155, 157, 184–185, 250, 286, 310, 329, 334
TOTAL8458334060% 

Tests Skipped Failures Errors Time
1820 22 💤 0 ❌ 0 🔥 7m 17s ⏱️

@github-actions
Copy link
Contributor

github-actions bot commented Dec 10, 2025 •
edited
Loading

Coverage

Coverage Report (Python 3.13) •
FileStmtsMissCoverMissing
metadata_store
   vortex.py132993%155, 157, 184–185, 250, 286, 310, 329, 334
TOTAL8458334060% 

Tests Skipped Failures Errors Time
1820 22 💤 0 ❌ 0 🔥 6m 55s ⏱️

@geoHeil geoHeil force-pushed the 12-09-bump_minimal_python_version_to_3.11 branch from 4fbbd2d to 04528f7 Compare December 10, 2025 13:38
@geoHeil geoHeil force-pushed the 12-10-vortex_metadata_store branch from 37def0a to 0e889c5 Compare December 10, 2025 13:38
@geoHeil geoHeil force-pushed the 12-09-bump_minimal_python_version_to_3.11 branch from 04528f7 to 6f4f64b Compare December 10, 2025 18:01
@geoHeil geoHeil force-pushed the 12-10-vortex_metadata_store branch from 0e889c5 to c511d41 Compare December 10, 2025 18:01
@geoHeil geoHeil force-pushed the 12-09-bump_minimal_python_version_to_3.11 branch from 6f4f64b to 8b3c9a4 Compare December 10, 2025 18:02
@geoHeil geoHeil force-pushed the 12-10-vortex_metadata_store branch from c511d41 to 5362402 Compare December 10, 2025 18:02
geoHeil
geoHeil commented Dec 10, 2025
View reviewed changes
@github-actions
Copy link
Contributor

github-actions bot commented Dec 10, 2025 •
edited
Loading

Test Results (Python 3.10)

1 728 tests  +39   1 668 ✅ ± 0   6m 20s ⏱️ -8s
    1 suites ± 0      60 💤 +39 
    1 files   ± 0       0 ❌ ± 0 

Results for commit dec71ab. ± Comparison against base commit 8b3c9a4.

♻️ This comment has been updated with latest results.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 10, 2025 •
edited
Loading

Coverage

Coverage Report (Python 3.10) •
FileStmtsMissCoverMissing
metadata_store
   vortex.py1328734%44–45, 47–48, 55–56, 138–140, 142, 145, 154–159, 161, 172–173, 177, 184–185, 200, 202–206, 208, 210–212, 218–219, 221–222, 226–231, 245, 247, 249–250, 252, 254, 257, 259–264, 266–267, 271–273, 277–279, 283, 285–286, 288, 299, 301–304, 306–307, 309–310, 312, 314–316, 318–319, 321, 325, 329, 334
TOTAL8458341859% 

Tests Skipped Failures Errors Time
1815 62 💤 0 ❌ 0 🔥 6m 45s ⏱️

@geoHeil geoHeil force-pushed the 12-10-vortex_metadata_store branch 5 times, most recently from fef8fe5 to bfed88b Compare December 10, 2025 20:10
@geoHeil geoHeil force-pushed the 12-10-vortex_metadata_store branch from bfed88b to 9839ca9 Compare December 17, 2025 09:50
@geoHeil geoHeil force-pushed the 12-09-bump_minimal_python_version_to_3.11 branch from 8b3c9a4 to f2bb01f Compare December 17, 2025 09:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@geoHeil