[core] add metaxy_deleted_at column

[geoHeil]

resolves: #245

[geoHeil]

• deletion documentation #511
• deletion dagster integration #510
• hard deletions cli integration #509
• cleanup ibis/clickhouse and use metadata for deletions where available #508
• hard delete lance #507
• hard delete delta #506
• hard delete for ibis #505
• hard deletions #504
• [core] add metaxy_deleted_at column #502 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[github-actions]

Coverage Report (Python 3.11) •

File	Stmts	Miss	Cover	Missing
_testing/parametric
metadata.py	155	37	76%	124–125, 130, 133, 135–136, 138, 141, 147–149, 151, 155–156, 158–159, 164–165, 168–169, 171, 174, 176, 326, 361, 532, 585, 598, 608, 613, 615, 743–745, 750, 759, 787
ext/sqlmodel
plugin.py	138	33	76%	116–118, 193–194, 198, 200, 202–204, 206, 208, 211, 214, 216–217, 220–221, 224–226, 229, 233, 482, 489, 492–493, 511, 516, 522, 532, 534–535
metadata_store
base.py	476	145	69%	3, 5–9, 11–15, 17–19, 25–27, 32, 36, 49–51, 56–58, 65, 68, 87, 89, 94, 99, 105, 110, 122, 150, 157, 160, 162, 232–233, 246–247, 260, 381, 442, 465, 551, 653, 710, 739, 741–742, 802, 879, 962–963, 1006–1007, 1010–1011, 1028, 1058–1059, 1064–1065, 1086, 1091, 1103, 1108, 1140, 1157, 1175, 1186, 1200, 1224, 1238, 1242, 1255, 1265–1266, 1290, 1317, 1352, 1386, 1443, 1517, 1533, 1540, 1548, 1554, 1558, 1578, 1625, 1654, 1679, 1713, 1739, 1752, 1754, 1811, 1890, 1892, 1895–1896, 1899–1900, 1904, 1914, 1927, 1929, 1931–1933, 1938–1939, 1942–1943, 1945–1947, 1949, 1952–1953, 1956–1959, 1962, 1971–1972, 1974–1975, 1978, 1981, 1983–1985, 1989–1990, 1993–1995, 1998, 2000, 2004
models
constants.py	34	32	5%	6, 9, 12, 18–19, 21–22, 24–25, 27–28, 30–31, 33–34, 43–44, 49–50, 52–53, 55–56, 58–59, 63, 77, 80, 96, 110, 128, 152
versioning
engine.py	205	52	74%	1, 3–7, 9–10, 12–13, 19–24, 32, 48, 55–56, 58, 60–61, 68–69, 74, 83, 100–105, 109, 123, 141, 177, 290, 412, 418, 454, 550, 585, 589, 613, 658, 711, 730, 733, 755, 759, 764
upstream_preparer.py	57	1	98%	146
TOTAL	8598	3397	60%

Tests	Skipped	Failures	Errors	Time
1969	22 💤	0 ❌	0 🔥	6m 35s ⏱️

[github-actions]

Coverage Report (Python 3.13) •

File	Stmts	Miss	Cover	Missing
_testing/parametric
metadata.py	155	37	76%	124–125, 130, 133, 135–136, 138, 141, 147–149, 151, 155–156, 158–159, 164–165, 168–169, 171, 174, 176, 326, 361, 532, 585, 598, 608, 613, 615, 743–745, 750, 759, 787
ext/sqlmodel
plugin.py	138	33	76%	116–118, 193–194, 198, 200, 202–204, 206, 208, 211, 214, 216–217, 220–221, 224–226, 229, 233, 482, 489, 492–493, 511, 516, 522, 532, 534–535
metadata_store
base.py	476	145	69%	3, 5–9, 11–15, 17–19, 25–27, 32, 36, 49–51, 56–58, 65, 68, 87, 89, 94, 99, 105, 110, 122, 150, 157, 160, 162, 232–233, 246–247, 260, 381, 442, 465, 551, 653, 710, 739, 741–742, 802, 879, 962–963, 1006–1007, 1010–1011, 1028, 1058–1059, 1064–1065, 1086, 1091, 1103, 1108, 1140, 1157, 1175, 1186, 1200, 1224, 1238, 1242, 1255, 1265–1266, 1290, 1317, 1352, 1386, 1443, 1517, 1533, 1540, 1548, 1554, 1558, 1578, 1625, 1654, 1679, 1713, 1739, 1752, 1754, 1811, 1890, 1892, 1895–1896, 1899–1900, 1904, 1914, 1927, 1929, 1931–1933, 1938–1939, 1942–1943, 1945–1947, 1949, 1952–1953, 1956–1959, 1962, 1971–1972, 1974–1975, 1978, 1981, 1983–1985, 1989–1990, 1993–1995, 1998, 2000, 2004
models
constants.py	34	32	5%	6, 9, 12, 18–19, 21–22, 24–25, 27–28, 30–31, 33–34, 43–44, 49–50, 52–53, 55–56, 58–59, 63, 77, 80, 96, 110, 128, 152
versioning
engine.py	205	52	74%	1, 3–7, 9–10, 12–13, 19–24, 32, 48, 55–56, 58, 60–61, 68–69, 74, 83, 100–105, 109, 123, 141, 177, 290, 412, 418, 454, 550, 585, 589, 613, 658, 711, 730, 733, 755, 759, 764
upstream_preparer.py	57	1	98%	146
TOTAL	8598	3397	60%

Tests	Skipped	Failures	Errors	Time
1969	22 💤	0 ❌	0 🔥	7m 26s ⏱️

[github-actions]

Coverage Report (Python 3.12) •

File	Stmts	Miss	Cover	Missing
_testing/parametric
metadata.py	155	37	76%	124–125, 130, 133, 135–136, 138, 141, 147–149, 151, 155–156, 158–159, 164–165, 168–169, 171, 174, 176, 326, 361, 532, 585, 598, 608, 613, 615, 743–745, 750, 759, 787
ext/sqlmodel
plugin.py	138	33	76%	116–118, 193–194, 198, 200, 202–204, 206, 208, 211, 214, 216–217, 220–221, 224–226, 229, 233, 482, 489, 492–493, 511, 516, 522, 532, 534–535
metadata_store
base.py	476	145	69%	3, 5–9, 11–15, 17–19, 25–27, 32, 36, 49–51, 56–58, 65, 68, 87, 89, 94, 99, 105, 110, 122, 150, 157, 160, 162, 232–233, 246–247, 260, 381, 442, 465, 551, 653, 710, 739, 741–742, 802, 879, 962–963, 1006–1007, 1010–1011, 1028, 1058–1059, 1064–1065, 1086, 1091, 1103, 1108, 1140, 1157, 1175, 1186, 1200, 1224, 1238, 1242, 1255, 1265–1266, 1290, 1317, 1352, 1386, 1443, 1517, 1533, 1540, 1548, 1554, 1558, 1578, 1625, 1654, 1679, 1713, 1739, 1752, 1754, 1811, 1890, 1892, 1895–1896, 1899–1900, 1904, 1914, 1927, 1929, 1931–1933, 1938–1939, 1942–1943, 1945–1947, 1949, 1952–1953, 1956–1959, 1962, 1971–1972, 1974–1975, 1978, 1981, 1983–1985, 1989–1990, 1993–1995, 1998, 2000, 2004
models
constants.py	34	32	5%	6, 9, 12, 18–19, 21–22, 24–25, 27–28, 30–31, 33–34, 43–44, 49–50, 52–53, 55–56, 58–59, 63, 77, 80, 96, 110, 128, 152
versioning
engine.py	205	52	74%	1, 3–7, 9–10, 12–13, 19–24, 32, 48, 55–56, 58, 60–61, 68–69, 74, 83, 100–105, 109, 123, 141, 177, 290, 412, 418, 454, 550, 585, 589, 613, 658, 711, 730, 733, 755, 759, 764
upstream_preparer.py	57	1	98%	146
TOTAL	8598	3397	60%

Tests	Skipped	Failures	Errors	Time
1969	22 💤	0 ❌	0 🔥	7m 40s ⏱️

[github-actions]

Test Results (Python 3.10)

1 969 tests   1 947 ✅  6m 33s ⏱️
    1 suites     22 💤
    1 files        0 ❌

Results for commit ab1b584.

♻️ This comment has been updated with latest results.

[github-actions]

Coverage Report (Python 3.10) •

File	Stmts	Miss	Cover	Missing
_testing/parametric
metadata.py	155	37	76%	124–125, 130, 133, 135–136, 138, 141, 147–149, 151, 155–156, 158–159, 164–165, 168–169, 171, 174, 176, 326, 361, 532, 585, 598, 608, 613, 615, 743–745, 750, 759, 787
ext/sqlmodel
plugin.py	138	33	76%	116–118, 193–194, 198, 200, 202–204, 206, 208, 211, 214, 216–217, 220–221, 224–226, 229, 233, 482, 489, 492–493, 511, 516, 522, 532, 534–535
metadata_store
base.py	476	145	69%	3, 5–9, 11–15, 17–19, 25–27, 32, 36, 49–51, 56–58, 65, 68, 87, 89, 94, 99, 105, 110, 122, 150, 157, 160, 162, 232–233, 246–247, 260, 381, 442, 465, 551, 653, 710, 739, 741–742, 802, 879, 962–963, 1006–1007, 1010–1011, 1028, 1058–1059, 1064–1065, 1086, 1091, 1103, 1108, 1140, 1157, 1175, 1186, 1200, 1224, 1238, 1242, 1255, 1265–1266, 1290, 1317, 1352, 1386, 1443, 1517, 1533, 1540, 1548, 1554, 1558, 1578, 1625, 1654, 1679, 1713, 1739, 1752, 1754, 1811, 1890, 1892, 1895–1896, 1899–1900, 1904, 1914, 1927, 1929, 1931–1933, 1938–1939, 1942–1943, 1945–1947, 1949, 1952–1953, 1956–1959, 1962, 1971–1972, 1974–1975, 1978, 1981, 1983–1985, 1989–1990, 1993–1995, 1998, 2000, 2004
models
constants.py	34	32	5%	6, 9, 12, 18–19, 21–22, 24–25, 27–28, 30–31, 33–34, 43–44, 49–50, 52–53, 55–56, 58–59, 63, 77, 80, 96, 110, 128, 152
versioning
engine.py	205	52	74%	1, 3–7, 9–10, 12–13, 19–24, 32, 48, 55–56, 58, 60–61, 68–69, 74, 83, 100–105, 109, 123, 141, 177, 290, 412, 418, 454, 550, 585, 589, 613, 658, 711, 730, 733, 755, 759, 764
upstream_preparer.py	57	1	98%	146
TOTAL	8598	3397	60%

Tests	Skipped	Failures	Errors	Time
1969	22 💤	0 ❌	0 🔥	6m 33s ⏱️

[Copilot]

Pull request overview

This PR implements soft-delete functionality for metadata rows, allowing rows to be marked as deleted via a timestamp without physical removal from the store.

• Adds a new metaxy_deleted_at column that defaults to NULL for active rows
• By default, soft-deleted rows are filtered out in read_metadata() calls
• Adds an include_deleted parameter to opt-in to viewing deleted rows
• Ensures backward compatibility with legacy data that lacks the new column

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
src/metaxy/models/constants.py	Adds METAXY_DELETED_AT constant and includes it in system column sets
src/metaxy/metadata_store/base.py	Implements soft-delete filtering logic in read_metadata() and adds default NULL value in _add_system_columns()
src/metaxy/versioning/engine.py	Adds METAXY_DELETED_AT to allowed system columns for upstream joins and moves imports to top
src/metaxy/metadata_store/ibis.py	Adds .lazy() call to ensure consistent LazyFrame behavior with Narwhals
tests/metadata_stores/test_provenance_golden_reference.py	Adds comprehensive test for soft-delete behavior with tombstone records
src/metaxy/_testing/parametric/metadata.py	Updates test helpers to include metaxy_deleted_at column with NULL default
tests/ext/__snapshots__/test_sqlmodel.ambr	Updates snapshots to include the new metaxy_deleted_at field in all records

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[danielgafni]

Looking good overall! Just small comments

[danielgafni]

LGTM!

[geoHeil]

should we already merge this or get the whole stack ready first?

[danielgafni]

Actually @geoHeil the sqlalchemy and sqlmodel plugins and tests have to be updated as well

[geoHeil]

Actually @geoHeil the sqlalchemy and sqlmodel plugins and tests have to be updated as well

added - tests were green. would we need any additional tests or is that good?

[danielgafni]

wrong PR sorry

[danielgafni]

Ok cool, let's merge it a little later as part of the whole stack - I don't want to redo my migrations for Anam's prod right now

[danielgafni]

I think it's worth noting that previous historical materializations of the same feature version will be effectively removed from the output as well.

[geoHeil]

You mean like:

include_soft_deleted: If `True`, include soft-deleted rows in the result. Previous historical materializations of the same feature version will be effectively removed from the output otherwise.