[integration / dagster] surface existing metadata - do not re-calcualte counts

[geoHeil]

Currently the record count per partition feature triggers a narwhals count/collect operation (triggers a possibly costly calculation). Instead, the previous UPSERT operation usually returns some metadata about added, modified, deleted record counts. We should re-use that pre-existing metadata

[danielgafni]

I was thinking about this again...

We are calculating stats in some other places besides Dagster, e.g. in CLI commands.

It would be nice to make a shared API for that, e.g. FeatureStatsCalculator or smth like this.

This API could cache stats calculations, which should automatically solve some of these problems in many places.

We need to check if nw.Expr is hashable. If it's not, then not sure how would we cache the filters argument.

[geoHeil]

In the end the filters translate to a SQL expression (text) which is hashable. We could LRU cache that.

But the initial point was not about caching arbitrary calculations. But re-using the DML metadata of the DBs to use the rows affected instead of re-calculating it.

[danielgafni]

in the end the filters translate to a SQL expression (text) which is hashable

It would be great if there was a method to do this reliably, especially if it's order-independent. Right now I'm not aware of a way to do it. We should invest in this since it's also needed for serialization.

But re-using the DML metadata of the DBs to use the rows affected instead of re-calculating it.

This sounds good too, I'm not sure if we can get it out of Ibis tho? Probably we can.

With DeltaLake I know we can use a DeltaLake object as destination to .write_delta, it will at least hold the right version. Other metadata like affected rows and stats might not be available (although the row count is easy to get on our side).

---

That being said, querying the table after writes is much more simple and backend-agnostic, even if there is some performance penalty for doing it.

[geoHeil]

Another possibility would be to write such statistics tables explicitly from metaxy (and manage/update) them.

Exactly - order independence is a problem. We would have to cache on the AST level to do it right. But with SQLGlot it should not be too hard to get to that.

[danielgafni]

Another possibility would be to write such statistics tables explicitly from metaxy (and manage/update) them.

IMO that's very unlikely to be implemented since it's easy to mess it up and have these stats tables in the wrong state, especially during runtime errors etc. I'd rather use the metadata itself as source of truth.