Support Zero Copy for Merge/Project

[agavra]

In the current implementation Merge and Project both materialize the resulting record. This means that if you apply both merge and project you are materializing the intermediate state unnecessarily. We should support "views" on underlying imprint records so that it is possible to avoid this side effect.

[hemantkrsh]

@agavra I would like to take this up.

The project() and merge() methods will by default return ImprintRecordView and a to_record()/materialize() method returns the materialized record.
ImprintRecordView is the logical view over the record which describes - how to get this new record from the old one - operation.

[agavra]

@hemantkrsh I think this one requires a little bit of design, it would be good to agree on the proposed APIs and behaviors before starting implementation on this. I don't think it needs a full doc, but we should have some discussion on this ticket and agree on the plan. Feel free to expand on your proposal though and I'll take a deeper look!

[hemantkrsh]

@agavra Agreed. Let me come with the APIs and the internals of its working.

[expanded-for-real]

@agavra I've been thinking about this a lot on the java side lately and I think it depends on the expected use-cases for merge and project. There's several ways this could go but it's possible to create a streaming framework that would allow you to chain merge, project, and other operations together, while lazily evaluating them up to a terminal operator (like materialize() or build(), etc).

The question is - what does one usually do after merge - more operations like projection, or just materialize anyways? The streaming framework offers a lot of flexibility but it might not be useful if the usual intent is to just merge two records and then send them off.

[agavra]

I think a merge then project is likely to be pretty common though anything much more complicated than that is less likely, so some kind of basic way to compose these operations is tempting to me.

I think a basic streaming pipeline that composes "views" is nice, and then delay the materialization until you call build. We could even KISS and just hard-code a merge+project operator in a single operator to get us 80% of the way there.

[hemantkrsh]

@agavra This is what I'm thinking.

By default, any operation on ImprintRecord returns <ImprintView, ImprintError>.

Old APIs:
fn merge(&self, other: &ImprintRecord) -> Result<ImprintRecord, ImprintError>
fn project(&self, field_ids: &[u16]) -> Result<ImprintRecord, ImprintError>

New APIs:
fn merge(&self, other: &ImprintRecord) -> Result<ImprintRecord, ImprintError>
fn project(&self, field_ids: &[u16]) -> Result<ImprintRecord, ImprintError>

Both ImprintRecord and ImprintView will implement a common trait, View.

ImprintView holds the list of operations (logical view) supporting lazy evaluation. Operations are of two types:
MergeOperation is a struct with left and right both of View type.
ProjectOperation is a struct of field_ids and a View type.

ImprintView will have materialize() which will materialize the operations one by one.
Internally, each operation returns a Result<(payload, DirectoryEntry),ImprintError> to the operation above.
In the future, this is where further optimizations could be done(if required), like using the project operation field_ids from the top of the operations to push down to all the operations down the stack.
For example, with merge, move the field_ids if they are projected, and with project, select the field_ids.

Let me know your thoughts on this approach.

[agavra]

I'm trying to think this through from a user experience perspective. I'd like for the user to only even need to think about a single abstraction (e.g. ImprintRecord) and not the difference between that and a "view". I'm not sure if there's a good way to implement this without introducing ImprintView like you suggest.

@expanded-for-real's comment got me thinking about whether this is really necessary, and I'm starting to think that it may not be. Instead I think if we just added merge_and_project to the operations we'd cover 80% of the use cases without introducing more API surface area. That also makes the design for this ticket much easier.

@hemantkrsh what do you think? If you're on board we can go ahead with a PR for that without further design as its a bit more straightforward.

[hemantkrsh]

Completely agree @agavra if the merge_and_project covers ~80% of the use cases then there is no need to complicate the API design. Furthermore, given that the serde is aimed towards streaming use cases, unlike batch, I don't expect many chained operations.
I will get back with the PR for this.