Is it possible to add new rows to an existing loom analysis?

[fsaad]

After ingesting an initial dataset with N rows into loom using loom.ingest and then running loom.infer, is it possible to then ingest new rows N+1 ... N+d and then run loom.infer to incorporate the new d data points into the CrossCat model? Or does one have to restart the entire analysis if new rows arrive after the initial ingest/infer step?

[fritzo]

IIRC loom.runner.infer does support incorporating new data onto an existing latent. This streaming inference is not as fully supported as batch inference, and Python support may be limited since we never used it in practice. But the low-level C++ engine fully supports streaming and batch incorporate. Loom's implementation also has more overhead than cgpm-style incorporate/unincorporate operators.

To add new samples to a model, you should be able to use loom.runner.infer with streaming rows_in and assign_out (for fully streaming inference), or a carefully constructed checkpoint_in for batch incorporate. You'll want to set config['schedule']['extra_passes'] = 0 to ensure data are only added (never removed) from the latent.