Round-tripping

[drtconway]

Hi Oxbow,

Fantastic project - thanks!

I'm working on a tool for doing some processing on VCFs, to produce a new VCF. My current implementation is already built on noodles + datafusion, so I'm keen to use oxbow to eliminate a pile of VCF -> arrow code.

Do you have concrete plans to produce code for going from RecordBatches to VCF?

Also, because of the way sorting works in datafusion, I find I need to keep an integer chrom_id column so that things sort in the desirable chr1, chr2, ... chr9, chr10, ... way. Is this something you would consider adding?

Thanks,
Tom.

[nvictus]

I find I need to keep an integer chrom_id column so that things sort in the desirable chr1, chr2, ... chr9, chr10, ... way.

We use dictionary encoding for chromosome columns, which is basically an Enum. Wouldn't that work in lieu of an integer column?

Do you have concrete plans to produce code for going from RecordBatches to VCF?

Writers that consume record batches is on the roadmap but we don't have a timeline for it yet.

[drtconway]

Thanks for the prompt response!

The dictionary encoding sounds like it would be perfect for the job, but the datafusion sorting still sorts by the lexical string unfortunately.

Great to hear writing is on the roadmap. When I'm more familiar with the library, its something I'd certainly consider writing as a contribution.

The other thing I forgot is it would be really nice to have a row ID - with the line number in the VCF (counting from e.g. 1 for the first variant - i.e. skipping the metadata.

I guess a neat mechanism for allowing these extra fields to be optional would be desirable. Actually, adding them to the Field enum would probably be ok, even if they are not part of the default field names, so they could be explicitly requested.

Tom.