API for writing containers incrementally

[alexbiehl]

I am constructing huge AVRO containers that I have to upload in multiple chunks. To help the use case It would be great if we had an API that let's you write containers incrementally:

packContainer
  :: (ToAvro a) 
  => Codec
  -> Schema -- ^ Writer schema
  -> ByteString -- ^ Sync bytes
  -> (Builder -- ^ Container header
       , [a] -> Builder -- ^ A function to feed a's and turn them into valid container blocks
       )

This allows to consume from a streaming source and fill buffers incrementally.

I can contribute if you think it'd be worthwhile.

[AlexeyRaga]

@alexbiehl cannot this one be used?

encodeContainerWithSync :: ToAvro a => Codec -> Schema -> BL.ByteString -> [[a]] -> BL.ByteString

Given that lists in Haskell are lazy, you can lazily produce a lazy bytestring and lazily upload it in chunks...

At least this was the way we've done it in the past :)

[AlexeyRaga]

Sorry, accidentally clicked the wrong button

[alexbiehl]

Unfortunately I don’t have a lazy list, my messages are coming from a Kafka topic.

[AlexeyRaga]

@alexbiehl Ah, we've been using lazy IO for it (both with conduit and with bare unsafeInterleaveIO to get these lazy lists.
Yeah, for sure, that's be a valuable addition for people who don't want to mess with lazy IO!