Add a backend for apache Avro

[ahasha]

I use Avro as a standard data format on most of my projects, and have been working on an Avro backend as a way of learning the odo project. I have a patch ready, and am just waiting on approval from my company's open source office to submit it for feedback.

[cpcloud]

@ahasha Excellent! We would happily review and likely accept such a patch :)

[ahasha]

One question -- how do you guys handle backend-specific dependencies? It looks like they go in the test requirements of conda.recipe/meta.yaml. Anywhere else I need to register them?

[cpcloud]

@ahasha You can also put them in https://github.com/blaze/odo/blob/master/recommended-requirements.txt if you like

[ahasha]

Quick question -- Are there patterns I should look at for ensuring invertibility when you transition from one data model to another and back? It seems like a hard problem in general, as some data models are less expressive than others. Pandas seems particularly difficult since it does a lot of implicit type casting that can destroy information.

Is there a way to attach a master datashape through conversions to use for disambiguation?

[cpcloud]

@ahasha The odo function accepts a dshape argument which is passed through every convert (and append) function. Is that enough to do what you want?

[ahasha]

I guess I'll find out! Just wanted to see if there were any examples for me to look at -- I get the sense you guys have built up some expertise with the mysterious world of implicit type conversions in pandas dataframes, and I'm a little intimidated to get started without something to look at.

I found you participating in a discussion of using Avro as a serialization format for pandas dataframes way back when (pandas-dev/pandas/pull/3525) saying "Avro looks kind of insane". How fresh is your memory of what made you say that?

[jreback]

see also here: pandas-dev/pandas#11752

[cpcloud]

@ahasha That was awhile ago :) I haven't looked at Avro in quite some time. I have zero memory of why I said that.