Skip to content

Adding an Avro backend #386

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ahasha wants to merge 39 commits into
base: master
Choose a base branch
from
Open

Adding an Avro backend #386

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
932bdde
Preliminary commit for Avro backend.
Nov 19, 2015
b2091f4
Adding test for avro to iterator
Nov 19, 2015
4cbe99f
Enabling read and write, adding append methods.
Nov 25, 2015
e07cfa7
Adding dependencies to build files
Nov 30, 2015
6cba475
Adding a convert edge from iterator back to avro
Dec 3, 2015
9ded2e1
Ensuring avro backend is registered on import
Dec 3, 2015
d7496ba
Fixing version problem with toolz
Dec 3, 2015
2adb2df
Adding support for array types
Dec 18, 2015
31ed159
Merge branch 'master' into avro
Dec 18, 2015
491c4d1
Attempting to fix travis build
Dec 18, 2015
2eeba0a
Reformatting test data
Dec 18, 2015
471becc
Adding conda selector for avro, since it is not Python 3 compatible
Dec 18, 2015
e05d1e1
Small fixes from code review.
Dec 19, 2015
31821d2
Converting tests to pytest idiom.
Dec 19, 2015
3773348
Changing import order to ensure test skip if avro not installed
Dec 19, 2015
2c0246a
Final small changes requested from code review
Dec 19, 2015
0d35449
Merge branch 'master' into avro
Dec 19, 2015
01b67a4
Use multipledispatch for schema discovery
Dec 22, 2015
a583e55
Bug fix for boolean types
Dec 22, 2015
eb004c7
Cleanup
Dec 29, 2015
58d4e6e
Adding dshape to avro schema generation, with doctest
Dec 30, 2015
64753e2
Bumping datashape version to fix build
Jan 4, 2016
d575ead
Merge branch 'master' into avro
Jan 7, 2016
5c2d082
Attempt to fix build for python3
Jan 29, 2016
78a591f
Whitespace error.
Jan 29, 2016
ee5e6f3
Python 3 support
Jan 29, 2016
ff32e9e
One more python3 bug I missed.
Jan 29, 2016
cbec204
Changing doctest not to be format sensitive.
Jan 29, 2016
4ebbf9b
Merge branch 'master' into avro
Feb 2, 2016
060c2b4
Updating avro type mappings
Feb 2, 2016
32900b9
Changing AVRO.uri attribute to AVRO.path, and allowing codec keyword …
Feb 2, 2016
95724cf
Dropping requirement for schema to be defined for new AVRO resource o…
Feb 2, 2016
76f2a72
Merge branch 'master' into avro
Jun 27, 2016
264fca2
Adding fastavro to meta.yaml
Jun 27, 2016
d9ca7ab
Merge branch 'master' into avro
Oct 8, 2016
1e56c1e
Correcting test failures due to incompatibility with pytest 3.0.3 as …
Oct 8, 2016
fd48794
Accidentally removed avro installation from travis.yml. Fixing
Oct 8, 2016
aba333a
Merge branch 'master' into avro
Jan 23, 2017
b66e7e3
Updating to new requirements tracking structure.
Jan 23, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions conda.recipe/meta.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,9 @@ requirements:

test:
requires:
- avro # [py27]
Copy link
Author

@ahasha ahasha Jan 7, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do I need to do more than this to avoid breaking the build?

Copy link
Member

@mrocklin mrocklin Jun 24, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a separate fastavro package?

Copy link
Author

@ahasha ahasha Jun 24, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is... not sure why this didn't break the build. Should it be added here?

Copy link
Member

@mrocklin mrocklin Jun 24, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This recipe affects building conda packages, which isn't actually used in testing. We would run into this issue upon releasing.

- avro-python3 # [not py27]
- fastavro
- pytest
- h5py
- pytables >=3.0.0
Expand Down
2 changes: 1 addition & 1 deletion etc/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
datashape >= 0.5.0
numpy >= 1.7
pandas >= 0.15.0
toolz >= 0.7.3
toolz >= 0.7.4
multipledispatch >= 0.4.7
networkx >= 1.0
dask >= 0.11.1
3 changes: 3 additions & 0 deletions etc/requirements_avro.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
fastavro==0.12.1
avro==1.8.1; python_version < '3'
avro-python3==1.8.1; python_version > '2.7'
3 changes: 3 additions & 0 deletions etc/requirements_ci.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
avro==1.8.1; python_version < '3'
avro-python3==1.8.1; python_version > '2.7'
bcolz==0.12.1
bokeh==0.12.3
boto3==1.4.1
Expand Down Expand Up @@ -48,3 +50,4 @@ tblib==1.3.0
toolz==0.8.0
tornado==4.4.2
zict==0.0.3
fastavro==0.12.1
3 changes: 3 additions & 0 deletions extra-requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,7 @@ sas7bdat
paramiko
pywebhdfs
boto
avro; python_version < '3'
avro-python3; python_version > '2.7'
fastavro
sqlalchemy-redshift
3 changes: 3 additions & 0 deletions odo/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,10 +61,13 @@
from .backends.sparksql import SparkDataFrame
with ignoring(ImportError):
from .backends.url import URL
with ignoring(ImportError):
from .backends.avro import AVRO
with ignoring(ImportError):
from .backends.dask import dask



restart_ordering() # Restart multipledispatch ordering and do ordering


Expand Down
Loading