how to read S3 resources that don't have a file extension

[gallamine]

Reading the code it seems that Odo will only read S3 resources that contain the file extension of .csv, .json or .txt. By default Hive writes data to S3 without any file extension. What is the preferred method of being able to read text files like this?

[llllllllll]

You can wrap the url in a CSV box object. For example:

import numpy as np
from odo import CSV, odo
odo(CSV('...'), np.ndarray)

[gallamine]

Tried the following:

import pandas as pd
from odo import odo
import odo
from odo import CSV
odo.resource(CSV('s3://mybucket/d=20150903/*'))

Yields:

/Users/william/anaconda/lib/python2.7/re.pyc in match(pattern, string, flags)
    139     """Try to apply the pattern at the start of the string, returning
    140     a match object, or None if no match was found."""
--> 141     return _compile(pattern, flags).match(string)
    142 
    143 def search(pattern, string, flags=0):

TypeError: expected string or buffer

This code

df = odo(CSV('s3://mybucket/d=20150903/f6373839-7775-49f6-8203-8bc3a6a59106-000000'), pd.DataFrame)

Results in:

TypeError: 'module' object is not callable

[llllllllll]

In the first case, you already have the resource, resource takes a string and turns it into an abstract resource; however, by constructing a CSV manually, you do not need to do that.

In the second case, you forgot to import the odo function from the odo module. You probably wanted to replace import odo with from odo import odo.

[gallamine]

Thanks. I'm running just the following now:

from odo import odo, CSV
import pandas as pd

df = odo(CSV('s3://mybucket/d=20150903/f6373839-7775-49f6-8203-8bc3a6a59106-000000'), pd.DataFrame)

and I still seem to be getting MDNotImplementedError errors (I can download the file via s3cmd so it definitely exists):

---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)
<ipython-input-11-c800854b8411> in <module>()
----> 1 df = odo(CSV('s3://mybucket/d=20150903/f6373839-7775-49f6-8203-8bc3a6a59106-000000'), pd.DataFrame)

/Users/william/anaconda/lib/python2.7/site-packages/odo/odo.pyc in odo(source, target, **kwargs)
     88     odo.append.append      - Add things onto existing things
     89     """
---> 90     return into(target, source, **kwargs)

/Users/william/anaconda/lib/python2.7/site-packages/multipledispatch/dispatcher.pyc in __call__(self, *args, **kwargs)
    162             self._cache[types] = func
    163         try:
--> 164             return func(*args, **kwargs)
    165 
    166         except MDNotImplementedError:

/Users/william/anaconda/lib/python2.7/site-packages/odo/into.pyc in into_type(a, b, dshape, **kwargs)
     22     with ignoring(NotImplementedError):
     23         if dshape is None:
---> 24             dshape = discover(b)
     25     return convert(a, b, dshape=dshape, **kwargs)
     26 

/Users/william/anaconda/lib/python2.7/site-packages/multipledispatch/dispatcher.pyc in __call__(self, *args, **kwargs)
    162             self._cache[types] = func
    163         try:
--> 164             return func(*args, **kwargs)
    165 
    166         except MDNotImplementedError:

/Users/william/anaconda/lib/python2.7/site-packages/odo/backends/csv.pyc in discover_csv(c, nrows, **kwargs)
    331 @discover.register(CSV)
    332 def discover_csv(c, nrows=1000, **kwargs):
--> 333     df = csv_to_dataframe(c, nrows=nrows, **kwargs)
    334     df = coerce_datetimes(df)
    335 

/Users/william/anaconda/lib/python2.7/site-packages/odo/backends/csv.pyc in csv_to_dataframe(c, dshape, chunksize, nrows, **kwargs)
    247     try:
    248         return _csv_to_dataframe(c, dshape=dshape, chunksize=chunksize,
--> 249                                  nrows=nrows, **kwargs)
    250     except StopIteration:
    251         if nrows:

/Users/william/anaconda/lib/python2.7/site-packages/odo/backends/csv.pyc in _csv_to_dataframe(c, dshape, chunksize, **kwargs)
    258 def _csv_to_dataframe(c, dshape=None, chunksize=None, **kwargs):
    259     header = {False: None, True: 0}.get(
--> 260         kwargs.pop('has_header', c.has_header), 'infer')
    261 
    262     sep = kwargs.pop(

/Users/william/anaconda/lib/python2.7/site-packages/odo/backends/csv.pyc in has_header(self)
    171     def has_header(self):
    172         if self._has_header is None:
--> 173             with sample(self) as fn:
    174                 with open(fn, mode='rb') as f:
    175                     raw = f.read()

/Users/william/anaconda/lib/python2.7/contextlib.pyc in __enter__(self)
     15     def __enter__(self):
     16         try:
---> 17             return self.gen.next()
     18         except StopIteration:
     19             raise RuntimeError("generator didn't yield")

/Users/william/anaconda/lib/python2.7/site-packages/odo/backends/csv.pyc in sample_csv(csv, length, **kwargs)
    185     should_delete = not os.path.exists(csv.path)
    186     if should_delete:
--> 187         with open_file(csv.path, mode='wb'):
    188             pass
    189     try:

/Users/william/anaconda/lib/python2.7/contextlib.pyc in __enter__(self)
     15     def __enter__(self):
     16         try:
---> 17             return self.gen.next()
     18         except StopIteration:
     19             raise RuntimeError("generator didn't yield")

/Users/william/anaconda/lib/python2.7/site-packages/odo/backends/csv.pyc in open_file(path, *args, **kwargs)
     79 @contextmanager
     80 def open_file(path, *args, **kwargs):
---> 81     f = compressed_open.get(ext(path), open)(path, *args, **kwargs)
     82     try:
     83         yield f

IOError: [Errno 2] No such file or directory: 's3://mybucket/d=20150903/f6373839-7775-49f6-8203-8bc3a6a59106-000000'

[gallamine]

As I single-step through the code it still seems that the RegexDispatcher is trying to match the regex, '^s3://.*\\.csv(\\.gz)?$' - which again is a problem since my files don't have extensions.

[cpcloud]

@gallamine Try this:

obj = S3(CSV('s3://path/to/file/without/csv.extension'))
# odo(obj, ...) or odo(..., obj)

[gallamine]

Tried @cpcloud suggestion. Get this error:

AttributeError                            Traceback (most recent call last)
<ipython-input-4-56daa64215f9> in <module>()
----> 1 obj = S3(CSV("s3://bucket/d=20150922/afe8e679-a7ac-4e51-b401-b2190d22a56b-000036"))

/Users/william/anaconda/lib/python2.7/site-packages/toolz/functoolz.pyc in memof(*args, **kwargs)
    351             return cache[k]
    352         else:
--> 353             result = func(*args, **kwargs)
    354             cache[k] = result
    355             return result

/Users/william/anaconda/lib/python2.7/site-packages/odo/backends/aws.pyc in S3(cls)
     92 @copydoc(_S3)
     93 def S3(cls):
---> 94     return type('S3(%s)' % cls.__name__, (_S3, cls), {'subtype': cls})
     95 
     96 

AttributeError: 'CSV' object has no attribute '__name__'

I can read the file, however, work if I create a custom resource:

@odo.resource.register('^s3://.*(\\.gz)?$')  # match anything in s3
def resource_s3_noext(uri, **kwargs):
    return S3(CSV)(uri)