Gdal::Ogr.get_driver_by_name returns nil for ESRI Shapefile

[svilenkov]

I've run into a problem inside a debian Docker image when trying to use ruby-gdal . The Gdal::Ogr.get_driver_by_name('ESRI Shapefile') returns nil on this linux platforms. According to the OGR docs the 'ESRI Shapefile' should be compiled by default. I have tested this also on a colleague's Ubuntu system and the same issues exists.

This ruby snippet should list all registered OGR drivers:

require 'gdal'
(0..Gdal::Ogr.get_driver_count-1).map { |n| Gdal::Ogr.get_driver(n).get_name }

output:

=> ["VRT", "GTiff", "NITF", "RPFTOC", "ECRGTOC", "HFA", "SAR_CEOS", "CEOS", "JAXAPALSAR", "GFF", "ELAS", "AIG", "AAIGrid", "GRASSASCIIGrid", "SDTS", "OGDI", "DTED", "PNG", "JPEG", "MEM", "JDEM", "GIF", "BIGGIF", "ESAT", "BSB", "XPM", "BMP", "DIMAP", "AirSAR", "RS2", "PCIDSK", "PCRaster", "ILWIS", "SGI", "SRTMHGT", "Leveller", "Terragen", "GMT", "netCDF", "HDF4", "HDF4Image", "ISIS3", "ISIS2", "PDS", "TIL", "ERS", "JPEG2000", "L1B", "FIT", "GRIB", "RMF", "WCS", "WMS", "MSGN", "RST", "INGR", "GSAG", "GSBG", "GS7BG", "COSAR", "TSX"]

There is no sign of 'ESRI Shapefile'. Also, I find it strange that the 'GTiff' ends up in the OGR (Vector) driver namespace.

Gdal::Ogr.get_driver_count gives a count of 61 available drivers. However If you set a manual loop end to 119, like in this example

(0..119).map { |n| Gdal::Ogr.get_driver(n).get_name }

we get more drivers available than Gdal::Ogr.get_driver_count says there are:

=> ["VRT", "GTiff", "NITF", "RPFTOC", "ECRGTOC", "HFA", "SAR_CEOS", "CEOS", "JAXAPALSAR", "GFF", "ELAS", "AIG", "AAIGrid", "GRASSASCIIGrid", "SDTS", "OGDI", "DTED", "PNG", "JPEG", "MEM", "JDEM", "GIF", "BIGGIF", "ESAT", "BSB", "XPM", "BMP", "DIMAP", "AirSAR", "RS2", "PCIDSK", "PCRaster", "ILWIS", "SGI", "SRTMHGT", "Leveller", "Terragen", "GMT", "netCDF", "HDF4", "HDF4Image", "ISIS3", "ISIS2", "PDS", "TIL", "ERS", "JPEG2000", "L1B", "FIT", "GRIB", "RMF", "WCS", "WMS", "MSGN", "RST", "INGR", "GSAG", "GSBG", "GS7BG", "COSAR", "TSX", "COASP", "R", "MAP", "PNM", "DOQ1", "DOQ2", "ENVI", "EHdr", "GenBin", "PAux", "MFF", "MFF2", "FujiBAS", "GSC", "FAST", "BT", "LAN", "CPG", "IDA", "NDF", "EIR", "DIPEx", "LCP", "GTX", "LOSLAS", "NTv2", "CTable2", "ACE2", "SNODAS", "ARG", "RIK", "USGSDEM", "GXF", "DODS", "HTTP", "BAG", "HDF5", "HDF5Image", "NWT_GRD", "NWT_GRC", "ADRG", "SRP", "BLX", "Rasterlite", "EPSILON", "PostGISRaster", "SAGA", "KMLSUPEROVERLAY", "XYZ", "HF2", "PDF", "OZI", "CTG", "E00GRID", "WEBP", "ZMap", "NGSGEOID", "MBTiles", "IRIS"]

But still no trace of ESRI Shapefile

I've also tried calling Gdal::Ogr.register_all before the driver loop, but still no luck.

Strangley on Mac OS X, when the same command is run from the ruby console the output is:

["ESRI Shapefile", "MapInfo File", "UK .NTF", "SDTS", "TIGER", "S57", "DGN", "VRT", "REC", "Memory", "BNA", "CSV", "GML", "GPX", "KML", "GeoJSON", "GMT", "SQLite", "PostgreSQL", "PCIDSK", "XPlane", "AVCBin", "AVCE00", "DXF", "Geoconcept", "GeoRSS", "GPSTrackMaker", "VFK", "PGDump", "OSM", "GPSBabel", "SUA", "OpenAir", "PDS", "WFS", "HTF", "AeronavFAA", "EDIGEO", "GFT", "SVG", "CouchDB", "Idrisi", "ARCGEN", "SEGUKOOA", "SEGY", "XLS", "ODS", "XLSX", "ElasticSearch", "PDF"]

ESRI Shapefile is recognized, so there are no problems on OS X.

Here's some information about the Linux, ruby, gdal installation where this issue exists:

root@87de77888e8e:# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:  Debian GNU/Linux 8.1 (jessie)
Release:  8.1
Codename: jessie

root@87de77888e8e:# uname -r
4.9.49-moby

root@87de77888e8e# ruby --version
ruby 2.2.2p95 (2015-04-13 revision 50295) [x86_64-linux]

Dockerfile that is used for building ruby gdal image.

FROM ruby:2.2.2

RUN DEBIAN_FRONTEND=noninteractive apt-get -qq update \
  && apt-get --fix-missing install -y --force-yes \
  build-essential g++ make pkg-config \
  wget vim git openssl libssl-dev\
  python-pip python-numpy python-dev \
  zlib1g-dev libkml-dev libproj-dev libgeos-dev libpq-dev \
  libgdal-dev libgdal1h gdal-bin python-gdal \
  readline-common

RUN gem update --system
RUN gem install bundler
RUN gem install gdal

COPY test.cpp /
COPY test.rb /
COPY test.sh /
RUN chmod +x /test.sh

RUN rm -rf /var/lib/apt/lists/*

When using ogrinfo --formats the 'ESRI Shapefile' shows up as available.

root@87de77888e8e:/# ogrinfo --formats
Supported Formats:
  -> "ESRI Shapefile" (read/write)
  -> "MapInfo File" (read/write)
  -> "UK .NTF" (readonly)
  -> "SDTS" (readonly)
  -> "TIGER" (read/write)
  -> "S57" (read/write)
  -> "DGN" (read/write)
  -> "VRT" (readonly)
  -> "REC" (readonly)
  -> "Memory" (read/write)
  -> "BNA" (read/write)
  -> "CSV" (read/write)
  -> "GML" (read/write)
  -> "GPX" (read/write)
  -> "LIBKML" (read/write)
  -> "KML" (read/write)
  -> "GeoJSON" (read/write)
  -> "GMT" (read/write)
  -> "SQLite" (read/write)
  -> "PostgreSQL" (read/write)
  -> "PCIDSK" (read/write)
  -> "XPlane" (readonly)
  -> "AVCBin" (readonly)
  -> "AVCE00" (readonly)
  -> "DXF" (read/write)
  -> "Geoconcept" (read/write)
  -> "GeoRSS" (read/write)
  -> "GPSTrackMaker" (read/write)
  -> "VFK" (readonly)
  -> "PGDump" (read/write)
  -> "OSM" (readonly)
  -> "GPSBabel" (read/write)
  -> "SUA" (readonly)
  -> "OpenAir" (readonly)
  -> "PDS" (readonly)
  -> "WFS" (readonly)
  -> "HTF" (readonly)
  -> "AeronavFAA" (readonly)
  -> "EDIGEO" (readonly)
  -> "GFT" (read/write)
  -> "SVG" (readonly)
  -> "CouchDB" (read/write)
  -> "Idrisi" (readonly)
  -> "ARCGEN" (readonly)
  -> "SEGUKOOA" (readonly)
  -> "SEGY" (readonly)
  -> "ODS" (read/write)
  -> "XLSX" (read/write)
  -> "ElasticSearch" (read/write)
  -> "PDF" (read/write)

versions from the Gdal/Ogr command line

root@87de77888e8e:/# ogrinfo --version
GDAL 1.10.1, released 2013/08/26
root@87de77888e8e:/# gdal-config --version
1.10.1

I've also tried building libgdal-dev from sources (gdal-1.10.1+dfsg) using

# build & install libgdal-dev
./configure
make
make install
# reinstall gem
gem uninstall gdal
gem install gdal -- --with-gdal-includes=/usr/include/gdal

I've also tested getting driver information compiling a c++ that calls the same methods that are used in the ruby SWIG bindings inside the ruby-gdal gem.

In ogr.cpp (inside ruby-gdal gem) we have these methods definitions/wrappers

OGRDriverShadow* GetDriverByName( char const *name ) {
  return (OGRDriverShadow*) OGRGetDriverByName( name );
}
_wrap_get_driver_by_name(int argc, VALUE *argv, VALUE self) {
  char *arg1 = (char *) 0 ;
  int res1 ;
  char *buf1 = 0 ;
  int alloc1 = 0 ;
  OGRDriverShadow *result = 0 ;
  VALUE vresult = Qnil;
  
  if ((argc < 1) || (argc > 1)) {
    rb_raise(rb_eArgError, "wrong # of arguments(%d for 1)",argc); SWIG_fail;
  }
  res1 = SWIG_AsCharPtrAndSize(argv[0], &buf1, NULL, &alloc1);
  if (!SWIG_IsOK(res1)) {
    SWIG_exception_fail(SWIG_ArgError(res1), Ruby_Format_TypeError( "", "char const *","GetDriverByName", 1, argv[0] ));
  }
  arg1 = reinterpret_cast< char * >(buf1);
  {
    if (!arg1) {
      SWIG_exception(SWIG_ValueError,"Received a NULL pointer.");
    }
  }
  {
    CPLErrorReset();
    result = (OGRDriverShadow *)GetDriverByName((char const *)arg1);
    CPLErr eclass = CPLGetLastErrorType();
    if ( eclass == CE_Failure || eclass == CE_Fatal ) {
      SWIG_exception( SWIG_RuntimeError, CPLGetLastErrorMsg() );
      
    }
  }
  vresult = SWIG_NewPointerObj(SWIG_as_voidptr(result), SWIGTYPE_p_OGRDriverShadow, 0 |  0 );
  if (alloc1 == SWIG_NEWOBJ) delete[] buf1;
  return vresult;
fail:
  if (alloc1 == SWIG_NEWOBJ) delete[] buf1;
  return Qnil;
}

/*
  Document-method: Gdal::Ogr.register_all

  call-seq:
    register_all

A module function.
*/
SWIGINTERN VALUE
_wrap_register_all(int argc, VALUE *argv, VALUE self) {
  if ((argc < 0) || (argc > 0)) {
    rb_raise(rb_eArgError, "wrong # of arguments(%d for 0)",argc); SWIG_fail;
  }
  {
    CPLErrorReset();
    OGRRegisterAll();
    CPLErr eclass = CPLGetLastErrorType();
    if ( eclass == CE_Failure || eclass == CE_Fatal ) {
      SWIG_exception( SWIG_RuntimeError, CPLGetLastErrorMsg() );
      
    }
  }
  return Qnil;
fail:
  return Qnil;
}

Then we have bindings for these defined methods

rb_define_module_function(mOgr, "register_all", VALUEFUNC(_wrap_register_all), -1);
rb_define_module_function(mOgr, "get_driver_by_name", VALUEFUNC(_wrap_get_driver_by_name), -1);

I've created test.cpp. It is suppose to do a similar output like the ruby snippet at the beggining of this text, but calling the same c++ methods as in the ruby bindings

#include "ogr_api.h"
#include "gdal_priv.h"

int main()
{
    const char *pszDriverName = "ESRI Shapefile";
    OGRSFDriverH poDriver;

    OGRRegisterAll();
    int count = OGRGetDriverCount();
    poDriver = OGRGetDriverByName(pszDriverName);
    printf("%d \n", count);
    for( int a = 0; a < count; a = a + 1 ) {
      OGRSFDriverH drv = OGRGetDriver(a);
      printf( "%s, ", OGR_Dr_GetName(drv));
    }
    printf("\n");

    if( poDriver == NULL )
    {
      printf( "%s driver not available.\n", pszDriverName );
      exit( 1 );
    } {
      printf( "%s driver is available. \n", pszDriverName );
    }
}

Tested with the following command:

g++ test.cpp -I /usr/include/gdal -lgdal -o test && ./test

output:

61
ESRI Shapefile, MapInfo File, UK .NTF, SDTS, TIGER, S57, DGN, VRT, REC, Memory, BNA, CSV, NAS, GML, GPX, LIBKML, KML, GeoJSON, Interlis 1, Interlis 2, GMT, SQLite, DODS, ODBC, PGeo, MSSQLSpatial, OGDI, PostgreSQL, MySQL, PCIDSK, XPlane, AVCBin, AVCE00, DXF, Geoconcept, GeoRSS, GPSTrackMaker, VFK, PGDump, OSM, GPSBabel, SUA, OpenAir, PDS, WFS, HTF, AeronavFAA, Geomedia, EDIGEO, GFT, SVG, CouchDB, Idrisi, ARCGEN, SEGUKOOA, SEGY, XLS, ODS, XLSX, ElasticSearch, PDF,
ESRI Shapefile driver is available.

So this leads to the conclusion that there is something wrong with gdal-ruby bindings that leads to many OGR drivers not to be available. I don't understand why does this difference exist and why do identical c++ methods return different drivers?

I've created a github repository with the Dockerfile, testing examples and instructions on what to run to see the driver outputs from ruby vs cpp https://github.com/svilenkov/ruby-gdal-driver-issue/

[svilenkov]

It could be that the the Gdal driver methods symbols somehow overwrite Ogr driver methods symbols since these 2 lines produce the same results

puts (0..119).map { |n| Gdal::Ogr.get_driver(n).get_name }.join(', ')

puts (0..Gdal::Gdal.get_driver_count-1).map { |n| Gdal::Gdal.get_driver(n).ShortName }.join(', ')

[svilenkov]

found a solution in an old issue: #2 (comment)
Problem can be avoided if you recompile & install the gem by replacing GetDriver with OGR_GetDriver and GetDriverByName with OGR_GetDriverByName in the ogr.cpp.

[denisahearn]

@svilenkov We are experiencing the same issue, so thanks for posting this.

We worked around the problem by including a sample shapefile with our application, and then using it to get at the ESRI Shapefile driver, like so:

shapefile_path = File.join(File.dirname(__FILE__), 'shapefile')
shapfile_source = Gdal::Ogr.open(shapefile_path)
shapfile_source.get_driver

Is fixing this issue as simple as opening a pull request with the modification to ogr.cpp that you point out above, and then asking this gem's maintainer to accept the pull request? If so, it feels like we should go this route so that the issue is fixed for everyone.

Thanks,
Denis

[svilenkov]

@denisahearn Nice, a clever workaround.

Is fixing this issue as simple as opening a pull request with the modification to ogr.cpp that you point out above, and then asking this gem's maintainer to accept the pull request? If so, it feels like we should go this route so that the issue is fixed for everyone.

That's true, although I haven't seen responses on this repo for quite some time. I've did the replacements and regenerated the SWIG bindings for the last 1.x gdal here: https://github.com/svilenkov/gdal-ruby/tree/gdal-1.11.5 Should probably make a pull-request out of this branch.

[denisahearn]

@svilenkov Thanks for posting the link to your forked repo.

It would be really nice if this were an actively maintained gem. It would also be nice it there was a version of this gem that worked with GDAL 2.x. (2.2.4 is the latest version as of this reply).

We rely on this gem in production, do you also? I would gladly help maintain this gem (or a fork of it), however I'm not sure what it would take to upgrade it to work with GDAL 2.x. I see that someone opened a pull request in this repo with changes to use GDAL 2.x back in May of 2016, but the gem's author didn't accept it (reasons are posted in the PR): #11.

@zhm Are you open to the idea of reviving this gem, and would you be willing to provide guidance on the best way to do that?

[clayreinhardt]

any plans for GDAL 3.x @svilenkov ?