[Bug] Ubuntu: libgrass_interface_generator does not build

[BadAssassin]

Describe the bug
python/libgrass_interface_generator does not compile under Ubuntu 20.04.5LTS. Build errors with "double free or corruption. Aborted"

To Reproduce
cd python/libgrass_interface_generator
make

Expected behavior
No errors.

System description (please complete the following information):

• Operating System: Ubuntu 20.04.5LTS running under Win11 WSL
• GRASS GIS version - 8.3dev - latest commit

Additional context
Output:

make /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib
make[1]: Entering directory '/home/brad/grass/python/libgrass_interface_generator'
make[1]: '/home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib' is up to date.
make[1]: Leaving directory '/home/brad/grass/python/libgrass_interface_generator'
make OBJ.x86_64-pc-linux-gnu/date.py OBJ.x86_64-pc-linux-gnu/gis.py OBJ.x86_64-pc-linux-gnu/raster.py OBJ.x86_64-pc-linux-gnu/gmath.py OBJ.x86_64-pc-linux-gnu/proj.py OBJ.x86_64-pc-linux-gnu/imagery.py OBJ.x86_64-pc-linux-gnu/vector.py OBJ.x86_64-pc-linux-gnu/rtree.py OBJ.x86_64-pc-linux-gnu/display.py OBJ.x86_64-pc-linux-gnu/stats.py OBJ.x86_64-pc-linux-gnu/dbmi.py OBJ.x86_64-pc-linux-gnu/raster3d.py OBJ.x86_64-pc-linux-gnu/arraystats.py OBJ.x86_64-pc-linux-gnu/cluster.py OBJ.x86_64-pc-linux-gnu/vedit.py OBJ.x86_64-pc-linux-gnu/segment.py OBJ.x86_64-pc-linux-gnu/rowio.py OBJ.x86_64-pc-linux-gnu/temporal.py OBJ.x86_64-pc-linux-gnu/ogsf.py OBJ.x86_64-pc-linux-gnu/nviz.py /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/date.py /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/gis.py /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/raster.py /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/gmath.py /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/proj.py /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/imagery.py /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/vector.py /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/rtree.py /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/display.py /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/stats.py /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/dbmi.py /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/raster3d.py /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/arraystats.py /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/cluster.py /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/vedit.py /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/segment.py /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/rowio.py /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/temporal.py /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/ogsf.py /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/nviz.py /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/date.pyc /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/gis.pyc /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/raster.pyc /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/gmath.pyc /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/proj.pyc /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/imagery.pyc /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/vector.pyc /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/rtree.pyc /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/display.pyc /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/stats.pyc /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/dbmi.pyc /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/raster3d.pyc /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/arraystats.pyc /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/cluster.pyc /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/vedit.pyc /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/segment.pyc /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/rowio.pyc /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/temporal.pyc /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/ogsf.pyc /home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/nviz.pyc
make[1]: Entering directory '/home/brad/grass/python/libgrass_interface_generator'
test -d OBJ.x86_64-pc-linux-gnu || mkdir -p OBJ.x86_64-pc-linux-gnu
GISRC=/home/brad/grass/dist.x86_64-pc-linux-gnu/demolocation/.grassrc83 GISBASE=/home/brad/grass/dist.x86_64-pc-linux-gnu PATH="/home/brad/grass/dist.x86_64-pc-linux-gnu/bin:/home/brad/grass/dist.x86_64-pc-linux-gnu/bin:/home/brad/grass/dist.x86_64-pc-linux-gnu/scripts:$PATH" PYTHONPATH="/home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python:/home/brad/grass/dist.x86_64-pc-linux-gnu/gui/wxpython:$PYTHONPATH" LD_LIBRARY_PATH="/home/brad/grass/dist.x86_64-pc-linux-gnu/bin:/home/brad/grass/dist.x86_64-pc-linux-gnu/bin:/home/brad/grass/dist.x86_64-pc-linux-gnu/scripts:/home/brad/grass/dist.x86_64-pc-linux-gnu/lib:/home/brad/grass/dist.x86_64-pc-linux-gnu/lib:" LC_ALL=C LANG=C LANGUAGE=C ./run.py --no-embed-preamble --strip-build-path "/home/brad/grass/dist.x86_64-pc-linux-gnu" --cpp "gcc -E -I/home/brad/grass/dist.x86_64-pc-linux-gnu/include -I/home/brad/grass/dist.x86_64-pc-linux-gnu/include -D__GLIBC_HAVE_LONG_LONG" -lgrass_datetime.8.3 /home/brad/grass/dist.x86_64-pc-linux-gnu/include/grass/datetime.h /home/brad/grass/dist.x86_64-pc-linux-gnu/include/grass/defs/datetime.h -o OBJ.x86_64-pc-linux-gnu/date.py
INFO: Status: Preprocessing /tmp/tmpoe2fngnr.h
INFO: Status: gcc -E -I/home/brad/grass/dist.x86_64-pc-linux-gnu/include -I/home/brad/grass/dist.x86_64-pc-linux-gnu/include -D__GLIBC_HAVE_LONG_LONG -U GNUC -dD "-D__extension__=" "-D__const=const" "-D__asm__(x)=" "-D__asm(x)=" "-DCTYPESGEN=1" "/tmp/tmpoe2fngnr.h"
INFO: Status: Parsing /tmp/tmpoe2fngnr.h
INFO: Status: Processing description list.
INFO: Status: Writing to OBJ.x86_64-pc-linux-gnu/date.py.
INFO: Status: Wrapping complete.
GISRC=/home/brad/grass/dist.x86_64-pc-linux-gnu/demolocation/.grassrc83 GISBASE=/home/brad/grass/dist.x86_64-pc-linux-gnu PATH="/home/brad/grass/dist.x86_64-pc-linux-gnu/bin:/home/brad/grass/dist.x86_64-pc-linux-gnu/bin:/home/brad/grass/dist.x86_64-pc-linux-gnu/scripts:$PATH" PYTHONPATH="/home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python:/home/brad/grass/dist.x86_64-pc-linux-gnu/gui/wxpython:$PYTHONPATH" LD_LIBRARY_PATH="/home/brad/grass/dist.x86_64-pc-linux-gnu/bin:/home/brad/grass/dist.x86_64-pc-linux-gnu/bin:/home/brad/grass/dist.x86_64-pc-linux-gnu/scripts:/home/brad/grass/dist.x86_64-pc-linux-gnu/lib:/home/brad/grass/dist.x86_64-pc-linux-gnu/lib:" LC_ALL=C LANG=C LANGUAGE=C ./run.py --no-embed-preamble --strip-build-path "/home/brad/grass/dist.x86_64-pc-linux-gnu" --cpp "gcc -E -I/home/brad/grass/dist.x86_64-pc-linux-gnu/include -I/home/brad/grass/dist.x86_64-pc-linux-gnu/include -D__GLIBC_HAVE_LONG_LONG" -lgrass_gis.8.3 /home/brad/grass/dist.x86_64-pc-linux-gnu/include/grass/gis.h /home/brad/grass/dist.x86_64-pc-linux-gnu/include/grass/defs/gis.h /home/brad/grass/dist.x86_64-pc-linux-gnu/include/grass/colors.h /home/brad/grass/dist.x86_64-pc-linux-gnu/include/grass/defs/colors.h -o OBJ.x86_64-pc-linux-gnu/gis.py
INFO: Status: Preprocessing /tmp/tmptkl6a8fm.h
INFO: Status: gcc -E -I/home/brad/grass/dist.x86_64-pc-linux-gnu/include -I/home/brad/grass/dist.x86_64-pc-linux-gnu/include -D__GLIBC_HAVE_LONG_LONG -U GNUC -dD "-D__extension__=" "-D__const=const" "-D__asm__(x)=" "-D__asm(x)=" "-DCTYPESGEN=1" "/tmp/tmptkl6a8fm.h"
INFO: Status: Parsing /tmp/tmptkl6a8fm.h
ERROR: /usr/include/x86_64-linux-gnu/sys/cdefs.h:245: Syntax error at '\n'
INFO: Status: Processing description list.
WARNING: Could not parse macro "#define serialize_int32_le(buf,x) do { ( buf ) [ 0 ] = ( ( x ) >> 0 ) & 0xFF ; ( buf ) [ 1 ] = ( ( x ) >> 8 ) & 0xFF ; ( buf ) [ 2 ] = ( ( x ) >> 16 ) & 0xFF ; ( buf ) [ 3 ] = ( ( x ) >> 24 ) & 0xFF ; } while ( 0 )"
WARNING: Could not parse macro "#define serialize_int32_be(buf,x) do { ( buf ) [ 0 ] = ( ( x ) >> 24 ) & 0xFF ; ( buf ) [ 1 ] = ( ( x ) >> 16 ) & 0xFF ; ( buf ) [ 2 ] = ( ( x ) >> 8 ) & 0xFF ; ( buf ) [ 3 ] = ( ( x ) >> 0 ) & 0xFF ; } while ( 0 )"
INFO: Status: Writing to OBJ.x86_64-pc-linux-gnu/gis.py.
INFO: Status: Wrapping complete.
GISRC=/home/brad/grass/dist.x86_64-pc-linux-gnu/demolocation/.grassrc83 GISBASE=/home/brad/grass/dist.x86_64-pc-linux-gnu PATH="/home/brad/grass/dist.x86_64-pc-linux-gnu/bin:/home/brad/grass/dist.x86_64-pc-linux-gnu/bin:/home/brad/grass/dist.x86_64-pc-linux-gnu/scripts:$PATH" PYTHONPATH="/home/brad/grass/dist.x86_64-pc-linux-gnu/etc/python:/home/brad/grass/dist.x86_64-pc-linux-gnu/gui/wxpython:$PYTHONPATH" LD_LIBRARY_PATH="/home/brad/grass/dist.x86_64-pc-linux-gnu/bin:/home/brad/grass/dist.x86_64-pc-linux-gnu/bin:/home/brad/grass/dist.x86_64-pc-linux-gnu/scripts:/home/brad/grass/dist.x86_64-pc-linux-gnu/lib:/home/brad/grass/dist.x86_64-pc-linux-gnu/lib:" LC_ALL=C LANG=C LANGUAGE=C ./run.py --no-embed-preamble --strip-build-path "/home/brad/grass/dist.x86_64-pc-linux-gnu" --cpp "gcc -E -I/home/brad/grass/dist.x86_64-pc-linux-gnu/include -I/home/brad/grass/dist.x86_64-pc-linux-gnu/include -D__GLIBC_HAVE_LONG_LONG" -lgrass_raster.8.3 /home/brad/grass/dist.x86_64-pc-linux-gnu/include/grass/raster.h /home/brad/grass/dist.x86_64-pc-linux-gnu/include/grass/defs/raster.h -o OBJ.x86_64-pc-linux-gnu/raster.py
INFO: Status: Preprocessing /tmp/tmpk0ja388s.h
INFO: Status: gcc -E -I/home/brad/grass/dist.x86_64-pc-linux-gnu/include -I/home/brad/grass/dist.x86_64-pc-linux-gnu/include -D__GLIBC_HAVE_LONG_LONG -U GNUC -dD "-D__extension__=" "-D__const=const" "-D__asm__(x)=" "-D__asm(x)=" "-DCTYPESGEN=1" "/tmp/tmpk0ja388s.h"
INFO: Status: Parsing /tmp/tmpk0ja388s.h
ERROR: /usr/include/x86_64-linux-gnu/sys/cdefs.h:245: Syntax error at '\n'
INFO: Status: Processing description list.
INFO: Status: Writing to OBJ.x86_64-pc-linux-gnu/raster.py.
INFO: Status: Wrapping complete.
double free or corruption (out)
Aborted
make[1]: *** [Makefile:110: OBJ.x86_64-pc-linux-gnu/raster.py] Error 134
make[1]: Leaving directory '/home/brad/grass/python/libgrass_interface_generator'
make: *** [Makefile:86: default] Error 2

[nilason]

INFO: Status: Wrapping complete.
double free or corruption (out)
Aborted
make[1]: *** [Makefile:110: OBJ.x86_64-pc-linux-gnu/raster.py] Error 134
make[1]: Leaving directory '/home/brad/grass/python/libgrass_interface_generator'
make: *** [Makefile:86: default] Error 2

This doesn't strike me as a ctypesgen issue as such.
Please make distclean, rebuild it all and then look for the first (and probably only) "Error 1" in the log.

[marisn]

There are multiple proj versions installed. Remove one of projs, make distclean && make. Then close this issue as solved.

See another example: #435 (comment)

[BadAssassin]

I'm clearly missing something, here. This used to work in the past. If I point it to the default PROJ, it compiles. If I point it to the dev version, it won't build. Why? There's really no point in having configure options if this is truly the case. And why it worked in the past, I guess is a mystery.

On 11/5/2022 9:00 AM, Māris Nartišs wrote: There are multiple proj versions installed. Remove one of projs, make distclean && make. Then close this issue as solved. See another example: #435 (comment) <#435 (comment)> — Reply to this email directly, view it on GitHub <#2629 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQZTHVJ4TWKVQOB2PLM5LBDWG2AATANCNFSM6AAAAAARXZA72E>. You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Best Regards, -Brad

[marisn]

If I point it to the default PROJ, it compiles. If I point it to the dev version, it won't build. Why?

It all boils down on how to tell library loader to load correct version during compilation and execution. I'd guess it is necessary to adjust LD_LIBRARY_PATH if path to proj lib has been set during configure phase. Unfortunately this is outside of my area of expertise as I'm too weak in Make.

[BadAssassin]

On 11/5/2022 3:07 PM, Māris Nartišs wrote: If I point it to the default PROJ, it compiles. If I point it to the dev version, it won't build. Why? It all boils down on how to tell library loader to load correct version during compilation and execution. I'd guess it is necessary to adjust LD_LIBRARY_PATH if path to proj lib has been set during configure phase. Unfortunately this is outside of my area of expertise as I'm too weak in Make. Message ID: ***@***.***>

I think I'm realizing what's going on. Something like what you suggest may work. I think what is happening is that the default proj is getting pulled in due to other tools GRASS depends on using it (gdal, etc.). I may have relinked everything before and that's why it worked. Either way, this can be closed. Thank you all for the assistance.

…

-- Best Regards, -Brad

[marisn]

It would be nice to get to the bottom of this to see if a solution is possible or not at all. But, as I stated, this is above my level.

[BadAssassin]

I did some investigation. This is a result of libgproj. libgproj links to gdal, which is also linked to proj, pulling in the version it is linked to. libgproj ends up linking to two different version. Everything is working as intended, but the near-circular dependency is an issue. Ideally, seem we should either refuse linking to external proj and use what gdal brings, or segregate the usage, somehow. Suggestions? Or do nothing and note it is an issue.

On 11/6/2022 9:27 AM, Māris Nartišs wrote: It would be nice to get to the bottom of this to see if a solution is possible or not at all. But, as I stated, this is above my level. Message ID: ***@***.***>

-- Best Regards, -Brad

[nilason]

This is a result of libgproj. libgproj links to gdal,

Why is libgproj linked to gdal?

[nilason]

This is a result of libgproj. libgproj links to gdal,

Why is libgproj linked to gdal?

Never mind, I misread it to libproj.

[nilason]

For good or for bad, I've had no problem (yet) to configure to a different PROJ version than the version GDAL is linked to (this is on mac).

Here are some info, which might be of any help:

# extract from build log
/usr/bin/clang -dynamiclib -compatibility_version 8.3 -current_version 8.3 \
	-install_name @rpath/libgrass_gproj.8.3.dylib \
	-o /dev/grass/dist.aarch64-apple-darwin21.6.0/lib/libgrass_gproj.8.3.dylib \
	-L/dev/grass/dist.aarch64-apple-darwin21.6.0/lib \
	-L/dev/grass/dist.aarch64-apple-darwin21.6.0/lib \
	-arch arm64 -Wl,-rpath,/usr/local/grass83/lib,-rpath,/dev/grass/dist.aarch64-apple-darwin21.6.0/lib \
	OBJ.aarch64-apple-darwin21.6.0/convert.o OBJ.aarch64-apple-darwin21.6.0/datum.o OBJ.aarch64-apple-darwin21.6.0/do_proj.o OBJ.aarch64-apple-darwin21.6.0/ellipse.o OBJ.aarch64-apple-darwin21.6.0/get_proj.o \
	-lgrass_gis.8.3 \
	-L/opt/local/lib -lgdal \
	-L/opt/local/lib/proj9/lib -lproj


# check libraries' links
otool -L /dev/grass/dist.aarch64-apple-darwin21.6.0/lib/libgrass_gproj.8.3.dylib 
/dev/grass/dist.aarch64-apple-darwin21.6.0/lib/libgrass_gproj.8.3.dylib:
	@rpath/libgrass_gproj.8.3.dylib (compatibility version 8.3.0, current version 8.3.0)
	@rpath/libgrass_gis.8.3.dylib (compatibility version 8.3.0, current version 8.3.0)
	/opt/local/lib/libgdal.30.dylib (compatibility version 31.0.0, current version 31.3.0)
	/opt/local/lib/proj9/lib/libproj.25.dylib (compatibility version 25.0.0, current version 25.9.1)
	...

otool -L /opt/local/lib/libgdal.30.dylib
/opt/local/lib/libgdal.30.dylib:
	/opt/local/lib/libgdal.30.dylib (compatibility version 31.0.0, current version 31.3.0)
	...
	/opt/local/lib/proj8/lib/libproj.22.dylib (compatibility version 25.0.0, current version 25.1.0)
	...


# GRASS version and dependencies
g.version -rge
version=8.3.dev
date=2022
revision=5d64de224
build_date=2022-11-07
build_platform=aarch64-apple-darwin21.6.0
build_off_t_size=8
libgis_revision=b35aca1c5
libgis_date=2022-11-07T17:50:28+00:00
proj=9.1.0
gdal=3.4.3
geos=3.11.0
sqlite=3.39.4

[BadAssassin]

Interesting to see that on Mac! It also looks like it uses an entirely different build process. Linux is still stuck using configure/make. Many projects have migrated to clang or cmake. I can force it manually, but nobody should be able to have to do that!

I did some investigation as to why this is happening:

Looking at include/grass/gprojects.h, it indicates some cleanup is required in the PROJ definitions, particularly for v6+. It claims PROJ6+ requires GDAL 3+ and includes ogr_srs_api.h from gdal. It took me some time to understand why this is: libgrass_gproj is a complete projection library. PROJ handles the underlying projection, but OGR from GDAL is used to do conversions to and from WKT.

It may be possible to split it up to avoid the conflict, but I haven't thought it through enough to determine if it will just come back.

[nilason]

Interesting to see that on Mac! It also looks like it uses an entirely different build process. Linux is still stuck using configure/make. Many projects have migrated to clang or cmake. I can force it manually, but nobody should be able to have to do that!

The same autotools (configure, make) are used on Mac, as on all other platforms. The default compiler is Clang, though, not gcc. Clang is also used for dynamic linking, which is (by default for Mac as stated in aclocal.m4) based on rpath. So not so very much different. Perhaps the "compatibility version x.z, current version x.z" part makes a difference?

[neteler]

Many projects have migrated to clang or cmake

@BadAssassin Please take a look at the PRs #289 and #348 which are draft changes for cmake support.

Update: all CMake support carried over to #2684