[C++] libarrow_flight symbol issue with pre-packaged Abseil >= 20240722.0

[WillAyd]

Describe the bug, including details regarding any error messages, version, and platform.

When using abseil-cpp>=20240722.0, the following error message appears in failed CI builds for Python (similar jobs may also fail):

ImportError: The pyarrow installation is not built with support for 'flight' (/opt/conda/envs/arrow/lib/libarrow_flight.so.2300: undefined symbol: _ZN4absl12lts_202407225Mutex4DtorEv)

My assumption is that the preinstalled abseil-cpp conda package is installed as a "Release" set of libraries, whereas CI may have debug builds of the Arrow C++ library. At least dating back to abseil/abseil-cpp@f3760b4, there is a header from abseil_synchronization which defines a destructor only for debug builds, so when the Arrow C++ debug build includes this header but searches for the destructor in the release-built conda C++ package, the missing symbol issue rears its head

Strangely, the referenced patch also appears in the abseil 20240116.2 release that CI currently uses. I'm not sure what causes it to be more pronounced with a version upgrade

Component(s)

C++

[WillAyd]

The more I look at this the more I think it is potentially a bug with gcc. For example, if I produce an MRE like:

#include "absl/synchronization/mutex.h"

void foo() {
  auto var = absl::Mutex{};
  var.Lock();
}

and compile that with g++-14:

$ g++-14 -Iliblib.so.p -I. -I.. -fdiagnostics-color=always -D_GLIBCXX_ASSERTIONS=1 -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -O0 -g -O0 -fPIC -DNOMINMAX -MD -MQ liblib.so.p/src.cc.o -MF liblib.so.p/src.cc.o.d -o liblib.so.p/src.cc.o -c ../src.cc

then no errant symbol appears:

$ objdump --syms liblib.so.p/src.cc.o | grep Dtor

If I run the same command but symbol reference a compiler installed by the conda compilers package:

$ /home/willayd/miniforge3/envs/abseil-debug/bin/x86_64-conda-linux-gnu-c++ -Iliblib.so.p -I. -I.. -fdiagnostics-color=always -D_GLIBCXX_ASSERTIONS=1 -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -O0 -g -O0 -fPIC -DNOMINMAX -MD -MQ liblib.so.p/src.cc.o -MF liblib.so.p/src.cc.o.d -o liblib.so.p/src.cc.o -c ../src.cc

then the errant reference appears:

$ objdump --syms liblib.so.p/src.cc.o | grep Dtor
0000000000000000         *UND*	0000000000000000 _ZN4absl12lts_202407225Mutex4DtorEv

On ubuntu 24.04 it looks like g++-14 provides version 14.2, whereas conda provides 14.3. So perhaps there is a regression between the minor versions

[WillAyd]

I can't reproduce the issue with g++ 14.3.0 in a docker image, so I think this is related to the g++ 14.3.0 that comes in the compilers package

[corpoverlords]

Using different Abseil is strictly UB per Abseil team's stance

// Abseil contains a number of possible configuration endpoints, based on
// parameters such as the detected platform, language version, or command-line
// flags used to invoke the underlying binary. As is the case with all
// libraries, binaries which contain Abseil code must ensure that separate
// packages use the same compiled copy of Abseil to avoid a diamond dependency
// problem, which can occur if two packages built with different Abseil
// configuration settings are linked together.

Two copies of Abseil configured with different options are incompatible, in exactly the same way that two different source snapshots of Abseil are incompatible. Abseil options must be set consistently across an entire program. They cannot be changed on a per-library basis.

Arrow should link to the "system" (conda in this setting) copy of Abseil and not download its own copy

[WillAyd]

Issue was resolved in #48414