Plugin fails to consider the type of the C library for Linux distributions

[jbartok]

At the moment the plugin is unable to determine the C library type for which it's supposed to provide JDKs.
This is a problem on Linux distributions because a JDK written for Alpine (ie. musl library) won't work on Ubuntu (i.e. glibc library).
For now all the plugin does is to prioritize glibc over musl, so it will work fine on Ubuntu and similar distributions, but won't work fine on Alpine.
Need to figure out a way to determine the exact library version the current JVM is running on.

[jbartok]

Maybe can use: https://github.com/lovell/detect-libc

[cromefire]

Is there any way to override it manually as a stop gap?

[jbartok]

Is there any way to override it manually as a stop gap?

Not that I'm aware of. But contributions are welcome.

[karobur]

UPDATE 2024-05-13
Following workaround does not work anymore on Alpine Linux 3.19 due to change in musl package. For 3.18 it's still working.

# New error on Alpie Linux 3.19 with musl@1.2.4_git20230717-r4
> Failed to calculate the value of task ':app:compileJava' property 'javaCompiler'.
   > Unable to download toolchain matching the requirements ({languageVersion=21, vendor=AMAZON, implementation=vendor-specific}) from 'https://api.foojay.io/disco/v3.0/ids/29f9fe87efdd0582e4ce1569fd1b33a9/redirect'.
      > Provisioned toolchain '/root/.gradle/jdks/amazon-corretto-21/amazon-corretto-21.0.3.9.1-linux-x64' could not be probed: Command returned unexpected result code: 6
        Error output:
        Error: dl failure on line 541
        Error: failed /root/.gradle/jdks/amazon-corretto-21/amazon-corretto-21.0.3.9.1-linux-x64/lib/server/libjvm.so, because Error relocating /root/.gradle/jdks/amazon-corretto-21/amazon-corretto-21.0.3.9.1-linux-x64/lib/server/libjvm.so: posix_fallocate64: symbol not found

• Alpine Linux 3.19 is using musl@1.2.4_git20230717-r4
• Alpine Linux 3.18 is using musl@1.2.4-r2

I'm not C expert so I'm unable to tell what have changed between musl@1.2.4_git20230717-r4 and musl@1.2.4-r2 and why it was introduced but sad fact is that latest available gcompat@1.1.0-r4 is not fully working with it and it's not possible to easily downgrade it 🙁

---

Original post 2024-04-04

For anyone who have encounter related issue on Alpine Linux distribution and foojay plugin e.g.

> Failed to calculate the value of task ':app:compileJava' property 'javaCompiler'.
   > Unable to download toolchain matching the requirements ({languageVersion=21, vendor=AMAZON, implementation=vendor-specific}) from 'https://api.foojay.io/disco/v3.0/ids/2ff40fc75fd514e8fd248f3d7fca9b58/redirect'.
      > Provisioned toolchain '/root/.gradle/jdks/amazon-corretto-21/amazon-corretto-21.0.2.14.1-linux-x64' could not be probed: A problem occurred starting process 'command '/root/.gradle/jdks/amazon-corretto-21/amazon-corretto-21.0.2.14.1-linux-x64/bin/java''

Official documentation suggest:

As a consequence, using multiple java toolchains is discouraged in environments with the musl library. This is the case in most Alpine distributions — consider using another distribution, like Ubuntu, instead.

If you don't want to switch over to other distribution, one of possible workaround is to install gcompat package offered within official Alpine Linux repository that is providing compatibility layer that works with foojay plugin.

This distribution contains the implementation of gcompat, a library which
provides glibc-compatible APIs for use on musl libc systems.

apk add gcompat

[karobur]

UPDATE 2024-05-13 Following workaround does not work anymore on Alpine Linux 3.19 due to change in musl package. For 3.18 it's still working.

# New error on Alpie Linux 3.19 with musl@1.2.4_git20230717-r4
> Failed to calculate the value of task ':app:compileJava' property 'javaCompiler'.
   > Unable to download toolchain matching the requirements ({languageVersion=21, vendor=AMAZON, implementation=vendor-specific}) from 'https://api.foojay.io/disco/v3.0/ids/29f9fe87efdd0582e4ce1569fd1b33a9/redirect'.
      > Provisioned toolchain '/root/.gradle/jdks/amazon-corretto-21/amazon-corretto-21.0.3.9.1-linux-x64' could not be probed: Command returned unexpected result code: 6
        Error output:
        Error: dl failure on line 541
        Error: failed /root/.gradle/jdks/amazon-corretto-21/amazon-corretto-21.0.3.9.1-linux-x64/lib/server/libjvm.so, because Error relocating /root/.gradle/jdks/amazon-corretto-21/amazon-corretto-21.0.3.9.1-linux-x64/lib/server/libjvm.so: posix_fallocate64: symbol not found

• Alpine Linux 3.19 is using musl@1.2.4_git20230717-r4
• Alpine Linux 3.18 is using musl@1.2.4-r2

I'm not C expert so I'm unable to tell what have changed between musl@1.2.4_git20230717-r4 and musl@1.2.4-r2 and why it was introduced but sad fact is that latest available gcompat@1.1.0-r4 is not fully working with it and it's not possible to easily downgrade it 🙁

Original post 2024-04-04

For anyone who have encounter related issue on Alpine Linux distribution and foojay plugin e.g.

> Failed to calculate the value of task ':app:compileJava' property 'javaCompiler'.
   > Unable to download toolchain matching the requirements ({languageVersion=21, vendor=AMAZON, implementation=vendor-specific}) from 'https://api.foojay.io/disco/v3.0/ids/2ff40fc75fd514e8fd248f3d7fca9b58/redirect'.
      > Provisioned toolchain '/root/.gradle/jdks/amazon-corretto-21/amazon-corretto-21.0.2.14.1-linux-x64' could not be probed: A problem occurred starting process 'command '/root/.gradle/jdks/amazon-corretto-21/amazon-corretto-21.0.2.14.1-linux-x64/bin/java''

Official documentation suggest:

As a consequence, using multiple java toolchains is discouraged in environments with the musl library. This is the case in most Alpine distributions — consider using another distribution, like Ubuntu, instead.

If you don't want to switch over to other distribution, one of possible workaround is to install gcompat package offered within official Alpine Linux repository that is providing compatibility layer that works with foojay plugin.

This distribution contains the implementation of gcompat, a library which
provides glibc-compatible APIs for use on musl libc systems.

apk add gcompat

I have updated my original post since on newer musl version this workaround it's no longer working 😞

[jbartok]

Comment by @jvandort, to be explored:

Looking online quickly, it seems there are a number of ways to detect the current machine's c standard library.

One is to check the output of the ldd --version (if ldd is installed). This is what prior versions of the JDK used when compiling, though the current version of the file does not have that code so it looks like they may have changed approaches https://github.com/openjdk/jdk/blob/e92e2fd4e0bc805d8f7d70f632cce0282eb1809b/make/autoconf/build-aux/config.guess#L37-L43

Another is to check for certain files in the /proc/self/map_files dir, which is the approach sqlite uses.

Here is a link to a discussion that another Java library used to solve this problem: https://github.com/dmlc/xgboost/issues/7915

They originally considered sqlite's approach but eventually decided to let musl users set an IS_MUSL flag. Though, since the point of this feature is to make things easy for the user, asking the user to set a flag may not be the best idea.

A final approach could be to detect well-known linux distros that are known to be musl. So if we're on alpine, assume musl

[johnjaylward]

Another option I saw for detecting GLIBC vs MUSL is to use getconf GNU_LIBC_VERSION.

On GNU systems this will return the version, like "glibc 2.40" and have a exit code of 0.
On MUSL systems, this will return "getconf: GNU_LIBC_VERSION: unknown variable" and have an exit code of 1

Not sure if that's better or worse than the ldd check that the java developers, xgboost, and AWS use.

Facebook notes that some systems lock down sh access and checks for musl should not rely only on sh. They suggest looking for the musl lib in certain locations as a work around : facebook/rocksdb#9956

Another option may be to work with the Gradle devs and have them publish their official docker images with the IS_MUSL environment variable set in the musl docker images. At least that way for docker based builds using the official images there is a guaranteed setting that should work.

[roded]

We recently started seeing these errors on Ubuntu machines:

09:19:44           > Cannot find a Java installation on your machine (Linux 6.5.0-1017-aws amd64) matching: 
{languageVersion=8, vendor=any vendor, implementation=vendor-specific, nativeImageCapable=false}. Some toolchain 
resolvers had provisioning failures: foojay (Unable to download toolchain matching the requirements ({languageVersion=8, 
vendor=any vendor, implementation=vendor-specific, nativeImageCapable=false}) from 'https://api.foojay.io/disco/v3.0/
ids/20eaba19a82279505f9bdb33913e94d7/redirect', due to: Unpacked JDK archive does not contain a Java home: /home/
jenkins/.gradle/.tmp/jdks/OpenJDK8U-jdk_x64_alpine-linux_hotspot_8u462b08-any#20vendor-83000492951773429296.tmp).

It's not clear why an Alpine package is downloaded for an Ubuntu machine considering the priorities described above. Nothing was changed recently from our end.

Any idea what the issue could be or how to workaround it?

Edit: I think this is related to the foojay repository. Seems like it lost packages?
The following returns only the OpenJDK8U-jdk_x64_alpine-linux_hotspot_8u462b08 package:
https://api.foojay.io/disco/v3.0/packages?package_type=jdk&latest=overall&version=8

Cause: foojayio/discoapi#124