Homebrew OpenJDK breaks `kserver`

[Baltoli]

The version of Java currently shipped by OpenJDK on Homebrew is 19, which causes a known issue with Nailgun:

• issue: https://youtrack.jetbrains.com/issue/SCL-20064
• workaround: https://openjdk.org/jeps/411#Issue-warnings

It's possible to get a working kserver by explicitly passing JAVA_TOOL_OPTIONS='-Djava.security.manager=allow' at the command line, but we should look at fixing this somehow.

[Baltoli]

I think we probably need to fix openjdk@11 as the required version, and update the docs accordingly: https://formulae.brew.sh/formula/openjdk@11#default

Doing this isn't a viable solution; the Homebrew packages for ant, maven etc. are rolling releases and so don't want to use a non-OpenJDK 19 version of Java. It would require a great deal of unsanitary hacking to get them to work with openjdk@11.

[Baltoli]

#3216 confirms that we don't yet support Java 19 for Nailgun-related reasons.

[radumereuta]

Java 17 should be supported.
The Nailgun issues starts with Java 18.

[Baltoli]

Same problem re. rolling releases whatever Java version below 19 we use, unfortunately.

[ehildenb]

Some notes:

• @dwightguth has some patches that make this work, can they be linked here for tracking?
• We need to change our deployments to Maven/S3. Our Maven packages are currently deployed to S3 bucket, but that code from Maven has bitrotted and isn't being maintained. Need to change to storing these artifacts to GitHub as one option, for example.

Options:

• Push through changes to NailGun (relies on facebook being responsive, need further changes to nailgun, risky).
• Switch to another kserver providers (eg. GraalVM).
• Disable kserver on Macs or detect when it won't work (newer than Java 17).
• Workaround at the top of the issues (disable security check) which will work for Java 18/19, but this will go away for Java 20. Could detect that this will work, and run the kserver that way.

[dwightguth]

Here is the change to upstream nailgun: facebookarchive/nailgun#204
Here is the branch to our fork of nailgun which contains the fix we want to deploy: https://github.com/dwightguth/nailgun/tree/ngsecuritymanagerk
Here is the devops issue we need to fix in order to deploy the change to nailgun: https://github.com/runtimeverification/devops/issues/140

[Robertorosmaninho]

Just as an update on it, https://github.com/facebookarchive/nailgun will not receive updates anymore, as suggested by their note:

In April, 2023, Buck1 was deprecated in favor of Buck2, which does not use Nailgun. As a result, this repository is now unmaintained.

Should we perhaps look for another option?

[dwightguth]

We're essentially maintaining the code we use ourselves at this point, in our fork of nailgun. We can consider other options, but I think it's fine to maintain it ourselves... It doesn't require much maintenance overall.

That being said, graalvm is the way to go if we don't want to use nailgun anymore.

Pros:

• almost as fast as nailgun
• Don't have to maintain it ourselves
• doesn't require you to start a server
• code to get k working is already written

Cons:

• not quite as fast as nailgun
• doesn't support arbitrary reflection, requiring you to keep track of exactly which attributes you reflectively access
• Less platform independent
• tends to cause the application to crash if you try to do something and it's not configured correctly

[dwightguth]

I personally would prefer graalvm because of the advantage of not having to start a server, but people had concerns about the cons I mentioned above when we considered it previously.

[radumereuta]

The kserver currently serves 3 purposes:

• eliminate the startup time by running in the same JVM
• reduce thread contention. We use a lot of threads, and having a single synchronized threadpool helps with performance
• reuse caches. Since we run in a single process, we don't have to reload cached parsers and definitions for each test.

From my understanding, GraalVM would help with the first one but not 2 and 3.
The other thing I was worried about was having a new dependency. K already has so many things that it requires. We had to invent kup because people found it very hard to install our tools.

[Robertorosmaninho]

Thanks for pointing that out, @radumereuta! We'll discuss it further at the next Planning Meeting!

[dwightguth]

Smart IPC could probably solve 2 and 3 even in the GraalVM. But you're correct that it wouldn't "just work", which it does now. We'd have to keep in mind the implementation cost of resolving these problems if we went that route.