Race condition on Instrumentation.retransformClasses() and class linking

[AlexEndt]

This JDK bug was already submitted by us years ago to OpenJDK: https://bugs.openjdk.org/browse/JDK-8277444
Some of our (Dynatrace's) AWS Lambda customers have experienced it recently. Since the bug is not being addressed by OpenJDK, we hope that maybe corretto wants to address it.

Customers report that this bug occurs rarely (less than 1% of their AWS Lambda invocations), but consistently.

Describe the bug

There is a race condition when a class is simultaneously retransformed via the Instrumentation API and linked.

While retransforming the class bytecode is reconstituted in src\hotspot\share\prims\jvmtiClassFileReconstituter.cpp, where in JvmtiClassFileReconstituter::copy_bytecodes there is a check if a method is already rewritten or not. If the flag is true, some references into the classpool have to be changed.

Now, while linking a class in src\hotspot\share\oops\instanceKlass.cpp in InstanceKlass::link_class_impl there is a call to rewrite the class and then set the is_rewritten flag.

If retransforming and linking both happen simultaneously the class is already (or partly) rewritten but the flag is still false. The reconstituter now does not translate the references back which will result in invalid bytecode, which will cause a VerifyError.

Even worse, if class verifying is turned off, which it is by default for java.* classes, this will cause the JVM to crash/segfault.

To Reproduce

Load/link a (preferably big) class and call Instrumentation.retransformClasses() on it simultaneously.

See the following class and follow the instructions in the comments to reproduce: Main.java
It's a .txt because .java uploads aren't allowed. Note that this reproducer is dependent on ASM 7.3.1, get it here: https://repo1.maven.org/maven2/org/ow2/asm/asm/7.3.1/asm-7.3.1.jar

Potential workaround:
Thread.sleep(1000) before calling retransformClasses() to wait for linking to be finished, since we cannot check if a class is already linked or not. This is not a guaranteed workaround.

Expected behavior

Instrumentation.retransformClasses() works and does does not give invalid bytecode to the ClassFileTransformer.

Platform information

Platform: AWS Lambda
Recently observed with:
- Corretto-21.0.6.7.1
- Corretto-17.0.13.11.1
We have internally reproduced it in both Windows and Linux, with OpenJDK JVM versions 8, 11 and 17