Corretto 21 JVM crash within EKS

[metodski]

Describe the bug

Our Java application running on Corretto 21.0.7.6.1 experienced a fatal JVM crash caused by a segmentation fault (SIGSEGV). It seems to occur randomly. The app is a short-lived microservice and a lot of instances are created and destroyed each day. It crashes very rarely maybe a few times a week.

Problematic frame:
V [libjvm.so+0x9249d0] JavaThread::is_interrupted(bool)+0x0
The crash happened in the context of a background thread named StatsD-Sender-1, part of the dogstatsd Java client.

To Reproduce

We have not been able to reliably reproduce the issue, but it occurred during normal operation of our service.

Expected behavior

The JVM should remain stable and not crash.

Platform information

OS: Red Hat Enterprise Linux 9.6 (Plow)
Version Corretto-21.0.7.6.1

Additional context

The application was running as a containerized Java process inside a Kubernetes cluster.
Datadog’s Java agent was in use.

hs_error_pid1.log

[eastig]

Hi @metodski,
Thank you for reporting the issue.
The datadog agent has history of crashes: https://github.com/search?q=repo%3ADataDog%2Fdd-trace-java+jvm+crash&type=issues

Can you provide other hs_error logs to check crashes happening in JavaThread::is_interrupted?

We recommend you to report the crash to DataDog: https://github.com/DataDog/dd-trace-java/issues

[simonis]

@jbachorik could you please take a quick look at this issue? Have you seen such crashes before and do you think it is related to Datadog's Java client or do you think it is Corretto specific?

[jbachorik]

Hm, the crash stack does not contain anything obviously DD related. But reporting the crash to DD and opening and escalation will allow us to gather more information in a controlled manner.

[simonis]

Following the disassembly at the PC where the crash occurs (i.e. 0x00007f37b403e9d0, libjvm.so+0x9249d0] or JavaThread::is_interrupted(bool)+0x0. Register rdi is 0 so we get a SIGSEGV with si_code: 1 (SEGV_MAPERR) and si_addr: 0x0000000000000378. rdi is the first argument register so for JavaThread::is_interrupted(bool) it should contain the this pointer (rsi, the first method parameter is 0, i.e. false).

0x00007f37b403e9d0:  48 8B 87 78 03 00 00             mov  rax, qword ptr [rdi + 0x378]
0x00007f37b403e9d7:  48 85 C0                         test rax, rax
0x00007f37b403e9da:  0F 84 80 00 00 00                je   0x7f37b403ea60
0x00007f37b403e9e0:  55                               push rbp
0x00007f37b403e9e1:  48 89 E5                         mov  rbp, rsp
0x00007f37b403e9e4:  41 54                            push r12
0x00007f37b403e9e6:  49 89 FC                         mov  r12, rdi
0x00007f37b403e9e9:  53                               push rbx
0x00007f37b403e9ea:  48 89 C7                         mov  rdi, rax
0x00007f37b403e9ed:  89 F3                            mov  ebx, esi
0x00007f37b403e9ef:  FF 15 3B A3 E0 00                call qword ptr [rip + 0xe0a33b]
0x00007f37b403e9f5:  48 85 C0                         test rax, rax
0x00007f37b403e9f8:  74 46                            je   0x7f37b403ea40
0x00007f37b403e9fa:  49 8B BC 24 78 03 00 00          mov  rdi, qword ptr [r12 + 0x378]
0x00007f37b403ea02:  48 85 FF                         test rdi, rdi
0x00007f37b403ea05:  74 49                            je   0x7f37b403ea50
0x00007f37b403ea07:  FF 15 FB 55 E0 00                call qword ptr [rip + 0xe055fb]
0x00007f37b403ea0d:  48 89 C7                         mov  rdi, rax
0x00007f37b403ea10:  E8 8B 1C FF FF                   call 0x7f37b40306a0
0x00007f37b403ea15:  20 C3                            and  bl, al
0x00007f37b403ea17:  74 1F                            je   0x7f37b403ea38
0x00007f37b403ea19:  49 8B BC 24 78 03 00 00          mov  rdi, qword ptr [r12 + 0x378]
0x00007f37b403ea21:  48 85 FF                         test rdi, rdi
0x00007f37b403ea24:  74 32                            je   0x7f37b403ea58
0x00007f37b403ea26:  FF 15 DC 55 E0 00                call qword ptr [rip + 0xe055dc]
0x00007f37b403ea2c:  48 89 C7                         mov  rdi, rax
0x00007f37b403ea2f:  31 F6                            xor  esi, esi
0x00007f37b403ea31:  E8 8A 1C FF FF                   call 0x7f37b40306c0
0x00007f37b403ea36:  89 D8                            mov  eax, ebx
0x00007f37b403ea38:  5B                               pop  rbx
0x00007f37b403ea39:  41 5C                            pop  r12
0x00007f37b403ea3b:  5D                               pop  rbp
0x00007f37b403ea3c:  C3                               ret  
0x00007f37b403ea3d:  0F 1F 00                         nop  dword ptr [rax]
0x00007f37b403ea40:  5B                               pop  rbx
0x00007f37b403ea41:  31 C0                            xor  eax, eax
0x00007f37b403ea43:  41 5C                            pop  r12
0x00007f37b403ea45:  5D                               pop  rbp
0x00007f37b403ea46:  C3                               ret  

And here's the disassembly of Unsafe_Park() which calls Parker::park(bool, long at ...fc1496 (notice that Parker::park(bool, long is missing from the native stack trace in the hs_err file):

0000000000fc13c0 <Unsafe_Park>:
  fc13c0:	55                   	push   %rbp
  fc13c1:	48 89 e5             	mov    %rsp,%rbp
  fc13c4:	41 57                	push   %r15
  fc13c6:	41 56                	push   %r14
  fc13c8:	41 55                	push   %r13
  fc13ca:	41 54                	push   %r12
  fc13cc:	49 89 fd             	mov    %rdi,%r13
  fc13cf:	53                   	push   %rbx
  fc13d0:	48 8d 9f 48 fc ff ff 	lea    -0x3b8(%rdi),%rbx
  fc13d7:	41 89 d6             	mov    %edx,%r14d
  fc13da:	49 89 cc             	mov    %rcx,%r12
  fc13dd:	48 83 ec 78          	sub    $0x78,%rsp
  fc13e1:	8b 83 68 04 00 00    	mov    0x468(%rbx),%eax
  fc13e7:	2d ad de 00 00       	sub    $0xdead,%eax
  fc13ec:	83 f8 01             	cmp    $0x1,%eax
  fc13ef:	0f 86 f3 01 00 00    	jbe    fc15e8 <Unsafe_Park+0x228>
  fc13f5:	48 8d 05 a4 eb 7a 00 	lea    0x7aeba4(%rip),%rax        # 176ffa0 <UseSystemMemoryBarrier>
  fc13fc:	80 38 00             	cmpb   $0x0,(%rax)
  fc13ff:	c7 83 44 04 00 00 06 	movl   $0x6,0x444(%rbx)
  fc1406:	00 00 00 
  fc1409:	75 05                	jne    fc1410 <Unsafe_Park+0x50>
  fc140b:	f0 83 04 24 00       	lock addl $0x0,(%rsp)
  fc1410:	48 8b 83 48 04 00 00 	mov    0x448(%rbx),%rax
  fc1417:	a8 01                	test   $0x1,%al
  fc1419:	0f 85 b1 01 00 00    	jne    fc15d0 <Unsafe_Park+0x210>
  fc141f:	8b 83 40 04 00 00    	mov    0x440(%rbx),%eax
  fc1425:	a8 0c                	test   $0xc,%al
  fc1427:	74 08                	je     fc1431 <Unsafe_Park+0x71>
  fc1429:	48 89 df             	mov    %rbx,%rdi
  fc142c:	e8 2f 7a 96 ff       	call   928e60 <JavaThread::handle_special_runtime_exit_condition()>
  fc1431:	4c 8d 3d a8 37 7b 00 	lea    0x7b37a8(%rip),%r15        # 1774be0 <JfrEventSetting::_jvm_event_settings>
  fc1438:	c7 83 44 04 00 00 06 	movl   $0x6,0x444(%rbx)
  fc143f:	00 00 00 
  fc1442:	48 c7 45 90 00 00 00 	movq   $0x0,-0x70(%rbp)
  fc1449:	00 
  fc144a:	48 c7 45 98 00 00 00 	movq   $0x0,-0x68(%rbp)
  fc1451:	00 
  fc1452:	c6 45 a0 00          	movb   $0x0,-0x60(%rbp)
  fc1456:	c6 45 a1 00          	movb   $0x0,-0x5f(%rbp)
  fc145a:	41 80 bf 01 01 00 00 	cmpb   $0x0,0x101(%r15)
  fc1461:	00 
  fc1462:	c6 45 a2 00          	movb   $0x0,-0x5e(%rbp)
  fc1466:	0f 85 8c 01 00 00    	jne    fc15f8 <Unsafe_Park+0x238>
  fc146c:	31 d2                	xor    %edx,%edx
  fc146e:	48 8d bd 60 ff ff ff 	lea    -0xa0(%rbp),%rdi
  fc1475:	4d 85 e4             	test   %r12,%r12
  fc1478:	0f 95 c2             	setne  %dl
  fc147b:	48 89 de             	mov    %rbx,%rsi
  fc147e:	e8 3d 16 00 00       	call   fc2ac0 <JavaThreadParkedState::JavaThreadParkedState(JavaThread*, bool)>
  fc1483:	31 f6                	xor    %esi,%esi
  fc1485:	48 8d bb 10 06 00 00 	lea    0x610(%rbx),%rdi
  fc148c:	45 84 f6             	test   %r14b,%r14b
  fc148f:	40 0f 95 c6          	setne  %sil
  fc1493:	4c 89 e2             	mov    %r12,%rdx
  fc1496:	e8 f5 19 d4 ff       	call   d02e90 <Parker::park(bool, long)>
  fc149b:	41 80 bf 01 01 00 00 	cmpb   $0x0,0x101(%r15)

And finally Parker::park(bool, long which calls JavaThread::is_interrupted(bool):

0000000000d02e90 <Parker::park(bool, long)>:
  d02e90:       31 c9                   xor    %ecx,%ecx
  d02e92:       87 0f                   xchg   %ecx,(%rdi)
  d02e94:       85 c9                   test   %ecx,%ecx
  d02e96:       7e 08                   jle    d02ea0 <Parker::park(bool, long)+0x10>
  d02e98:       f3 c3                   repz ret 
  d02e9a:       66 0f 1f 44 00 00       nopw   0x0(%rax,%rax,1)
  d02ea0:       55                      push   %rbp
  d02ea1:       48 89 e5                mov    %rsp,%rbp
  d02ea4:       41 57                   push   %r15
  d02ea6:       41 56                   push   %r14
  d02ea8:       41 55                   push   %r13
  d02eaa:       41 54                   push   %r12
  d02eac:       49 89 d6                mov    %rdx,%r14
  d02eaf:       53                      push   %rbx
  d02eb0:       41 89 f7                mov    %esi,%r15d
  d02eb3:       41 89 f4                mov    %esi,%r12d
  d02eb6:       48 89 fb                mov    %rdi,%rbx
  d02eb9:       48 83 ec 58             sub    $0x58,%rsp
  d02ebd:       66 48 8d 3d bb 70 a2    data16 lea 0xa270bb(%rip),%rdi        # 1729f80 <_GLOBAL_OFFSET_TABLE_+0xb90>
  d02ec4:       00 
  d02ec5:       66 66 48 e8 f3 fc 59    data16 data16 rex.W call 2a2bc0 <__tls_get_addr@plt>
  d02ecc:       ff 
  d02ecd:       31 f6                   xor    %esi,%esi
  d02ecf:       4c 8b 28                mov    (%rax),%r13
  d02ed2:       4c 89 ef                mov    %r13,%rdi
  d02ed5:       e8 f6 1a c2 ff          call   9249d0 <JavaThread::is_interrupted(bool)>
  d02eda:       4d 85 f6                test   %r14,%r14

It copies rdi from r13 before the call which is plausible, because r13 is 0 as well in the hs_err file when the crash happens. It also initializes the first method paramter (i.e. esi) to 0 which is also plausible, because Parker::park() calls JavaThread::is_interrupted() with a false argument:

void Parker::park(bool isAbsolute, jlong time) {

  // Optional fast-path check:
  // Return immediately if a permit is available.
  // We depend on Atomic::xchg() having full barrier semantics
  // since we are doing a lock-free update to _counter.
  if (Atomic::xchg(&_counter, 0) > 0) return;

  JavaThread *jt = JavaThread::current();

  // Optional optimization -- avoid state transitions if there's
  // an interrupt pending.
  if (jt->is_interrupted(false)) {
    return;
  }

So it looks like JavaThread::current() which reads the current JavaThread from a thread local variable returns 0 for the current thread. I'm not sure how this can happen? According to the comment in the assertion in Thread::current() this can happen if a thread has already been detached:

class JavaThread: public Thread {
  ...
  static JavaThread* current() {
    return JavaThread::cast(Thread::current());
  }
  ...
}
...
inline Thread* Thread::current() {
  Thread* current = current_or_null();
  assert(current != nullptr, "Thread::current() called on detached thread");
  return current;
}
...
inline Thread* Thread::current_or_null() {
#ifndef USE_LIBRARY_BASED_TLS_ONLY
  return _thr_current;
#else
  if (ThreadLocalStorage::is_initialized()) {
    return ThreadLocalStorage::thread();
  }
  return nullptr;
#endif
}
...
#ifndef USE_LIBRARY_BASED_TLS_ONLY
// Current thread is maintained as a thread-local variable
THREAD_LOCAL Thread* Thread::_thr_current = nullptr;
#endif

The hs_err file also doesn't contain a list of active Java threads because of:

[error occurred during error reporting (printing all threads), id 0xb, SIGSEGV (0xb) at pc=0x00007f37b46ad69d]

so it seems that indeed the HotSpot internal bookkeeping of Java threads was corrupted.

[metodski]

@simonis Thanks for the explanation. I submitted a ticket in dd-trace-java as well