Apple Silicon: pthread_jit_write_protect_np crash with hardened runtime and pty4j-unix-spawn-helper

[kshivang]

Issue Description

PTY4J 0.13.10-1 causes crashes on Apple Silicon (ARM64) macOS when used with hardened runtime code signing. The crash occurs during JVM initialization with pthread_jit_write_protect_np and EXC_BREAKPOINT (SIGTRAP).

Environment

• macOS Version: 15.5 (24F74)
• Hardware: Apple Silicon (ARM64)
• PTY4J Version: 0.13.10-1
• Java Version: Amazon Corretto 22 ARM64
• Code Signing: Developer ID with hardened runtime

Crash Details

Error Type: EXC_BREAKPOINT (SIGTRAP) - Apple Mobile File Integrity (AMFI) fatal report type 309
Stack Trace: Crash occurs in pthread_jit_write_protect_np during Threads::create_vm() JVM initialization
Root Cause: The pty4j-unix-spawn-helper binary is incompatible with Apple Silicon W^X (Write XOR Execute) memory protection under hardened runtime

Steps to Reproduce

1. Create a macOS app bundle using PTY4J 0.13.10-1
2. Sign the app with Developer ID and hardened runtime entitlements
3. Include these entitlements (standard for Compose Desktop apps):

   <key>com.apple.security.cs.allow-jit</key>
   <true/>
   <key>com.apple.security.cs.allow-unsigned-executable-memory</key>
   <true/>
   <key>com.apple.security.cs.disable-library-validation</key>
   <true/>

4. Launch the app on Apple Silicon macOS
5. App crashes immediately with pthread_jit_write_protect_np error

Expected Behavior

App should launch without crashing, as it does in development mode (without hardened runtime).

Actual Behavior

Immediate crash with SIGTRAP during JVM initialization.

Workaround Found

The issue can be resolved by:

1. Removing pty4j-unix-spawn-helper binaries from the PTY4J jar
2. Using only libpty.dylib (similar to IntelliJ IDEA's approach)

This suggests the spawn helper binary is the specific component incompatible with Apple Silicon hardened runtime.

Additional Context

• Development Mode: Works perfectly with ./gradlew run (no hardened runtime)
• IntelliJ IDEA: Works on Apple Silicon because it uses only libpty.dylib, not spawn helper
• PTY4J 0.12.13: Did not have this issue (no spawn helper binary)
• PTY4J 0.13.10-1: Introduced pty4j-unix-spawn-helper which causes the crashes

Question

Is there a way to make pty4j-unix-spawn-helper compatible with Apple Silicon hardened runtime, or should apps exclude it and use only libpty.dylib on Apple Silicon?

This affects any Compose Desktop or JavaFX application using PTY4J that needs to be distributed through the Mac App Store or signed with Developer ID.

[kshivang]

Solution Found! 🎉

Root Cause: The Apple Silicon hardened runtime compatibility issues appear to be specific to PTY4J 0.13.10-1.

Working Solution: Downgrading to PTY4J 0.13.9 resolves the problem completely.

Technical Details:

• PTY4J 0.13.9: Works perfectly with hardened runtime + JIT entitlements on Apple Silicon
• PTY4J 0.13.10-1: Causes JVM crashes during startup (pthread_jit_write_protect_np failures)

Key Changes Between Versions:

Between 0.13.9 and 0.13.10-1, there were significant dependency updates:

• JNA: 5.12.1 → 5.14.0
• SLF4J: 1.7.36 → 2.0.13 (major version jump)
• JetBrains Annotations: 20.1.0 → 24.0.1
• Kotlin: 2.0.20 → 2.1.21

The JNA updates likely introduced changes to native library handling that don't play well with Apple Silicon's hardened runtime restrictions.

Recommendation:

For production applications targeting macOS with hardened runtime (required for notarization), stick with PTY4J 0.13.9 until the Apple Silicon compatibility issues in 0.13.10-1 are resolved.

Our Final Working Configuration:

gradle/libs.versions.toml: pty4j = "0.13.9"

BOSS.entitlements includes JIT entitlements for hardened runtime compatibility.

Application now runs perfectly with hardened runtime enabled and is ready for Apple notarization! ✅