Skip to content

DTLS handshake fails on FIPS-enabled Linux: X25519 not allowed #866

@samthesloth

Description

@samthesloth

Description

The Go SDK fails to establish WebRTC connections on FIPS 140-enabled Linux systems (e.g., Rocky Linux / RHEL with fips=1). The DTLS handshake fails because pion defaults to X25519 for ECDH key exchange, which is not NIST-approved and rejected by Go's FIPS-enforcing crypto stack.

Error

Closing PeerConnection from DTLS CloseNotify
Failed to start manager: handshake error: crypto/ecdh: use of X25519 is not allowed in FIPS 140-only mode
Failed to start SCTP: DTLS not established

Environment

  • OS: Rocky Linux 9 with FIPS mode enabled (/proc/sys/crypto/fips_enabled = 1)
  • Go: 1.26.1 (reads FIPS flag at runtime via crypto/fips140)
  • server-sdk-go: v2.13.2
  • pion/webrtc: v4 (via SDK dependency)

Root Cause

In transport.go, the SettingEngine does not specify DTLS elliptic curves:

se := webrtc.SettingEngine{}
se.SetSRTPProtectionProfiles(dtls.SRTP_AEAD_AES_128_GCM, dtls.SRTP_AES128_CM_HMAC_SHA1_80)
se.SetDTLSRetransmissionInterval(dtlsRetransmissionInterval)
// No SetDTLSEllipticCurves → pion defaults to X25519

When no curves are specified, pion uses X25519 (Curve25519). On FIPS-enabled systems, Go's crypto/ecdh package rejects X25519 since it is not a NIST-approved curve.

Suggested Fix

Add FIPS-compatible elliptic curves to the SettingEngine setup in transport.go:

se := webrtc.SettingEngine{}
se.SetSRTPProtectionProfiles(dtls.SRTP_AEAD_AES_128_GCM, dtls.SRTP_AES128_CM_HMAC_SHA1_80)
se.SetDTLSEllipticCurves(dtlsElliptic.P256, dtlsElliptic.P384) // FIPS-approved curves
se.SetDTLSRetransmissionInterval(dtlsRetransmissionInterval)

This is backward-compatible — P-256 and P-384 are universally supported by WebRTC implementations and LiveKit server already accepts them.

Alternatively, expose a ConnectOption so callers can configure DTLS curves:

func WithDTLSEllipticCurves(curves ...dtlsElliptic.Curve) ConnectOption {
    return func(p *signalling.ConnectParams) {
        p.DTLSEllipticCurves = curves
    }
}

Impact

Any Go SDK client running on a FIPS-enabled system (common in government, defense, and regulated industries) cannot connect to LiveKit rooms. Browser-based clients are unaffected since Chrome/Firefox use their own FIPS-compatible WebRTC stacks.

Workaround

Currently the only workaround is to locally patch transport.go or use a replace directive in go.mod pointing to a patched copy of the SDK.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions