Add H.264 video E2EE and data channel encryption for Go publishers

[samthesloth]

Summary

The Go SDK currently supports audio sample E2EE (EncryptGCMAudioSample from #573/#583) and PCM track encryption (#674), but there's no equivalent for H.264 video frames or data channel messages. This means Go publishers (screen sharing, server-side compositing, IoT cameras) can't participate in E2EE rooms.

Use Case

We're building a remote desktop system where a Go runner captures the screen via FFmpeg, encodes to H.264, and publishes via the Go SDK. The JS viewer subscribes using the standard built-in roomOpts.encryption + ExternalE2EEKeyProvider + e2ee-worker.js. We need the Go publisher to produce encrypted frames that the JS FrameCryptor can decrypt.

What We Built (fork)

We have a working implementation on a fork that adds:

1. H.264 video frame encryption (encryption.go)

EncryptGCMH264Sample / EncryptGCMH264SampleCustomCipher which match the JS FrameCryptor format exactly:

• Parses H.264 NALUs via findNALUIndices to find the first slice (IDR/non-IDR)
• Leaves everything before the slice + 2 bytes unencrypted (used as AES-GCM AAD)
• Samples with no slice NALUs (SPS/PPS only) pass through unmodified
• Encrypts payload with AES-128-GCM (random 12-byte IV)
• Applies RBSP escaping (writeRBSP) to prevent accidental H.264 start codes
• Frame format: [header][ciphertext+tag][IV][IV_LENGTH][KID]

Corresponding DecryptGCMH264Sample for Go-side decryption.

Uses the existing DeriveKeyFromString (PBKDF2, salt "LKFrameEncryptionKey", 100k iterations) so the same shared passphrase works on both sides.

2. Data channel encryption (data_cryptor.go, e2ee.go)

• DataCryptor encrypts outgoing DataPacket values into EncryptedPacket protobuf wrappers
• Decrypts incoming EncryptedPacket messages back to their original types
• Supports: user messages, chat, RPC request/response/ack, stream headers/chunks/trailers
• ExternalKeyProvider with SetKeyFromPassphrase / SetRawKey and key index support
• room.SetEncryption(&EncryptionOptions{KeyProvider: kp}) — integration point

3. Room/engine integration (room.go, engine.go)

• Room.SetEncryption() wires up the DataCryptor on the engine
• engine.publishDataPacket() encrypts outgoing data
• engine.handleDataPacket() decrypts incoming EncryptedPacket

Compatibility

Tested against the JS SDK's livekit-client.e2ee.worker.mjs:

• JS subscriber decrypts Go-published H.264 frames via FrameCryptor
• JS sends encrypted data (input commands, RPC) -> Go decrypts
• Go sends encrypted data (monitor list, RPC responses) -> JS decrypts
• Shared passphrase derivation is identical on both sides

Scope

File	Change	Lines
encryption.go	H.264 encrypt/decrypt + NALU helpers + RBSP	~246 added
e2ee.go	EncryptionOptions, KeyProvider interface, ExternalKeyProvider	new file
data_cryptor.go	DataCryptor for data packet encrypt/decrypt	new file
engine.go	Wire DataCryptor into publish/receive paths	~32 added
room.go	SetEncryption() method	~8 added

Happy to open a PR if there's interest. Would also like guidance on:

• Whether H.265 NALU support should be included (we only implemented H.264)
• Preferred API surface (we followed the EncryptGCMAudioSample pattern)
• Whether DataCryptor should be opt-in per room.SetEncryption() or automatic when keys are set

[cnderrauber]

We are happy to accept the PR to implement encryption.

Whether H.265 NALU support should be included (we only implemented H.264)
Preferred API surface (we followed the EncryptGCMAudioSample pattern)
Whether DataCryptor should be opt-in per room.SetEncryption() or automatic when keys are set

Good to have H265 support and Audio e2ee like API. For data Cryptor, I preferred to have a room option like EnableDataEncryption as go sdk has individual media track level encryption option so a dedicated data encryption option looks good for user. cc @boks1971 @lukasIO for more suggestions.

[boks1971]

Looks good to me. Agreed with @cnderrauber on the room option. In general, matching JS implementation would be great. Will defer to @lukasIO for nuances as there were some discussions/tricky bits about data channel encryption.

[lukasIO]

What we did in other SDKs that support data channel encryption is to introduce a new field to define e2ee settings in order to make DC encryption the default for new users without breaking things for existing users.

It was our way to let users opt in to DC encryption by setting encryption via this new field.
Agree with @cnderrauber that it might make sense for the go SDK to leave it as opt-in via a dedicated room option.