fix: improve stack frame sanitization to prevent ReDoS vulnerabilities

[frontegg-david]

Summary by CodeRabbit

• Bug Fixes
  • Improved stack-trace sanitization to prevent ReDoS and handle unusually long or malformed lines, preserving prior redaction behavior while increasing robustness and stability.
  • Enhanced processing to more reliably redact function and path details in stack traces across different formats.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

📝 Walkthrough

Walkthrough

Replaces global-regex stack-trace sanitization with line-by-line processing to mitigate ReDoS and avoid regex backtracking. Adds defensive checks (skip lines >1000 chars, early pattern tests), uses index-based extraction for two stack-line patterns, preserves prior redaction semantics, and makes no public API changes.

Changes

Cohort / File(s)	Summary
VM adapter & double VM wrapper libs/enclave-vm/src/adapters/vm-adapter.ts, libs/enclave-vm/src/double-vm/double-vm-wrapper.ts	Rewrote sanitizeStackTrace to process stack traces line-by-line instead of using global regex replaces. Added defenses: skip overly long lines (>1000 chars), early pattern presence checks, and index-based extraction to avoid regex backtracking. Handles two patterns ("at func (path:line:col)" and "at path:line:col") and returns joined sanitized lines. No exported signatures changed.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• feat: implement safe error handling to prevent prototype chain escape attacks #26: Modifies sanitizeStackTrace in the same files to mitigate ReDoS via tighter stack-trace processing and related changes.

Poem

🐰 I hopped through stacks both long and wide,
I trimmed the lines where dangers hide,
No greedy regex left to chase,
I redacted paths with gentle pace,
A safer trace — one little hop of grace. 🥕

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: improving stack frame sanitization to prevent ReDoS vulnerabilities. Both affected files implement line-by-line processing with defensive checks to prevent regex backtracking attacks.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 0

🧹 Nitpick comments (1)

libs/enclave-vm/src/double-vm/double-vm-wrapper.ts (1)

56-57: Consider redacting or truncating long lines instead of skipping.

Lines exceeding 1000 characters are skipped entirely without redaction. If a maliciously crafted or unusual stack trace contains sensitive paths in a long line, they would leak through unredacted.

Consider either:

1. Truncating the line and appending [REDACTED]
2. Replacing the entire line with [REDACTED - line too long]

♻️ Suggested approach

     // Defense-in-depth: skip very long lines
-    if (line.length > 1000) continue;
+    if (line.length > 1000) {
+      lines[i] = '    at [REDACTED - line too long]';
+      continue;
+    }

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 77e32da and a43661b.

📒 Files selected for processing (2)

• libs/enclave-vm/src/adapters/vm-adapter.ts
• libs/enclave-vm/src/double-vm/double-vm-wrapper.ts

🚧 Files skipped from review as they are similar to previous changes (1)

• libs/enclave-vm/src/adapters/vm-adapter.ts

🧰 Additional context used

📓 Path-based instructions (2)

**/*.{ts,tsx,js,jsx}

📄 CodeRabbit inference engine (AGENTS.md)

**/*.{ts,tsx,js,jsx}: Use ESLint flat config with unused parameters and variables prefixed with underscore _
Prefer workspace import paths (e.g., import { validate } from 'ast-guard') over deep cross-library relative imports

Files:

• libs/enclave-vm/src/double-vm/double-vm-wrapper.ts

libs/**

⚙️ CodeRabbit configuration file

libs/**: Contains publishable SDK libraries. Review for API correctness, breaking changes, and consistency with docs. When public APIs change, ensure there is a matching docs/draft/docs/** update (not direct edits under docs/docs/**).

Files:

• libs/enclave-vm/src/double-vm/double-vm-wrapper.ts

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)

• GitHub Check: Test
• GitHub Check: Performance

🔇 Additional comments (3)

libs/enclave-vm/src/double-vm/double-vm-wrapper.ts (3)

36-40: LGTM!

Clear documentation of the ReDoS vulnerability being mitigated and the approach taken. This helps future maintainers understand the security rationale.

---

75-81: LGTM!

Pattern 2 handling is correct. The regex /:\d+:\d+$/ is anchored at the end, limiting backtracking potential. The combination of line-length limits and pre-checks makes this safe from ReDoS.

The overall line-by-line approach successfully mitigates the ReDoS vulnerability while preserving the redaction semantics.

---

59-73: Pattern 1 does not handle async function stack frames.

The logic extracts the function name by finding the first space after at , then checking if the next character is (. For standard frames like at foo (path:1:2), this works correctly. However, for async frames like at async foo (path:1:2), the first space comes after async (position 5), so the check afterAt.charAt(spaceIdx + 1) === '(' fails with 'f' instead of '(', and the line is not redacted. Pattern 2 also won't match since it expects lines ending with :line:column without parentheses.

While async operations are blocked by default in the enclave, errors could still propagate with async frames in their stack traces. If this needs to be handled, consider:

Potential fix

       const atIdx = line.indexOf('at ');
       if (atIdx !== -1) {
         const afterAt = line.substring(atIdx + 3).trimStart();
+        // Skip "async " prefix if present
+        const prefix = afterAt.startsWith('async ') ? 'async ' : '';
+        const nameStart = afterAt.substring(prefix.length);
-        const spaceIdx = afterAt.indexOf(' ');
+        const spaceIdx = nameStart.indexOf(' ');
         if (spaceIdx !== -1 && afterAt.charAt(spaceIdx + 1 + prefix.length) === '(') {
-          const funcName = afterAt.substring(0, spaceIdx);
+          const funcName = nameStart.substring(0, spaceIdx);
           lines[i] = line.substring(0, atIdx) + 'at ' + funcName + ' ([REDACTED])';
           continue;
         }
       }