Skip to content

Conversation

@vaind
Copy link
Contributor

@vaind vaind commented Nov 4, 2025

Summary

This PR fixes two critical issues affecting CI reliability and device provider functionality:

1. Mutex Race Condition Fix

Resolves intermittent "Access denied" errors when multiple CI jobs simultaneously attempt to create the same Global mutex.

Root Cause: When multiple processes race to create a mutex, one succeeds while others get UnauthorizedAccessException. The previous fix in PR #15 attempted to handle this with OpenExisting() first, but a narrow race window remained:

  1. Process A: OpenExisting() → fails (doesn't exist)
  2. Process B: OpenExisting() → fails (doesn't exist)
  3. Process A: Creates mutex → succeeds
  4. Process B: Tries to create mutex → UnauthorizedAccessException

Solution:

  • Added retry loop (max 3 attempts with 50ms delay) for mutex creation
  • On UnauthorizedAccessException, retry instead of failing
  • Process B will successfully open the mutex that Process A created on retry

Testing:

  • ✅ All 33 DeviceLock tests pass
  • ✅ Includes test with 5 concurrent processes racing to create same mutex
  • ✅ Multiple stress test runs - all passed

2. DebugOutputForwarder Background Job Fix

Resolves "An empty pipe element is not allowed" error when commands run with timeout in background jobs.

Root Cause: $this.DebugOutputForwarder was used inside scriptblocks executed via Start-Job, where $this context is lost, resulting in an empty pipe element.

Solution:

  • Pass DebugOutputForwarder as parameter to scriptblocks
  • Works correctly in both timeout (background job) and non-timeout (direct execution) scenarios

Testing:

  • ✅ All 50 Device tests pass (13 skipped due to missing test fixtures)

Changes

  • DeviceLockManager.ps1: Added retry mechanism for mutex creation
  • DeviceProvider.ps1: Pass DebugOutputForwarder as scriptblock parameter

Related Issues

Fixes intermittent CI failures in sentry-xbox PR #76

Addresses intermittent "Access denied" errors when multiple CI jobs
simultaneously attempt to create the same Global mutex. The previous fix
in commit 0906fba attempted to handle this by opening existing mutexes first,
but a narrow race window remained:

Race condition timeline:
1. Process A: OpenExisting() → WaitHandleCannotBeOpenedException (doesn't exist)
2. Process B: OpenExisting() → WaitHandleCannotBeOpenedException (doesn't exist)
3. Process A: New-Object Mutex(...) → succeeds
4. Process B: New-Object Mutex(...) → UnauthorizedAccessException (A just created it)

Solution:
- Wrap mutex open/create in retry loop (max 3 attempts)
- On UnauthorizedAccessException, sleep 50ms and retry
- Handles race conditions from both OpenExisting() and New-Object()
- Single unified catch block for cleaner code

This ensures that when process B gets UnauthorizedAccessException, it will
retry and successfully open the mutex that process A just created.

Tested with:
- Existing concurrent process tests (5 jobs racing to create same mutex)
- Multiple stress test runs - all passed

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@vaind vaind force-pushed the chore/mutex-improvements branch from e6c1543 to d01acac Compare November 4, 2025 16:17
vaind and others added 2 commits November 4, 2025 17:19
…ound jobs

Fixes "An empty pipe element is not allowed" error when commands run with timeout
and are executed in background jobs via Start-Job.

The issue occurred because $this.DebugOutputForwarder was evaluated inside a
scriptblock running in a background job, where $this context is lost, resulting
in an empty pipe element.

Changes:
- Added DebugOutputForwarder as class property
- Modified InvokeCommand scriptblock to accept debugForwarder parameter
- Pass DebugOutputForwarder value when starting background job
- Pass DebugOutputForwarder value when executing scriptblock directly

This ensures debug output forwarding works correctly in both timeout
(background job) and non-timeout (direct execution) scenarios.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@vaind vaind changed the title chore: improve intermittent mutex issues recovery fix: resolve mutex race conditions and background job issues Nov 4, 2025
@vaind vaind merged commit 04e0ded into main Nov 4, 2025
12 checks passed
@vaind vaind deleted the chore/mutex-improvements branch November 4, 2025 18:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants