Skip to content

Deadlock due to Recursive Mutex Locking #173

@APshenkin

Description

@APshenkin

Describe the bug
The library experiences a deadlock when initialization fails due to insufficient lockable memory (mlock) on memory-constrained systems, particularly Android devices with a 64KB mlock limit. When the system has insufficient lockable memory available and memguard attempts to initialize, a deadlock occurs due to recursive mutex locking in the getOrCreateKey function.

Root Cause

  1. The getOrCreateKey function acquires a lock on the key mutex
    keyMtx.Lock()
  2. When memory locking fails (due to mlock limits), an error occurs
    Panic(err)
  3. The error handling calls a Panic function
  4. The Panic function attempts to acquire the same key mutex that is already held
    keyMtx.Lock()
  5. This results in a deadlock, causing the application to freeze

To Reproduce
Steps to reproduce the behaviour:

  1. Set up an Android environment (or create env in some other way with limited mlock)
  2. Consume some of the available lockable memory
  3. Initialize memguard library for the first time (e.g. call memguard.NewEnclave())
  4. Observe application freeze during initialization

Expected behaviour
Getting crash or error

System (please complete the following information):

  • OS and Kernel Versions: Android (64KB mlock limit)
  • Memguard Version:
  • Go Version: 1.24.0

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions