Skip to content

Blocking ExtendLockLifetime might cause ThreadPool starvation #110

@foriequal0

Description

@foriequal0

We hold per-user RedLock for every request. It works fine under a normal load.
But the problem starts when sometimes our server response time gets slower.

The request inbound rate remains the same for our scenario.
However the response time got longer, concurrent request count went higher.
Hence ExtendLockLifetime is called more.

https://github.com/samcook/RedLock.net/blob/master/RedLockNet.SERedis/RedLock.cs#L314
ExtendLockLifetime is called by Timer, and Timer callbacks are dispatched to ThreadPool.
ThreadPool doesn't like blocking tasks, and it is crucial to have performant ThreadPool for performant async C#.
In our case, all threads in the ThreadPool are blocked by the sudden surge of ExtendLockLifetime invocations, thus it causes ASP.NET server to almost stall.

.NET tries to spawn new threads to resolve starvation, but it cannot catch up with the rate of blocked threads with the default threading config. (and they do not recommend manually adjusting it)
https://learn.microsoft.com/en-us/dotnet/core/runtime-config/threading

Even if we change the config to spin up new threads quickly, it is not performant enough with that amount of threads due to the massive number of threads and context switches.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions