-
Notifications
You must be signed in to change notification settings - Fork 39
Description
The use of WhenAll in https://github.com/microsoft/Omex/blob/main/src/Hosting.Services/OmexStatelessService.cs and it's Stateful equivalent means that if an action throws, the action ends but the service replica is not restarted, since WhenAll waits for all other tasks to end also.
This prevents registering of more than one action which should cause a service replica restart on failure and leaves the developer with only two options in such cases:
- Swallow exceptions, log failures and rely passive monitoring to alert on the service failures. This would require manual intervention to recover.
- Add complex task management logic to the service code, wrapping multiple actions into a single action for registration with the service instance.
Neither of these options is a good solution.
A better approach would be to update these service classes to apply WhenAny to the enumeration of tasks.
If any task throws a non-recoverable exception, an appropriate exception would be thrown here to cause the service replica to recycle and the remaining tasks would be cancelled/ended.
If a single task ends gracefully, remaining tasks are waited using WhenAny again.
This would allow for a single action throwing to trigger service recovery, while also allowing for graceful ending of actions.
Relevant reference docs:
https://learn.microsoft.com/en-us/dotnet/api/microsoft.servicefabric.services.runtime.statelessservice.runasync?view=azure-dotnet#remarks
https://learn.microsoft.com/en-us/dotnet/api/microsoft.servicefabric.services.runtime.statelessservice.runasync?view=azure-dotnet#microsoft-servicefabric-services-runtime-statelessservice-runasync(system-threading-cancellationtoken)