System.NotSupportedException: Durable entities are not supported by the current backend configuration

[jaliyaudagedara]

durabletask-netherite: 1.5.1 says it added support for isolated entities, but it still doesn't seem to work.

Exception:

System.NotSupportedException: Durable entities are not supported by the current backend configuration

Example Project:
AzureFunctionsDemo.Netherite.zip

csproj

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <AzureFunctionsVersion>v4</AzureFunctionsVersion>
    <OutputType>Exe</OutputType>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>
  <ItemGroup>
    <FrameworkReference Include="Microsoft.AspNetCore.App" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker" Version="1.22.0" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.DurableTask" Version="1.1.2" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.DurableTask.Netherite" Version="1.5.1" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http" Version="3.1.0" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore" Version="1.2.1" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.17.2" />
    <PackageReference Include="Microsoft.ApplicationInsights.WorkerService" Version="2.22.0" />
    <PackageReference Include="Microsoft.Azure.Functions.Worker.ApplicationInsights" Version="1.2.0" />
  </ItemGroup>
  <ItemGroup>
    <None Update="host.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
    <None Update="local.settings.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
      <CopyToPublishDirectory>Never</CopyToPublishDirectory>
    </None>
  </ItemGroup>
  <ItemGroup>
    <Using Include="System.Threading.ExecutionContext" Alias="ExecutionContext" />
  </ItemGroup>
</Project>

host.json

{
  "version": "2.0",
  "extensions": {
    "durableTask": {
      "storageProvider": {
        "type": "Netherite"
      }
    }
  },
  "logging": {
    "applicationInsights": {
      "samplingSettings": {
        "isEnabled": true,
        "excludedTypes": "Request"
      },
      "enableLiveMetricsFilters": true
    }
  }
}

local.settings.json

{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "UseDevelopmentStorage=true",
    "EventHubsConnection": "SingleHost",
    "FUNCTIONS_WORKER_RUNTIME": "dotnet-isolated"
  }
}

Entity

public class CounterEntity : TaskEntity<int>
{
    readonly ILogger logger;

    public CounterEntity(ILogger<CounterEntity> logger)
    {
        this.logger = logger;
    }

    public void Add(int amount) => State += amount;

    public void Reset() => State = 0;

    public int Get() => State;

    [Function(nameof(CounterEntity))]
    public Task RunEntityAsync([EntityTrigger] TaskEntityDispatcher dispatcher)
    {
        return dispatcher.DispatchAsync(this);
    }
}

Function

public static class Function1
{
    [Function(nameof(HttpStart))]
    public static async Task<HttpResponseData> HttpStart(
        [HttpTrigger(AuthorizationLevel.Anonymous, "get", "post")] HttpRequestData req,
        [DurableClient] DurableTaskClient client,
        FunctionContext executionContext)
    {
        ILogger logger = executionContext.GetLogger(nameof(HttpStart));

        string instanceId = await client.ScheduleNewOrchestrationInstanceAsync(nameof(EntityOrchestrator));

        logger.LogInformation("Started orchestration with ID = '{instanceId}'.", instanceId);

        return await client.CreateCheckStatusResponseAsync(req, instanceId);
    }

    [Function(nameof(EntityOrchestrator))]
    public static async Task<int> EntityOrchestrator(
        [OrchestrationTrigger] TaskOrchestrationContext context)
    {
        ILogger logger = context.CreateReplaySafeLogger(nameof(EntityOrchestrator));

        var entityId = new EntityInstanceId(nameof(CounterEntity), "myCounter");

        // System.NotSupportedException: Durable entities are not supported by the current backend configuration.
        await context.Entities.CallEntityAsync(entityId, nameof(CounterEntity.Add), 10);

        int currentValue = await context.Entities.CallEntityAsync<int>(entityId, nameof(CounterEntity.Get));

        return currentValue;
    }
}

Verbose Log

                  %%%%%%
                 %%%%%%
            @   %%%%%%    @
          @@   %%%%%%      @@
       @@@    %%%%%%%%%%%    @@@
     @@      %%%%%%%%%%        @@
       @@         %%%%       @@
         @@      %%%       @@
           @@    %%      @@
                %%
                %


Azure Functions Core Tools
Core Tools Version:       4.0.5611 Commit hash: N/A +591b8aec842e333a87ea9e23ba390bb5effe0655 (64-bit)
Function Runtime Version: 4.31.1.22191

[2024-05-02T04:49:24.710Z] Found C:\Users\Jaliya\Desktop\AzureFunctionsDemo.Netherite\AzureFunctionsDemo.Netherite\AzureFunctionsDemo.Netherite.csproj. Using for user secrets file configuration.
[2024-05-02T04:49:25.007Z] Building host: version spec: , startup suppressed: 'False', configuration suppressed: 'False', startup operation id: '04917112-e7b1-43c8-bd7d-3bc7ec16a575'
[2024-05-02T04:49:25.014Z] Reading host configuration file 'C:\Users\Jaliya\Desktop\AzureFunctionsDemo.Netherite\AzureFunctionsDemo.Netherite\bin\Debug\net8.0\host.json'
[2024-05-02T04:49:25.016Z] Host configuration file read:
[2024-05-02T04:49:25.017Z] {
[2024-05-02T04:49:25.017Z]   "version": "2.0",
[2024-05-02T04:49:25.018Z]   "logging": {
[2024-05-02T04:49:25.019Z]     "applicationInsights": {
[2024-05-02T04:49:25.022Z]       "samplingSettings": {
[2024-05-02T04:49:25.023Z]         "isEnabled": true,
[2024-05-02T04:49:25.024Z]         "excludedTypes": "Request"
[2024-05-02T04:49:25.025Z]       },
[2024-05-02T04:49:25.026Z]       "enableLiveMetricsFilters": true
[2024-05-02T04:49:25.027Z]     }
[2024-05-02T04:49:25.028Z]   },
[2024-05-02T04:49:25.029Z]   "extensions": {
[2024-05-02T04:49:25.030Z]     "durableTask": {
[2024-05-02T04:49:25.031Z]       "storageProvider": {
[2024-05-02T04:49:25.032Z]         "type": "Netherite"
[2024-05-02T04:49:25.033Z]       }
[2024-05-02T04:49:25.034Z]     }
[2024-05-02T04:49:25.035Z]   }
[2024-05-02T04:49:25.043Z] }
[2024-05-02T04:49:25.060Z] Extension Bundle not loaded. Loading extensions from C:\Users\Jaliya\Desktop\AzureFunctionsDemo.Netherite\AzureFunctionsDemo.Netherite\bin\Debug\net8.0. BundleConfigured: False, PrecompiledFunctionApp: False, LegacyBundle: False, DotnetIsolatedApp: True, isLogicApp: False
[2024-05-02T04:49:25.062Z] Script Startup resetting load context with base path: 'C:\Users\Jaliya\Desktop\AzureFunctionsDemo.Netherite\AzureFunctionsDemo.Netherite\bin\Debug\net8.0\.azurefunctions'.
[2024-05-02T04:49:25.072Z] Loading startup extension 'NetheriteProvider'
[2024-05-02T04:49:25.123Z] Loaded extension 'NetheriteProvider' (1.0.0.0)
[2024-05-02T04:49:25.134Z] Loading startup extension 'DurableTask'
[2024-05-02T04:49:25.137Z] Loaded extension 'DurableTask' (2.0.0.0)
[2024-05-02T04:49:25.138Z] Loading startup extension 'Startup'
[2024-05-02T04:49:25.139Z] Loaded extension 'Startup' (1.0.0.0)
[2024-05-02T04:49:25.151Z] Reading host configuration file 'C:\Users\Jaliya\Desktop\AzureFunctionsDemo.Netherite\AzureFunctionsDemo.Netherite\bin\Debug\net8.0\host.json'
[2024-05-02T04:49:25.152Z] Host configuration file read:
[2024-05-02T04:49:25.153Z] {
[2024-05-02T04:49:25.154Z]   "version": "2.0",
[2024-05-02T04:49:25.155Z]   "logging": {
[2024-05-02T04:49:25.156Z]     "applicationInsights": {
[2024-05-02T04:49:25.158Z]       "samplingSettings": {
[2024-05-02T04:49:25.159Z]         "isEnabled": true,
[2024-05-02T04:49:25.160Z]         "excludedTypes": "Request"
[2024-05-02T04:49:25.166Z]       },
[2024-05-02T04:49:25.167Z]       "enableLiveMetricsFilters": true
[2024-05-02T04:49:25.168Z]     }
[2024-05-02T04:49:25.169Z]   },
[2024-05-02T04:49:25.170Z]   "extensions": {
[2024-05-02T04:49:25.171Z]     "durableTask": {
[2024-05-02T04:49:25.172Z]       "storageProvider": {
[2024-05-02T04:49:25.173Z]         "type": "Netherite"
[2024-05-02T04:49:25.175Z]       }
[2024-05-02T04:49:25.175Z]     }
[2024-05-02T04:49:25.181Z]   }
[2024-05-02T04:49:25.182Z] }
[2024-05-02T04:49:25.618Z] Using the Netherite storage provider.
[2024-05-02T04:49:26.223Z] Initializing Warmup Extension.
[2024-05-02T04:49:26.232Z] Resolved secret storage provider BlobStorageSecretsRepository
[2024-05-02T04:49:26.364Z] Durable extension configuration loaded: {"httpSettings":{"defaultAsyncRequestSleepTimeMilliseconds":30000},"hubName":"TestHubName","storageProvider":{"KeepInstanceIdsInMemory":true,"EmergencyShutdownOnFatalExceptions":true,"HubName":"TestHubName","StorageConnectionName":"AzureWebJobsStorage","EventHubsConnectionName":"EventHubsConnection","WorkerId":"RAVANA-TPP15","PartitionCount":12,"LoadInformationAzureTableName":"DurableTaskPartitions","FasterTuningParameters":null,"MaxConcurrentActivityFunctions":400,"MaxConcurrentOrchestratorFunctions":160,"OrchestrationDispatcherCount":1,"ActivityDispatcherCount":1,"InstanceCacheSizeMB":3200,"PartitionManagement":"EventProcessorHost","PartitionManagementParameters":null,"ActivityScheduler":"Locavore","CacheOrchestrationCursors":false,"EventBehaviourForContinueAsNew":"Carryover","ThrowExceptionOnInvalidDedupeStatus":true,"TakeStateCheckpointWhenStoppingPartition":false,"MaxNumberBytesBetweenCheckpoints":20971520,"MaxNumberEventsBetweenCheckpoints":10000,"IdleCheckpointFrequencyMs":60000,"UseLocalDirectoryForPartitionStorage":null,"PersistStepsFirst":false,"PersistDequeueCountBeforeStartingWorkItem":false,"PackPartitionTaskMessages":100,"PartitionStartupTimeoutMinutes":15,"TransportLogLevelLimit":"Debug","StorageLogLevelLimit":"Debug","EventLogLevelLimit":"Debug","WorkItemLogLevelLimit":"Debug","ClientLogLevelLimit":"Debug","LoadMonitorLogLevelLimit":"Debug","LogLevelLimit":"Debug"},"tracing":{"traceInputsAndOutputs":false,"allowVerboseLinuxTelemetry":false,"traceReplayEvents":false,"distributedTracingEnabled":false,"distributedTracingProtocol":"HttpCorrelationProtocol","version":"V1"},"notifications":{"eventGrid":null},"maxConcurrentActivityFunctions":400,"maxConcurrentOrchestratorFunctions":160,"maxConcurrentEntityFunctions":null,"localRpcEndpointEnabled":null,"maxEntityOperationBatchSize":5000,"extendedSessionsEnabled":false,"extendedSessionIdleTimeoutInSeconds":30,"maxOrchestrationActions":100000,"overridableExistingInstanceStates":"NonRunningStates","entityMessageReorderWindowInMinutes":30,"useGracefulShutdown":false,"rollbackEntityOperationsOnExceptions":true,"throwStatusExceptionsOnRaiseEvent":null,"useAppLease":true,"storeInputsInOrchestrationHistory":false,"appLeaseOptions":{"renewInterval":"00:00:25","acquireInterval":"00:05:00","leaseInterval":"00:01:00"}}. HubName: TestHubName. AppName: . SlotName: . ExtensionVersion: 2.13.2.
[2024-05-02T04:49:26.429Z] Opened local gRPC endpoint: http://localhost:4001. InstanceId: . Function: . HubName: TestHubName. AppName: . SlotName: . ExtensionVersion: 2.13.2. SequenceNumber: 0.
[2024-05-02T04:49:26.471Z] Initializing Host. OperationId: '04917112-e7b1-43c8-bd7d-3bc7ec16a575'.
[2024-05-02T04:49:26.481Z] Host initialization: ConsecutiveErrors=0, StartupCount=1, OperationId=04917112-e7b1-43c8-bd7d-3bc7ec16a575
[2024-05-02T04:49:26.541Z] Loading functions metadata
[2024-05-02T04:49:26.544Z] Worker indexing is enabled
[2024-05-02T04:49:26.550Z] Fetching metadata for workerRuntime: dotnet-isolated
[2024-05-02T04:49:26.551Z] Reading functions metadata (Worker)
[2024-05-02T04:49:28.971Z] {
[2024-05-02T04:49:28.974Z]   "ProcessId": 29312,
[2024-05-02T04:49:28.975Z]   "RuntimeIdentifier": "win-x64",
[2024-05-02T04:49:28.976Z]   "WorkerVersion": "1.18.0.0",
[2024-05-02T04:49:28.977Z]   "ProductVersion": "1.18.0\u002B1e81e6133d75399dc71a808ae9d06db8f4566d99",
[2024-05-02T04:49:28.978Z]   "FrameworkDescription": ".NET 8.0.4",
[2024-05-02T04:49:28.979Z]   "OSDescription": "Microsoft Windows 10.0.22621",
[2024-05-02T04:49:28.984Z]   "OSArchitecture": "X64",
[2024-05-02T04:49:28.985Z]   "CommandLine": "C:\\Users\\Jaliya\\Desktop\\AzureFunctionsDemo.Netherite\\AzureFunctionsDemo.Netherite\\bin\\Debug\\net8.0\\AzureFunctionsDemo.Netherite.dll --host 127.0.0.1 --port 26596 --workerId c4005bb1-90dc-4021-8168-3c5d54ab018d --requestId 5b556d67-ee31-467f-b3fc-f012c8437d97 --grpcMaxMessageLength 2147483647 --functions-uri http://127.0.0.1:26596/ --functions-worker-id c4005bb1-90dc-4021-8168-3c5d54ab018d --functions-request-id 5b556d67-ee31-467f-b3fc-f012c8437d97 --functions-grpc-max-message-length 2147483647"
[2024-05-02T04:49:28.987Z] }
[2024-05-02T04:49:29.186Z] 3 functions found (Worker)
[2024-05-02T04:49:29.205Z] Reading functions metadata (Custom)
[2024-05-02T04:49:29.218Z] 1 functions found (Custom)
[2024-05-02T04:49:29.250Z] 3 functions loaded
[2024-05-02T04:49:29.255Z] Azure Functions .NET Worker (PID: 29312) initialized in debug mode. Waiting for debugger to attach...
[2024-05-02T04:49:29.259Z] LoggerFilterOptions
[2024-05-02T04:49:29.260Z] {
[2024-05-02T04:49:29.261Z]   "MinLevel": "None",
[2024-05-02T04:49:29.262Z]   "Rules": [
[2024-05-02T04:49:29.263Z]     {
[2024-05-02T04:49:29.264Z]       "ProviderName": null,
[2024-05-02T04:49:29.265Z]       "CategoryName": null,
[2024-05-02T04:49:29.280Z]       "LogLevel": null,
[2024-05-02T04:49:29.281Z]       "Filter": "<AddFilter>b__0"
[2024-05-02T04:49:29.285Z]     },
[2024-05-02T04:49:29.286Z]     {
[2024-05-02T04:49:29.287Z]       "ProviderName": "Microsoft.Azure.WebJobs.Script.WebHost.Diagnostics.SystemLoggerProvider",
[2024-05-02T04:49:29.288Z]       "CategoryName": null,
[2024-05-02T04:49:29.289Z]       "LogLevel": "None",
[2024-05-02T04:49:29.290Z]       "Filter": null
[2024-05-02T04:49:29.291Z]     },
[2024-05-02T04:49:29.292Z]     {
[2024-05-02T04:49:29.293Z]       "ProviderName": "Microsoft.Azure.WebJobs.Script.WebHost.Diagnostics.SystemLoggerProvider",
[2024-05-02T04:49:29.294Z]       "CategoryName": null,
[2024-05-02T04:49:29.295Z]       "LogLevel": null,
[2024-05-02T04:49:29.296Z]       "Filter": "<AddFilter>b__0"
[2024-05-02T04:49:29.297Z]     },
[2024-05-02T04:49:29.303Z]     {
[2024-05-02T04:49:29.311Z]       "ProviderName": "Azure.Functions.Cli.Diagnostics.ColoredConsoleLoggerProvider",
[2024-05-02T04:49:29.313Z]       "CategoryName": null,
[2024-05-02T04:49:29.317Z]       "LogLevel": null,
[2024-05-02T04:49:29.318Z]       "Filter": "<AddFilter>b__0"
[2024-05-02T04:49:29.319Z]     }
[2024-05-02T04:49:29.321Z]   ]
[2024-05-02T04:49:29.323Z] }
[2024-05-02T04:49:29.324Z] LoggerFilterOptions
[2024-05-02T04:49:29.325Z] {
[2024-05-02T04:49:29.326Z]   "MinLevel": "None",
[2024-05-02T04:49:29.327Z]   "Rules": [
[2024-05-02T04:49:29.328Z]     {
[2024-05-02T04:49:29.331Z]       "ProviderName": null,
[2024-05-02T04:49:29.335Z]       "CategoryName": null,
[2024-05-02T04:49:29.337Z]       "LogLevel": null,
[2024-05-02T04:49:29.338Z]       "Filter": "<AddFilter>b__0"
[2024-05-02T04:49:29.339Z]     },
[2024-05-02T04:49:29.340Z]     {
[2024-05-02T04:49:29.341Z]       "ProviderName": "Microsoft.Azure.WebJobs.Script.WebHost.Diagnostics.SystemLoggerProvider",
[2024-05-02T04:49:29.344Z]       "CategoryName": null,
[2024-05-02T04:49:29.344Z]       "LogLevel": "None",
[2024-05-02T04:49:29.349Z]       "Filter": null
[2024-05-02T04:49:29.351Z]     },
[2024-05-02T04:49:29.352Z]     {
[2024-05-02T04:49:29.353Z]       "ProviderName": "Microsoft.Azure.WebJobs.Script.WebHost.Diagnostics.SystemLoggerProvider",
[2024-05-02T04:49:29.356Z]       "CategoryName": null,
[2024-05-02T04:49:29.357Z]       "LogLevel": null,
[2024-05-02T04:49:29.358Z]       "Filter": "<AddFilter>b__0"
[2024-05-02T04:49:29.359Z]     },
[2024-05-02T04:49:29.361Z]     {
[2024-05-02T04:49:29.367Z]       "ProviderName": "Azure.Functions.Cli.Diagnostics.ColoredConsoleLoggerProvider",
[2024-05-02T04:49:29.368Z]       "CategoryName": null,
[2024-05-02T04:49:29.369Z]       "LogLevel": null,
[2024-05-02T04:49:29.371Z]       "Filter": "<AddFilter>b__0"
[2024-05-02T04:49:29.372Z]     }
[2024-05-02T04:49:29.373Z]   ]
[2024-05-02T04:49:29.374Z] }
[2024-05-02T04:49:29.375Z] LanguageWorkerOptions
[2024-05-02T04:49:29.376Z] {
[2024-05-02T04:49:29.377Z]   "WorkerConfigs": [
[2024-05-02T04:49:29.383Z]     {
[2024-05-02T04:49:29.384Z]       "Description": {
[2024-05-02T04:49:29.385Z]         "Language": "dotnet-isolated",
[2024-05-02T04:49:29.386Z]         "DefaultRuntimeName": null,
[2024-05-02T04:49:29.387Z]         "DefaultRuntimeVersion": null,
[2024-05-02T04:49:29.388Z]         "SupportedArchitectures": null,
[2024-05-02T04:49:29.389Z]         "SupportedOperatingSystems": null,
[2024-05-02T04:49:29.390Z]         "SupportedRuntimeVersions": null,
[2024-05-02T04:49:29.391Z]         "SanitizeRuntimeVersionRegex": null,
[2024-05-02T04:49:29.392Z]         "WorkerIndexing": "true",
[2024-05-02T04:49:29.399Z]         "Extensions": [
[2024-05-02T04:49:29.400Z]           ".dll"
[2024-05-02T04:49:29.401Z]         ],
[2024-05-02T04:49:29.402Z]         "UseStdErrorStreamForErrorsOnly": false,
[2024-05-02T04:49:29.403Z]         "DefaultExecutablePath": "C:\\Program Files\\dotnet\\dotnet.exe",
[2024-05-02T04:49:29.404Z]         "DefaultWorkerPath": "C:\\Users\\Jaliya\\Desktop\\AzureFunctionsDemo.Netherite\\AzureFunctionsDemo.Netherite\\bin\\Debug\\net8.0\\AzureFunctionsDemo.Netherite.dll",
[2024-05-02T04:49:29.406Z]         "WorkerDirectory": "C:\\Users\\Jaliya\\Desktop\\AzureFunctionsDemo.Netherite\\AzureFunctionsDemo.Netherite\\bin\\Debug\\net8.0",
[2024-05-02T04:49:29.407Z]         "Arguments": [],
[2024-05-02T04:49:29.412Z]         "WorkerArguments": null
[2024-05-02T04:49:29.414Z]       },
[2024-05-02T04:49:29.415Z]       "Arguments": {
[2024-05-02T04:49:29.416Z]         "ExecutablePath": "C:\\Program Files\\dotnet\\dotnet.exe",
[2024-05-02T04:49:29.417Z]         "ExecutableArguments": [],
[2024-05-02T04:49:29.418Z]         "WorkerPath": "C:\\Users\\Jaliya\\Desktop\\AzureFunctionsDemo.Netherite\\AzureFunctionsDemo.Netherite\\bin\\Debug\\net8.0\\AzureFunctionsDemo.Netherite.dll",
[2024-05-02T04:49:29.419Z]         "WorkerArguments": []
[2024-05-02T04:49:29.420Z]       },
[2024-05-02T04:49:29.421Z]       "CountOptions": {
[2024-05-02T04:49:29.423Z]         "SetProcessCountToNumberOfCpuCores": false,
[2024-05-02T04:49:29.423Z]         "ProcessCount": 1,
[2024-05-02T04:49:29.431Z]         "MaxProcessCount": 10,
[2024-05-02T04:49:29.432Z]         "ProcessStartupInterval": "00:00:10",
[2024-05-02T04:49:29.434Z]         "ProcessStartupTimeout": "30.00:00:00",
[2024-05-02T04:49:29.435Z]         "InitializationTimeout": "30.00:00:00",
[2024-05-02T04:49:29.436Z]         "EnvironmentReloadTimeout": "00:00:30",
[2024-05-02T04:49:29.437Z]         "ProcessRestartInterval": "00:00:10",
[2024-05-02T04:49:29.439Z]         "ProcessShutdownTimeout": "00:00:10"
[2024-05-02T04:49:29.445Z]       }
[2024-05-02T04:49:29.447Z]     }
[2024-05-02T04:49:29.448Z]   ]
[2024-05-02T04:49:29.449Z] }
[2024-05-02T04:49:29.451Z] ConcurrencyOptions
[2024-05-02T04:49:29.452Z] {
[2024-05-02T04:49:29.454Z]   "DynamicConcurrencyEnabled": false,
[2024-05-02T04:49:29.455Z]   "MaximumFunctionConcurrency": 500,
[2024-05-02T04:49:29.458Z]   "CPUThreshold": 0.8,
[2024-05-02T04:49:29.465Z]   "SnapshotPersistenceEnabled": true
[2024-05-02T04:49:29.466Z] }
[2024-05-02T04:49:29.468Z] FunctionResultAggregatorOptions
[2024-05-02T04:49:29.469Z] {
[2024-05-02T04:49:29.470Z]   "BatchSize": 1000,
[2024-05-02T04:49:29.471Z]   "FlushTimeout": "00:00:30",
[2024-05-02T04:49:29.477Z]   "IsEnabled": true
[2024-05-02T04:49:29.478Z] }
[2024-05-02T04:49:29.480Z] SingletonOptions
[2024-05-02T04:49:29.481Z] {
[2024-05-02T04:49:29.482Z]   "LockPeriod": "00:00:15",
[2024-05-02T04:49:29.483Z]   "ListenerLockPeriod": "00:00:15",
[2024-05-02T04:49:29.484Z]   "LockAcquisitionTimeout": "10675199.02:48:05.4775807",
[2024-05-02T04:49:29.485Z]   "LockAcquisitionPollingInterval": "00:00:05",
[2024-05-02T04:49:29.486Z]   "ListenerLockRecoveryPollingInterval": "00:01:00"
[2024-05-02T04:49:29.487Z] }
[2024-05-02T04:49:29.493Z] ScaleOptions
[2024-05-02T04:49:29.508Z] {
[2024-05-02T04:49:29.509Z]   "ScaleMetricsMaxAge": "00:02:00",
[2024-05-02T04:49:29.510Z]   "ScaleMetricsSampleInterval": "00:00:10",
[2024-05-02T04:49:29.511Z]   "MetricsPurgeEnabled": true,
[2024-05-02T04:49:29.512Z]   "IsTargetScalingEnabled": true,
[2024-05-02T04:49:29.513Z]   "IsRuntimeScalingEnabled": false
[2024-05-02T04:49:29.514Z] }
[2024-05-02T04:49:29.516Z] Starting JobHost
[2024-05-02T04:49:29.518Z] Starting Host (HostId=ravanatpp15-1740672217, InstanceId=2f1a15f1-adcf-46b3-9c9e-db9c8ba98210, Version=4.31.1.22191, ProcessId=32232, AppDomainId=1, InDebugMode=False, InDiagnosticMode=False, FunctionsExtensionVersion=(null))
[2024-05-02T04:49:29.581Z] Generating 3 job function(s)
[2024-05-02T04:49:29.583Z] Worker process started and initialized.
[2024-05-02T04:49:29.638Z] Found the following functions:
[2024-05-02T04:49:29.640Z] Host.Functions.CounterEntity
[2024-05-02T04:49:29.641Z] Host.Functions.EntityOrchestrator
[2024-05-02T04:49:29.642Z] Host.Functions.HttpStart
[2024-05-02T04:49:29.643Z]
[2024-05-02T04:49:29.656Z] HttpOptions
[2024-05-02T04:49:29.657Z] Initializing function HTTP routes
[2024-05-02T04:49:29.658Z] Mapped function route 'api/HttpStart' [get,post] to 'HttpStart'
[2024-05-02T04:49:29.659Z]
[2024-05-02T04:49:29.657Z] {
[2024-05-02T04:49:29.665Z]   "DynamicThrottlesEnabled": false,
[2024-05-02T04:49:29.666Z]   "EnableChunkedRequestBinding": false,
[2024-05-02T04:49:29.668Z]   "MaxConcurrentRequests": -1,
[2024-05-02T04:49:29.670Z]   "MaxOutstandingRequests": -1,
[2024-05-02T04:49:29.672Z]   "RoutePrefix": "api"
[2024-05-02T04:49:29.673Z] }
[2024-05-02T04:49:29.668Z] Host initialized (119ms)
[2024-05-02T04:49:29.679Z] Starting task hub worker. Extension GUID 19835f93-2b67-4cf4-abbc-3447b5c97159. InstanceId: . Function: . HubName: TestHubName. AppName: . SlotName: . ExtensionVersion: 2.13.2. SequenceNumber: 1.

Functions:

        HttpStart: [GET,POST] http://localhost:7155/api/HttpStart

        CounterEntity: entityTrigger

        EntityOrchestrator: orchestrationTrigger

[2024-05-02T04:49:29.765Z] Task hub worker started. Latency: 00:00:00.0819941. Extension GUID 19835f93-2b67-4cf4-abbc-3447b5c97159. InstanceId: . Function: . HubName: TestHubName. AppName: . SlotName: . ExtensionVersion: 2.13.2. SequenceNumber: 2.
[2024-05-02T04:49:29.771Z] Host started (230ms)
[2024-05-02T04:49:29.774Z] Job host started
[2024-05-02T04:49:34.299Z] Host lock lease acquired by instance ID '000000000000000000000000CF6A328F'.
[2024-05-02T04:49:37.131Z] Executing HTTP request: {
[2024-05-02T04:49:37.132Z]   "requestId": "18af3dfa-a4b1-46cc-aae3-c3b94b17c92d",
[2024-05-02T04:49:37.133Z]   "method": "GET",
[2024-05-02T04:49:37.134Z]   "userAgent": "PostmanRuntime/7.38.0",
[2024-05-02T04:49:37.135Z]   "uri": "/api/HttpStart"
[2024-05-02T04:49:37.136Z] }
[2024-05-02T04:49:37.233Z] Executing 'Functions.HttpStart' (Reason='This function was programmatically called via the host APIs.', Id=d166e356-cf2b-4110-beef-3795f3de1ec5)
[2024-05-02T04:49:37.574Z] Scheduling new EntityOrchestrator orchestration with instance ID '30d0c36174444304acea30ce96b91e7d' and 0 bytes of input data.
[2024-05-02T04:49:37.683Z] 30d0c36174444304acea30ce96b91e7d: Function 'EntityOrchestrator (Orchestrator)' scheduled. Reason: NewInstance. IsReplay: False. State: Scheduled. RuntimeStatus: Pending. HubName: TestHubName. AppName: . SlotName: . ExtensionVersion: 2.13.2. SequenceNumber: 3.
[2024-05-02T04:49:47.939Z] 30d0c36174444304acea30ce96b91e7d: Function 'EntityOrchestrator (Orchestrator)' started. IsReplay: False. Input: (null). State: Started. RuntimeStatus: Running. HubName: TestHubName. AppName: . SlotName: . ExtensionVersion: 2.13.2. SequenceNumber: 4. TaskEventId: -1
[2024-05-02T04:49:47.958Z] Executing 'Functions.EntityOrchestrator' (Reason='(null)', Id=afcf1d5a-f107-449b-97ba-8910e1b738ea)
[2024-05-02T04:49:48.016Z] Started orchestration with ID = '30d0c36174444304acea30ce96b91e7d'.
[2024-05-02T04:49:48.111Z] Executed 'Functions.HttpStart' (Succeeded, Id=d166e356-cf2b-4110-beef-3795f3de1ec5, Duration=10901ms)
[2024-05-02T04:49:48.141Z] Executed HTTP request: {
[2024-05-02T04:49:48.144Z]   "requestId": "18af3dfa-a4b1-46cc-aae3-c3b94b17c92d",
[2024-05-02T04:49:48.146Z]   "identities": "",
[2024-05-02T04:49:48.148Z]   "status": "202",
[2024-05-02T04:49:48.149Z]   "duration": "11009"
[2024-05-02T04:49:48.152Z] }
[2024-05-02T04:49:48.407Z] An error occurred while executing the orchestrator function 'EntityOrchestrator'.
[2024-05-02T04:49:48.409Z] Result: An error occurred while executing the orchestrator function 'EntityOrchestrator'.
Exception: System.NotSupportedException: Durable entities are not supported by the current backend configuration.
[2024-05-02T04:49:48.412Z]    at DurableTask.Core.Entities.OrchestrationEntityContext.CheckEntitySupport() in /_/src/DurableTask.Core/Entities/OrchestrationEntityContext.cs:line 80
[2024-05-02T04:49:48.415Z]    at DurableTask.Core.Entities.OrchestrationEntityContext.EmitRequestMessage(OrchestrationInstance target, String operationName, Boolean oneWay, Guid operationId, Nullable`1 scheduledTimeUtc, String input) in /_/src/DurableTask.Core/Entities/OrchestrationEntityContext.cs:line 240
[2024-05-02T04:49:48.439Z]    at Microsoft.DurableTask.Worker.Shims.TaskOrchestrationContextWrapper.TaskOrchestrationEntityContext.SendOperationMessage(String instanceId, String operationName, Object input, Boolean oneWay, Nullable`1 scheduledTime)
[2024-05-02T04:49:48.446Z]    at Microsoft.DurableTask.Worker.Shims.TaskOrchestrationContextWrapper.TaskOrchestrationEntityContext.CallEntityInternalAsync(EntityInstanceId id, String operationName, Object input)
[2024-05-02T04:49:48.450Z]    at Microsoft.DurableTask.Worker.Shims.TaskOrchestrationContextWrapper.TaskOrchestrationEntityContext.CallEntityAsync[TResult](EntityInstanceId id, String operationName, Object input, CallEntityOptions options)
[2024-05-02T04:49:48.462Z]    at AzureFunctionsDemo.Netherite.Function1.EntityOrchestrator(TaskOrchestrationContext context) in C:\Users\Jaliya\Desktop\AzureFunctionsDemo.Netherite\AzureFunctionsDemo.Netherite\Function1.cs:line 36
[2024-05-02T04:49:48.473Z]    at AzureFunctionsDemo.Netherite.DirectFunctionExecutor.ExecuteAsync(FunctionContext context) in C:\Users\Jaliya\Desktop\AzureFunctionsDemo.Netherite\AzureFunctionsDemo.Netherite\obj\Debug\net8.0\Microsoft.Azure.Functions.Worker.Sdk.Generators\Microsoft.Azure.Functions.Worker.Sdk.Generators.FunctionExecutorGenerator\GeneratedFunctionExecutor.g.cs:line 46
[2024-05-02T04:49:48.478Z]    at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
[2024-05-02T04:49:48.488Z]    at Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore.FunctionsHttpProxyingMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\extensions\Worker.Extensions.Http.AspNetCore\src\FunctionsMiddleware\FunctionsHttpProxyingMiddleware.cs:line 34
[2024-05-02T04:49:48.492Z]    at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.FunctionsOrchestrator.EnsureSynchronousExecution(FunctionContext functionContext, FunctionExecutionDelegate next, FunctionsOrchestrationContext orchestrationContext) in /_/src/Worker.Extensions.DurableTask/FunctionsOrchestrator.cs:line 81
[2024-05-02T04:49:48.496Z]    at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.FunctionsOrchestrator.RunAsync(TaskOrchestrationContext context, Object input) in /_/src/Worker.Extensions.DurableTask/FunctionsOrchestrator.cs:line 51
Stack:    at DurableTask.Core.Entities.OrchestrationEntityContext.CheckEntitySupport() in /_/src/DurableTask.Core/Entities/OrchestrationEntityContext.cs:line 80
[2024-05-02T04:49:48.507Z]    at DurableTask.Core.Entities.OrchestrationEntityContext.EmitRequestMessage(OrchestrationInstance target, String operationName, Boolean oneWay, Guid operationId, Nullable`1 scheduledTimeUtc, String input) in /_/src/DurableTask.Core/Entities/OrchestrationEntityContext.cs:line 240
[2024-05-02T04:49:48.511Z]    at Microsoft.DurableTask.Worker.Shims.TaskOrchestrationContextWrapper.TaskOrchestrationEntityContext.SendOperationMessage(String instanceId, String operationName, Object input, Boolean oneWay, Nullable`1 scheduledTime)
[2024-05-02T04:49:48.514Z]    at Microsoft.DurableTask.Worker.Shims.TaskOrchestrationContextWrapper.TaskOrchestrationEntityContext.CallEntityInternalAsync(EntityInstanceId id, String operationName, Object input)
[2024-05-02T04:49:48.523Z]    at Microsoft.DurableTask.Worker.Shims.TaskOrchestrationContextWrapper.TaskOrchestrationEntityContext.CallEntityAsync[TResult](EntityInstanceId id, String operationName, Object input, CallEntityOptions options)
[2024-05-02T04:49:48.525Z]    at AzureFunctionsDemo.Netherite.Function1.EntityOrchestrator(TaskOrchestrationContext context) in C:\Users\Jaliya\Desktop\AzureFunctionsDemo.Netherite\AzureFunctionsDemo.Netherite\Function1.cs:line 36
[2024-05-02T04:49:48.527Z]    at AzureFunctionsDemo.Netherite.DirectFunctionExecutor.ExecuteAsync(FunctionContext context) in C:\Users\Jaliya\Desktop\AzureFunctionsDemo.Netherite\AzureFunctionsDemo.Netherite\obj\Debug\net8.0\Microsoft.Azure.Functions.Worker.Sdk.Generators\Microsoft.Azure.Functions.Worker.Sdk.Generators.FunctionExecutorGenerator\GeneratedFunctionExecutor.g.cs:line 46
[2024-05-02T04:49:48.529Z]    at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
[2024-05-02T04:49:48.553Z]    at Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore.FunctionsHttpProxyingMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\extensions\Worker.Extensions.Http.AspNetCore\src\FunctionsMiddleware\FunctionsHttpProxyingMiddleware.cs:line 34
[2024-05-02T04:49:48.543Z] Executed 'Functions.EntityOrchestrator' (Failed, Id=afcf1d5a-f107-449b-97ba-8910e1b738ea, Duration=599ms)
[2024-05-02T04:49:48.562Z] System.Private.CoreLib: Exception while executing function: Functions.EntityOrchestrator. Microsoft.Azure.WebJobs.Extensions.DurableTask: Durable entities are not supported by the current backend configuration.
[2024-05-02T04:49:48.558Z]    at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.FunctionsOrchestrator.EnsureSynchronousExecution(FunctionContext functionContext, FunctionExecutionDelegate next, FunctionsOrchestrationContext orchestrationContext) in /_/src/Worker.Extensions.DurableTask/FunctionsOrchestrator.cs:line 81
[2024-05-02T04:49:48.590Z]    at Microsoft.Azure.Functions.Worker.Extensions.DurableTask.FunctionsOrchestrator.RunAsync(TaskOrchestrationContext context, Object input) in /_/src/Worker.Extensions.DurableTask/FunctionsOrchestrator.cs:line 51.
[2024-05-02T04:49:48.590Z] 30d0c36174444304acea30ce96b91e7d: Function 'EntityOrchestrator (Orchestrator)' failed with an error. Reason: Durable entities are not supported by the current backend configuration.. IsReplay: False. State: Failed. RuntimeStatus: Failed. HubName: TestHubName. AppName: . SlotName: . ExtensionVersion: 2.13.2. SequenceNumber: 5. TaskEventId: -1

[sebastianburckhardt]

Just a little update on what I have figured out so far, nothing definitive yet.

I am honestly a bit stomped... when debugging your sample app (thank you!) I can see in the debugger that the field DurabilityProvider.entityOrchestrationService is null. However, I have so far no idea of how this could happen - it is set at construction time using

this.innerService = service;
this.entityOrchestrationService = service as IEntityOrchestrationService;

and the debugger shows that this.innerService is of type DurableTask.Netherite.NetheriteOrchestrationService which definitely implements IEntityOrchestrationService. It seems impossible?

The problem does not show up in my local build (that is why our tests did not find it), which makes it difficult to debug.
In particular I haven't figured out how to attach the debugger prior to the extension getting constructed so I can step through the constructor call. Need to keep trying to do that.

[sebastianburckhardt]

Now I have a debugger attached just after the two lines above... unfortunately the debugger just confirms the weirdness of this. In immediate window:

(service as IOrchestrationService) == null
false
this.entityOrchestrationService == null
true

[jaliyaudagedara]

@sebastianburckhardt thanks for looking into this.

On a separate note, is there any documentation that we can follow to debug through (like you are doing?).

Yesterday after seeing this issue, I tried debugging through it after downloading the source code for durabletask, durabletask-netherite etc, it got really complicated.

[sebastianburckhardt]

On a separate note, is there any documentation that we can follow to debug through (like you are doing?).

No, I am also struggling to figure it out.

Somehow it looks in the debugger like an older version of Netherite is being picked up even though your project specifies 1.5.1. But I don't know why that would happen.

[jaliyaudagedara]

No, I am also struggling to figure it out.

😢

Somehow it looks in the debugger like an older version of Netherite is being picked up even though your project specifies 1.5.1.
But I don't know why that would happen.

Hmm, interesting.

[sebastianburckhardt]

I am suspecting NuGet package cache issues. This could just be what is happening on my local machine though while I am trying to debug. Not sure if it would actually apply to you.

[jaliyaudagedara]

@sebastianburckhardt

Noticed there were no integration tests for Durable Entities. Added some: #391

But I am not sure whether these are getting executed, not seeing them running in the PR.

[davidmrdavid]

quick theory - I think this is the problem:

durabletask-netherite/src/Functions.Worker.Extensions.DurableTask.Netherite/Properties/AssemblyInfo.cs

Line 7 in f8b5634

[assembly: ExtensionInformation("Microsoft.Azure.DurableTask.Netherite.AzureFunctions", "1.5.0", true)]

^ This file doesn't seem updated, which means the Host process is actually loading an older version of Netherite. That explains what Sebastian was describing here: #390 (comment)

[davidmrdavid]

Yep, that was it. That was hard to debug. Having to manually update this file is problematic, leads to release mistakes, so I'll refactor this AssemblyInfo file to be auto-generated.

FYI @nytian - we may need to trigger another release here, I'll let you know.

[jaliyaudagedara]

Thanks @davidmrdavid, that makes sense!