Candidate datastore lock prevents sequential commit operations via RESTCONF

[chrissandi]

Hi,

I would like to raise an issue related to RESTCONF behavior during batch service provisioning.

We created a bash script to test batch provisioning of services via the RESTCONF API. The script runs in two phases:

Phase 1 – Create Services

• Executes 20 sequential POST requests to /restconf/data/clixon-controller:services
• Each request creates a VPLS service instance with a unique vpls-id (vk-vlan201 through vk-vlan220)
• All other service parameters are identical (same devices, interfaces, loopback IPs, etc.)
• No delay between requests

Phase 2 – Commit Services

• After all 20 services are created in the candidate datastore, the script executes 20 sequential POST requests to /restconf/operations/clixon-controller:controller-commit
• Each commit targets a specific service instance using the service-instance parameter
• No delay between commit requests

Observed Issue

When executing multiple sequential controller-commit operations via RESTCONF, subsequent commit requests fail with HTTP 412 errors, indicating that the candidate datastore is still locked by the previous transaction. This occurs even though each commit targets a different service instance.

Error response example:

{
  "ietf-restconf:errors": {
    "error": {
      "error-type": "application",
      "error-tag": "operation-failed",
      "error-severity": "error",
      "error-message": "Candidate db is locked by transaction \"Controller commit actions:FORCE push:COMMIT\" initiated by user:anonymous with client-id:1 using transport:cl:restconf"
    }
  }
}

From the client perspective, the requests are sent sequentially and only after the previous request has completed. However, it appears that the datastore lock is not released in time for the next commit, leading to lock contention.

I would appreciate any guidance on whether this behavior is expected.

Thank you.

[olofhagsand]

There are multiple issues at play here.
First, I assume any lock eventually releases so that it is a temporary contention, i.e., it is not a permanent situation.
Second, the service transactions are done sequentially, one at a time. They may not run in parallel, even if they touch different parts of the config/device tree.
Third, the controller-commit rpc initiates a transaction and returns a transaction-id. The POST request is asynchronous meaning that the reply does not indicate that the initiated transaction is completed. This means that an immediate new controller-commit may block waiting for the ongoing transaction to complete.
This is described here: https://clixon-controller-docs.readthedocs.io/en/latest/restconf.html#transactions
You need to wait for the transaction to complete either via register a notification or a polling mechanism as described in the user guide.

[olofhagsand]

The CLI does this for you, ie it waits for a notification before returning. But in RESTCONF you need to do this yourself.

[chrissandi]

In our case, this does not appear to be temporary contention. The candidate datastore enters a locked state that is never released, and once this happens we cannot perform another commit manually.

After checking /clixon-controller:transactions, I only see 4 transactions in total, even though 20 controller-commit requests were sent. Out of these, only 3 correspond to commit attempts, and the last one ends in FAILED / RESOLVED (timeout waiting for action daemon). After this point, the datastore remains locked and no new commit transactions are created.

{
  "clixon-controller:transactions": {
    "transaction": [
      {
        "tid": "1",
        "username": "anonymous",
        "result": "SUCCESS",
        "state": "DONE",
        "description": "Controller connect OPEN",
        "timestamp": "2025-12-31T08:53:59.047385Z"
      },
      {
        "tid": "2",
        "username": "anonymous",
        "result": "SUCCESS",
        "state": "DONE",
        "description": "Controller commit actions:FORCE push:COMMIT",
        "timestamp": "2025-12-31T08:55:17.230104Z"
      },
      {
        "tid": "3",
        "username": "anonymous",
        "result": "SUCCESS",
        "reason": "No device  configuration changed, no push necessary",
        "state": "DONE",
        "description": "Controller commit actions:FORCE push:COMMIT",
        "timestamp": "2025-12-31T08:55:17.292344Z"
      },
      {
        "tid": "4",
        "username": "anonymous",
        "result": "FAILED",
        "reason": "Timeout waiting for action daemon",
        "state": "RESOLVED",
        "description": "Controller commit actions:FORCE push:COMMIT",
        "origin": "Actions",
        "timestamp": "2025-12-31T08:56:17.304271Z"
      }
    ]
  }
}

[olofhagsand]

OK, we have seen that behavior but have not been able to re-create it deterministically.
That error indicates that the backend notifies the action daemon(pyapi) of a new service change, the backend then waits for the action daemon to complete the transaction. But for some reason this does not happen.
The errors are as I see it:

1. pyapi does not receive and/or does not complete the transaction and the backend timeouts
2. the backend does not release the transaction lock after the timeout
   Can you verify that (2) holds as well? I.e., you get a lock error after the "action timeout"?

[chrissandi]

Yes, I can confirm that (2) holds as well. After the “Timeout waiting for action daemon” occurs, the transaction lock is not released, and any commit attempts fail with a lock-related error. The system remains blocked even after waiting well beyond the timeout period.

[olofhagsand]

OK, (2) is a definite bug.
(1) is unclear to us, why it happens.
Could you please send some more details on how to recreate (1) preferably a minimal scenario so that we can recreate it?

I can see there is a weakness in the design where the backend notifies the pyapi and then waits for transaction-actions-done (which doesnt happen). The problem is that the notification is not acked, ie the backend does not know if pyapi has accepted the work. Notifications cannot be acked, so there needs to be another mechanism for the ack, Currently this is the timeout.

[chrissandi]

Steps to Reproduce

For (1), the reproduction is minimal and everything is done from a single script. The steps are only the following:

Steps

1. Create multiple service instances (e.g., VPLS) via POST /restconf/data/clixon-controller:services

• Services differ only by service-id
• All other parameters are identical

Example:

curl --request POST \
  --url http://localhost:8082/restconf/data/clixon-controller:services \
  --header 'accept: application/yang-data+json' \
  --header 'authorization: Basic YWRtaW46YWRtaW4=' \
  --header 'content-type: application/yang-data+json' \
  --data '{
    "vpls:vpls": [{
      "vpls-id": "vk-vlan201",
      "vlan-id": 100,
      "customer-name": "Alv-kommun-Vis-Elev",
      "bridge-domain-group": "vk",
      "bridge-domain-prefix": "22",
      "vfi-prefix": "12",
      "pseudowire-prefix": "1000",
      "hub": {
        "agg-1": {"device": "ok300-agg1","loopback-ip": "172.19.0.8"},
        "agg-2": {"device": "ok1000-agg1","loopback-ip": "172.19.0.5"},
        "interface-id": "0/0/0/1",
        "interface-type": "Bundle-Ether"
      },
      "spokes": [{
        "id": 1,
        "vlan-encapsulation": "single-tagged",
        "dist-1": {
          "device": "oka38-dist2",
          "loopback-ip": "10.105.4.116",
          "interface-id": "0/0/0/19",
          "interface-type": "TenGigE"
        },
        "dist-2": {
          "device": "oka38-dist3",
          "loopback-ip": "10.105.4.142",
          "interface-id": "0/0/0/21",
          "interface-type": "TenGigE"
        }
      }]
    }]
  }'

2. Send burst sequential commits via POST /restconf/operations/clixon-controller:controller-commit

• ~20 commit requests
• No delay between requests
• Each commit targets a single service-instance
• No polling / notifications

Example:

curl --request POST \
  --url http://localhost:8082/restconf/operations/clixon-controller:controller-commit \
  --header 'accept: application/yang-data+json' \
  --header 'content-type: application/yang-data+json' \
  --data '{
    "clixon-controller:input": {
      "device": "*",
      "push": "COMMIT",
      "source": "ds:candidate",
      "actions": "FORCE",
      "service-instance": "vpls[vpls-id='\''vk-vlan201'\'']"
    }
  }'

This is sufficient to reproduce the behavior where only a few transactions are created, followed by an action daemon timeout, after which the backend remains locked.

[olofhagsand]

Thank you for instructions.
Part (2) should now be fixed, ie that the datastore is still locked after a service timeout
However, we still have not understood why it timeouts.