validator_operator/validator_major_upgrades

[giscus[bot]]

validator_operator/validator_major_upgrades

https://docs.dev.sync.global/validator_operator/validator_major_upgrades.html

[branciard]

What does "fresh participant" means ?

fresh participant (a fresh participant needs to be deployed to upgrade across non-backwards-compatible changes to the Canton software)

What is the workflow when migration on major upgrade ?:

• check the "auto" dump well generated
• wait for new charts
• "fresh participant" ? use a new DB or drop content of existing ?
• start only once with migrating=true and new migration id. The initial "auto" dump is used to keep existing participants ? as said :

Upon (re-)initialization, the validator backend automatically consumes the migration dump and initializes the validator participant based on the contents of this dump.)

• check it is ok. how ?
• remove the migrating flag: migrating=false ?
• restart all ( migrating=false but with new migration id )

[julientinguely-da]

A fresh participant is a completely new participant node with a clean and empty database. It's required for major upgrades.

The migration workflow can be summarized in these steps:

• Create a dump file that exports all essential data from your current running participant (parties, contracts, etc.). Check for the Wrote domain migration dump log to ensure it has been created.
• Deploy the new Canton version using a fresh participant. Start it for the first time with a migrating=true flag but keep the incremented migration_id. This tells the new participant to load the state from your dump file into its new database.
• Remove the migrating=true flag but keep the new migration_id. It will now run normally with all your data restored on the new node.
• You can check that it worked successfully by checking through the Ingested transactions logs.

Hope that answers your questions.

[mkocur-ledger]

Basically the same question but some more context is needed.

Upon (re-)initialization, the validator backend automatically consumes the migration dump and initializes the validator participant based on the contents of this dump. App databases are preserved."
https://docs.dev.sync.global/validator_operator/validator_major_upgrades.html#overview

But then when I click the link preserved I can read:

a fresh participant needs to be deployed to upgrade across non-backwards-compatible changes to the Canton software

And as @julientinguely-da mentions:

A fresh participant is a completely new participant node with a clean and empty database.

Another quote from https://docs.dev.sync.global/validator_operator/validator_major_upgrades.html#deploying-the-validator-app-and-participant-kubernetes:

You do not need to redeploy the postgres release. The updated Canton participant will use a new database on the PostgreSQL instance, whereas the validator app will reuse the existing state

So my understanding is that participant will create a new database (participant-$migration_id), provided that we use splice-postgresql chart and we allowed cnadmin user to manage the database. Still, not sure how this is supposed to work since the name is not having $migration_id included:
https://github.com/hyperledger-labs/splice/blob/53d0ea8f47424e2b40b2cc296f441f8446c6f304/cluster/helm/splice-participant/templates/participant.yaml#L57-L58
https://github.com/hyperledger-labs/splice/blob/53d0ea8f47424e2b40b2cc296f441f8446c6f304/cluster/helm/splice-participant/templates/participant.yaml#L155

However, on a production setup one rarely does that. Instead, we have our own database instance (let's say RDS) and we provisioned a database for participant using external tool. Then we configured the participant to use it:

splice-participant:
  persistence:
    host: "mydbhost"
    port: "5432"
    secretName: "participant-postgres"
    databaseName: canton-participant-0
    schema: participant

So what is the flow now?

1. Wait for wrote domain migration dump
2. Stop participant
3. Drop the database and recreate (or create a new one canton-participant-1 and reconfigure participant to use it? But what's the benefit?)
4. Set migrating=true and increment migration_id and bump the version
5. Start participant and check for Ingested transactions

[julientinguely-da]

You’re right step 3 is not really well explained; you do have to change the databaseName to include the new migration id. Otherwise it’s the same as described in my previous comment.

It does not matter which sql backend you use as long as the participant container is able to run (on helm install):
https://github.com/hyperledger-labs/splice/blob/e7960a5ddc0e2fd1a8b64e308a55fe3340007647/cluster/helm/splice-participant/templates/participant.yaml#L155

Thank you for your remark, we will improve these docs to make it clearer.

[martinflorian-da]

The latest version of the docs adds several clarifications on points that came up in this thread. I suggest we close/hide this thread. @waynecollier-da

[derrix060]

Note that if you can't find the Wrote domain migration dump logs, go inside the container where the validator-app is running and check if it does have a migration dump and the dump :

[2025-06-11 18:58:23] ~  $ kubectl exec -it validator-app-5574fb4677-v68zf -- bash
Defaulted container "validator-app" out of: validator-app, validator-init (init)
root@validator-app-5574fb4677-v68zf:/app# ls -lha /domain-upgrade-dump
total 31M
drwxr-xr-x 3 root root 4.0K Jun 11 13:01 .
drwxr-xr-x 1 root root 4.0K Jun  9 11:39 ..
-rw-r--r-- 1 root root  31M Jun 11 13:01 domain_migration_dump.json
drwx------ 2 root root  16K May 19 11:12 lost+found

Please take a note on the day that the migration_dump was created, in case you have migrated more than once before :)

[martinflorian-da]

Thanks for this tip! We've included it in the main docs now. @waynecollier-da we can IMO close/resolve/hide this.

[lucasallirol-da]

Hi team,

I run into the issue where I couldn't find Wrote domain migration dump nor Ingested transaction in my logs (I checked roughly 15 hours after hard migration happened), and struggled quite a lot to determine if my node was caught up or not.
I managed eventually to find the troubleshooting part of this doc, which only exit on the dev doc and not Testnet not mainnet.
Even then I struggled a bit to figure how to inspect the contents of the domain-migration-validator-pvc PVC (Helm) (derrix tip was perfect but I didn't see it soon enough).

In the end my migration dump was there and I could upgrade with no issue. It would be great to add this troubleshooting part on maindoc before next hard migration and add this alternative method to check for migration dump other than checking the logs.
Thanks!

[stas-sbi]

... we can IMO close/resolve/hide this.

Shall we keep it until the docs reach mainnet considering the comments are shared across envs?

[stas-sbi]

To ensure that the validator participant has caught up and the migration dump has been created as expected, operators can check the logs of the validator-app pod for Wrote domain migration dump messages.

To ensure that the validator app has caught up, operators can check the logs of the validator-app pod for the message Ingested transaction. If the latest such message is 10 or more minutes old, the validator app has very likely (with a large safety margin) caught up to the state on the participant, and hence to the state of the existing (paused) synchronizer.

Would a validator create a dump before it has caught up? If no then the check for "Ingested transaction" before migrating seems redundant and should probably be removed to reduce complexity of the procedure

[moritzkiefer-da]

Would a validator create a dump before it has caught up? If no then the check for "Ingested transaction" before migrating seems redundant and should probably be removed to reduce complexity of the procedure

There are two levels of caught up: Participant is caught up and validator is caught up. The validator waits for the participant before creating the dump but not for its own stores to catch up at this point.

[moritzkiefer-da]

If you get an error like "Migration ID was incremented (to 1) but no migration dump for restoring from was specified." you are missing the migrating: true flag (for helm) or -M argument (for docker compose).

[martinflorian-da]

This is part of the main docs now, we can hide/resolve/remove. @waynecollier-da

[moritzkiefer-da]

If you get an error like this, you likely have not upgraded the participant correctly to 0.5.1.

com.digitalasset.canton.common.sequencer.grpc.SequencerInfoLoader.getBootstrapInfoSynchronizerParametersWithRetry' was not successful. Retrying after 3s. Result: Outcome(Left(DeserializationFailure(ValueConversionError(synchronizerId,Fingerprint decoding of `global-domain::1220be58c29e65de40bf273be1dc2b266d43a9a002ea5b18955aeef7aac881bb471a::34-0` failed with: StringConversionError(String contains reserved delimiter `::`.,None))))). 

[moritzkiefer-da]

If you see an error like this

Caused by: java.lang.RuntimeException: validator_backend app initialization: Initialize node failed
        ... 10 more
Caused by: java.lang.RuntimeException: validator_backend app initialization: Restoring participant users data failed
        ... 10 more
Caused by: java.lang.IllegalArgumentException: createUser requires at least one right

you've hit a bug where users with no rights are not correctly restored.

You can work around this by manually removing those users from the dump, e.g., download, and then edit with JQ using

 jq '.participant_users.users |= map(select(.id == "YOUR_USER_WITH_NO_RIGHTS"))' old_dump.json new_dump.json

and then reupload it into the container.

[martinflorian-da]

how can we identify which users are broken, on each boot loop it errors on different ones?

Using the jq commands above it should remove all affected users from the dump?

Another approach however, that can be used before a migration:

If you're able to open a Canton console on your participants before a migration, you can check for affected users using something like this:

participant.ledger_api.users.list(pageSize=100000).users.map(user => (user.id, participant.ledger_api.users.rights.list(user.id, user.identityProviderId))).filter(x => x._1 != "participant_admin" && x._2.actAs.isEmpty && x._2.readAs.isEmpty && x._2.executeAs.isEmpty) 

If you participant.ledger_api.users.delete(...) affected users before the migration, you will not get this issue and will not need to edit your dump.

After the migration, you can recreate the original state via participant.ledger_api.users.create(...).

[martinflorian-da]

So for example:

@ participant.ledger_api.users.list(pageSize=100000).users.map(user => (user.id, participant.ledger_api.users.rights.list(user.id, user.identityProviderId))).filter(x => x._1 != "participant_admin" && x._2.actAs.isEmpty && x._2.readAs.isEmpty && x._2.executeAs.isEmpty) 
res3: Seq[(String, UserRights)] = Vector(
  (
    "MartinTest",
    UserRights(
      actAs = Set(),
      readAs = Set(),
      executeAs = Set(),
      readAsAnyParty = false,
      executeAsAnyParty = false,
      participantAdmin = false,
      identityProviderAdmin = false
    )
  )
)

We have found the problematic user MartinTest. We remove it:

@ participant.ledger_api.users.delete("MartinTest")

Then after the migration we can add it back:

@ participant.ledger_api.users.create("MartinTest") 
res3: User = User(id = "MartinTest", primaryParty = None, isDeactivated = false, annotations = Map(), identityProviderId = "")

[carloscip]

Hey @martinflorian-da trying to check preemptively as shown in here if theres any Ledger-API user without party-level permission, but not able to make the console retrieve it. Getting Request failed for participant.\n GrpcRequestRefusedByServer: PERMISSION_DENIED with category category = GenericErrorCategory(Some(PERMISSION_DENIED),ERROR,None,true,-1,1),\n cause = \"A security-sensitive error has been received\" (using a proper JWT token generated for the admin user). Any ideas what might be the issue? The console connection is properly made as previous logs state Request GetApiInfo has succeeded for participant/ledger-api.

[martinflorian-da]

@carloscip This sounds to me like some issue with your token...

Some ideas for further debugging:

• check the participant logs around the time of your last attempt - these typically contain a bit more information about why the request failed
• try decoding the exact token string you are trying to use to confirm it makes sense (it's a JWT, so base64-encoded JSON)
• confirm that the user named in the token's sub field has admin rights on the participant
• if you're accustomed to using the ledger API via JSON/HTTP directly, you could also attempt the same procedure over there (listing users and their rights)

Note: If you never touched any users or user rights on the participant directly (i.e., if you only managed users over the validator app so far), it is highly unlikely that you have "users without rights".

[carloscip]

@martinflorian-da thanks for the suggestions, will try again and check them to see if I can get another hint on what's the issue. And regarding users permissions, no the only thing done to the users was through the wallet ui (creating users through it). And the admin user that was created via OIDC provider (during validator initialization it should handle all the permissions).

[nicu-da]

If you're seeing errors related to PSQL exceptions during the upgrade you might have old data from another migration in the participant database.

org.postgresql.util.PSQLException: ERROR: operator does not exist: character varying = integer\n  Hint: No operator matches the given name and argument types. You might need to add explicit type casts

To fix it scale down/stop your participant, drop the participant database for the migration that you're trying to migrate to, scale the participant back up and restart the validator app.

[nicu-da]

If you do not have a participant_$MigrationId then re-check that you followed the correct docs to update the participant and use the new database for the participant

[dave-axymos]

In order for the migration to the new synchronizer to be safe and successful, it is important that the validator node is fully caught up on the existing synchronizer before proceeding to Deploying the Validator App and Participant.

• To ensure that the validator participant has caught up and the migration dump has been created as expected, operators can check the logs of the validator-app pod for Wrote domain migration dump messages.

• To ensure that the validator app has caught up, operators can check the logs of the validator-app pod for the message Ingested transaction. If the latest such message is 10 or more minutes old, the validator app has very likely (with a large safety margin) caught up to the state on the participant, and hence to the state of the existing (paused) synchronizer.

• Note that the sequencers of the existing (old) synchronizer will be kept available by SVs for a limited time after the migration to the new synchronizer has been completed. Once they are shut down, the validator will not be able to catch up anymore. You should therefore ensure that your node is caught up and migrated to the new synchronizer in a timely manner after the migration.

So if I check my validator logs for "Ingested transactions", I can see that the last message is more than 10 minutes old:

validator-1  | {"@timestamp":"2025-11-12T13:58:19.931Z","message":"Ingested transaction (\n  updateId = 1220b2b878a5148c9ceb3f1c0115f9d0bdb3b94af1efc4346d77667a12c17afb6647,\n  synchronizerId = global-domain::1220be58c29e...,\n  offset = 405827,\n  recordTime = 2025-11-12T13:58:18.066405Z,\n  newAcsSize = 17,\n  ingestedCreatedEvents = CreatedEvent(\n    contractId = 0065d05ab494dd71ee136009c6b0eae47927675fdc703a0afcaef643c63196d38eca111220151e1beb95f15665d0fe29ab6e9a465ea82cbdb2eb94e28d415c660ccc0bca47,\n    templateId = 3ca1343ab26b...:Splice.ValidatorLicense:ValidatorLivenessActivityRecord,\n    payload = Record(fields = Seq(PartyId(DSO::1220be58c29e...), PartyId(axymos-dev-2::122048c20443...), Record(fields = 18665)))\n  ),\n  numFilteredCreatedEvents = 1\n)","logger_name":"o.l.s.s.d.DbMultiDomainAcsStore:validator=validator_backend/endUserParty=axymos-dev-2::122048c20443","thread_name":"canton-env-ec-84","level":"DEBUG","trace-id":"1fdb5256880c8aaa99eec9a2f3464767","span-name":"ledger ingestion loop","span-id":"8fb38c41a9b0f4d0"}

But if I search for "Wrote migration dump":

docker compose logs validator | grep "Wrote domain migration dump"

It finds no results. I did also try shelling into the running container as @derrix060
mentioned above but there's no migration dump file there

The docs above just say it should be automatically created but no steps to debug if it's not created. Anything I can try?

[nicu-da]

You can try to run the validator with the latest version before the migration, in this case 0.4.25, start it as usual (on the old migration id, without the -M flag) and check that it creates the dump.

[dave-axymos]

thanks @nicu-da! I had left it running on the old version. I'm trialling a restart now to see if it helps. I can see this in the logs:

{
  "@timestamp": "2025-11-13T10:28:24.617Z",
  "message": "Version mismatch detected, please download the latest bundle. Your executable is on 0.4.25, while the application you are connecting to is on 0.5.1",
  "logger_name": "o.l.s.s.a.a.c.MinimalScanConnection:validator=validator_backend",
  "thread_name": "canton-env-ec-32",
  "level": "INFO"
}

Presume I can ignore that for now. I am seeing some Migration dump related logs now but seems to be erroring on a permission denied now:

java.io.FileNotFoundException: /domain-upgrade-dump/2025-11-12T14:00:23.918979Z-acs-snapshot (Permission denied)
    at java.base/java.io.FileOutputStream.open0(Native Method)
    at java.base/java.io.FileOutputStream.open(FileOutputStream.java:289)
    at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:230)
    at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:118)
    at org.lfdecentralizedtrust.splice.migration.DomainMigrationEncoding$.$anonfun$encode$1(DomainMigrationEncoding.scala:34)
    at org.lfdecentralizedtrust.splice.util.BackupDump$.withParentDirectoryFor(BackupDump.scala:56)
    at org.lfdecentralizedtrust.splice.migration.DomainMigrationEncoding$.encode(DomainMigrationEncoding.scala:37)
    at org.lfdecentralizedtrust.splice.validator.migration.DomainMigrationDump.toHttp(DomainMigrationDump.scala:54)
    at org.lfdecentralizedtrust.splice.validator.migration.DomainMigrationDump$.$anonfun$codec$1(DomainMigrationDump.scala:74)
    at io.circe.Encoder$$anon$1.apply(Encoder.scala:84)
    at io.circe.Codec$$anon$4.apply(Codec.scala:93)
    at io.circe.syntax.package$EncoderOps$.asJson$extension(package.scala:24)
    at org.lfdecentralizedtrust.splice.migration.DomainMigrationTrigger.$anonfun$exportMigrationDump$1(DomainMigrationTrigger.scala:119)
    at org.lfdecentralizedtrust.splice.migration.DomainMigrationTrigger.$anonfun$exportMigrationDump$1$adapted(DomainMigrationTrigger.scala:116)
    at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:467)
    at com.daml.executors.QueueAwareExecutorService$TrackingRunnable.run(QueueAwareExecutorService.scala:98)
    at com.daml.metrics.InstrumentedExecutorServiceMetrics$InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorServiceMetrics.scala:190)
    at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
    at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)

{
  "@timestamp": "2025-11-13T10:33:54.149Z",
  "message": "Finished generating DomainMigrationDump(\n  synchronizerId = global-domain::1220be58c29e...,\n  migrationId = 1,\n  participant = NodeIdentitiesDump(id = PAR::axymos-dev-2::122048c20443..., numberOfKeys = 3, authorizedStoreSnapshotSize = 110885, version = Some('0.4.25')),\n  participantUsers = ParticipantUsersData(\n    identityProviders = Seq(),\n    users = Seq(\n      ParticipantUser(\n        id = \"administrator\",\n        primaryParty = Some(axymos-dev-2::122048c20443...),\n        rights = \"com.daml.ledger.javaapi.data.User$Right$CanActAs@2c17bb03\",\n        isDeactivated = false,\n        annotations = Map(),\n        identityProviderId = None()\n      ),\n      ParticipantUser(\n        id = \"alice\",\n        primaryParty = Some(xns-alice::122048c20443...),\n        rights = Seq(\"com.daml.ledger.javaapi.data.User$Right$CanActAs@5e2d7eb9\", \"com.daml.ledger.javaapi.data.User$Right$CanReadAs@786c66ff\"),\n        isDeactivated = false,\n        annotations = \"username\" -> \"alice\",\n        identityProviderId = None()\n      ),\n      ParticipantUser(\n        id = \"app-provider\",\n        primaryParty = Some(app-provider::122048c20443...),\n        rights = \"com.daml.ledger.javaapi.data.User$Right$CanActAs@1ec8eb29\",\n        isDeactivated = false,\n        annotations = Map(),\n        identityProviderId = None()\n      ),\n      ParticipantUser(\n        id = \"bob\",\n        primaryParty = Some(xns-bob::122048c20443...),\n        rights = Seq(\"com.daml.ledger.javaapi.data.User$Right$CanActAs@f169450\", \"com.daml.ledger.javaapi.data.User$Right$CanReadAs@3afbebbd\"),\n        isDeactivated = false,\n        annotations = \"username\" -> \"bob\",\n        identityProviderId = None()\n      ),\n      ParticipantUser(\n        id = \"carol\",\n        primaryParty = Some(xns-carol::122048c20443...),\n        rights = Seq(\"com.daml.ledger.javaapi.data.User$Right$CanActAs@21bb3983\", \"com.daml.ledger.javaapi.data.User$Right$CanReadAs@78f31f3e\"),\n        isDeactivated = false,\n        annotations = \"username\" -> \"carol\",\n        identityProviderId = None()\n      ),\n      ParticipantUser(\n        id = \"ledger-api-user\",\n        primaryParty = Some(axymos-dev-2::122048c20443...),\n        rights = Seq(\n          \"com.daml.ledger.javaapi.data.User$Right$ParticipantAdmin@7eb83daf\",\n          \"com.daml.ledger.javaapi.data.User$Right$CanActAs@437cb064\",\n          \"com.daml.ledger.javaapi.data.User$Right$CanActAs@26bf1db5\",\n          \"com.daml.ledger.javaapi.data.User$Right$CanActAs@5eb04fc0\"\n        ),\n        isDeactivated = false,\n        annotations = Map(),\n        identityProviderId = None()\n      ),\n      ParticipantUser(\n        id = \"operator\",\n        primaryParty = Some(xns-operator::122048c20443...),\n        rights = Seq(\n          \"com.daml.ledger.javaapi.data.User$Right$CanActAs@3e5aa1c7\",\n          \"com.daml.ledger.javaapi.data.User$Right$CanReadAs@1803d9f8\",\n          \"com.daml.ledger.javaapi.data.User$Right$CanActAs@468f3df8\",\n          \"com.daml.ledger.javaapi.data.User$Right$CanReadAs@1a9da4a6\"\n        ),\n        isDeactivated = false,\n        annotations = \"username\" -> \"operator\",\n        identityProviderId = None()\n      ),\n      ParticipantUser(\n        id = \"participant_admin\",\n        primaryParty = None(),\n        rights = \"com.daml.ledger.javaapi.data.User$Right$ParticipantAdmin@7eb83daf\",\n        isDeactivated = false,\n        annotations = Map(),\n        identityProviderId = None()\n      ),\n      ParticipantUser(id = \"provider\", primaryParty = Some(xns-provider::122048c20443...), rights = Seq(), isDeactivated = false, annotations = \"username\" -> \"provider\", identityProviderId = None()),\n      ParticipantUser(\n        id = \"test-nikki-1\",\n        primaryParty = Some(test-nikki-1::122048c20443...),\n        rights = \"com.daml.ledger.javaapi.data.User$Right$CanActAs@71a26aab\",\n        isDeactivated = false,\n        annotations = Map(),\n        identityProviderId = None()\n      ),\n      ParticipantUser(\n        id = \"xns-provider::122048c20443eadfb9cd7f705e0ecfe151a1c2ed1db68db23e6666263e92378a8f17\",\n        primaryParty = Some(xns-provider::122048c20443...),\n        rights = Seq(\"com.daml.ledger.javaapi.data.User$Right$CanActAs@7768b17e\", \"com.daml.ledger.javaapi.data.User$Right$CanReadAs@48b72296\"),\n        isDeactivated = false,\n        annotations = \"username\" -> \"xns-provider::122048c20443eadfb9cd7f705e0ecfe151a1c2ed1db68db23e6666263e92378a8f17\",\n        identityProviderId = None()\n      ),\n      ParticipantUser(\n...",
  "logger_name": "o.l.s.v.m.DomainMigrationDumpGenerator:validator=validator_backend",
  "thread_name": "canton-env-ec-50",
  "level": "INFO",
  "trace-id": "6baa73022a4a7b32105cb1ee9e27d743",
  "span-name": "DecentralizedSynchronizerMigrationTrigger",
  "span-id": "44e56e114291bd4c"
}
{
  "@timestamp": "2025-11-13T10:33:54.149Z",
  "message": "Checking whether the task is stale, as its processing failed with ",
  "logger_name": "o.l.s.v.m.DecentralizedSynchronizerMigrationTrigger:validator=validator_backend",
  "thread_name": "canton-env-ec-76",
  "level": "INFO",
  "stack_trace": "java.io.FileNotFoundException: /domain-upgrade-dump/2025-11-12T14:00:23.918979Z-acs-snapshot (Permission denied)\n\tat java.base/java.io.FileOutputStream.open0(Native Method)\n\tat java.base/java.io.FileOutputStream.open(FileOutputStream.java:289)\n\tat java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:230)\n\tat java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:118)\n\tat org.lfdecentralizedtrust.splice.migration.DomainMigrationEncoding$.$anonfun$encode$1(DomainMigrationEncoding.scala:34)\n\tat org.lfdecentralizedtrust.splice.util.BackupDump$.withParentDirectoryFor(BackupDump.scala:56)\n\tat org.lfdecentralizedtrust.splice.migration.DomainMigrationEncoding$.encode(DomainMigrationEncoding.scala:37)\n\tat org.lfdecentralizedtrust.splice.validator.migration.DomainMigrationDump.toHttp(DomainMigrationDump.scala:54)\n\tat org.lfdecentralizedtrust.splice.validator.migration.DomainMigrationDump$.$anonfun$codec$1(DomainMigrationDump.scala:74)\n\tat io.circe.Encoder$$anon$1.apply(Encoder.scala:84)\n\tat io.circe.Codec$$anon$4.apply(Codec.scala:93)\n\tat io.circe.syntax.package$EncoderOps$.asJson$extension(package.scala:24)\n\tat org.lfdecentralizedtrust.splice.migration.DomainMigrationTrigger.$anonfun$exportMigrationDump$1(DomainMigrationTrigger.scala:119)\n\tat org.lfdecentralizedtrust.splice.migration.DomainMigrationTrigger.$anonfun$exportMigrationDump$1$adapted(DomainMigrationTrigger.scala:116)\n\tat scala.concurrent.impl.Promise$Transformation.run(Promise.scala:467)\n\tat com.daml.executors.QueueAwareExecutorService$TrackingRunnable.run(QueueAwareExecutorService.scala:98)\n\tat com.daml.metrics.InstrumentedExecutorServiceMetrics$InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorServiceMetrics.scala:190)\n\tat java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423)\n\tat java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)\n\tat java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)\n\tat java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)\n\tat java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)\n\tat java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)\n",
  "trace-id": "6baa73022a4a7b32105cb1ee9e27d743",
  "span-name": "DecentralizedSynchronizerMigrationTrigger",
  "span-id": "44e56e114291bd4c"
}
{
  "@timestamp": "2025-11-13T10:33:54.150Z",
  "message": "Task that failed with following exception is not stale: /domain-upgrade-dump/2025-11-12T14:00:23.918979Z-acs-snapshot (Permission denied) ",
  "logger_name": "o.l.s.v.m.DecentralizedSynchronizerMigrationTrigger:validator=validator_backend",
  "thread_name": "canton-env-ec-76",
  "level": "DEBUG",
  "trace-id": "6baa73022a4a7b32105cb1ee9e27d743",
  "span-name": "DecentralizedSynchronizerMigrationTrigger",
  "span-id": "44e56e114291bd4c"
}
{
  "@timestamp": "2025-11-13T10:33:54.151Z",
  "message": "The operation 'processTaskWithRetry' has failed with an exception. Total maximum number of retries 35 exceeded. Giving up. ",
  "logger_name": "o.l.s.v.m.DecentralizedSynchronizerMigrationTrigger:validator=validator_backend",
  "thread_name": "canton-env-ec-76",
  "level": "INFO",
  "stack_trace": "java.io.FileNotFoundException: /domain-upgrade-dump/2025-11-12T14:00:23.918979Z-acs-snapshot (Permission denied)\n\tat java.base/java.io.FileOutputStream.open0(Native Method)\n\tat java.base/java.io.FileOutputStream.open(FileOutputStream.java:289)\n\tat java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:230)\n\tat java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:118)\n\tat org.lfdecentralizedtrust.splice.migration.DomainMigrationEncoding$.$anonfun$encode$1(DomainMigrationEncoding.scala:34)\n\tat org.lfdecentralizedtrust.splice.util.BackupDump$.withParentDirectoryFor(BackupDump.scala:56)\n\tat org.lfdecentralizedtrust.splice.migration.DomainMigrationEncoding$.encode(DomainMigrationEncoding.scala:37)\n\tat org.lfdecentralizedtrust.splice.validator.migration.DomainMigrationDump.toHttp(DomainMigrationDump.scala:54)\n\tat org.lfdecentralizedtrust.splice.validator.migration.DomainMigrationDump$.$anonfun$codec$1(DomainMigrationDump.scala:74)\n\tat io.circe.Encoder$$anon$1.apply(Encoder.scala:84)\n\tat io.circe.Codec$$anon$4.apply(Codec.scala:93)\n\tat io.circe.syntax.package$EncoderOps$.asJson$extension(package.scala:24)\n\tat org.lfdecentralizedtrust.splice.migration.DomainMigrationTrigger.$anonfun$exportMigrationDump$1(DomainMigrationTrigger.scala:119)\n\tat org.lfdecentralizedtrust.splice.migration.DomainMigrationTrigger.$anonfun$exportMigrationDump$1$adapted(DomainMigrationTrigger.scala:116)\n\tat scala.concurrent.impl.Promise$Transformation.run(Promise.scala:467)\n\tat com.daml.executors.QueueAwareExecutorService$TrackingRunnable.run(QueueAwareExecutorService.scala:98)\n\tat com.daml.metrics.InstrumentedExecutorServiceMetrics$InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorServiceMetrics.scala:190)\n\tat java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423)\n\tat java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)\n\tat java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)\n\tat java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)\n\tat java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)\n\tat java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)\n",
  "span-id": "3e95ad7bb47b9ba3",
  "span-parent-id": "44e56e114291bd4c",
  "trace-id": "6baa73022a4a7b32105cb1ee9e27d743",
  "span-name": "processTaskWithRetry"
}
{
  "@timestamp": "2025-11-13T10:33:54.152Z",
  "message": "Skipping processing of \nReadyTask(readyAt = 2025-11-13T10:31:39.893820Z, work = Task(migrationId = 1))\ndue to unexpected failure",
  "logger_name": "o.l.s.v.m.DecentralizedSynchronizerMigrationTrigger:validator=validator_backend",
  "thread_name": "canton-env-ec-76",
  "level": "ERROR",
  "stack_trace": "java.io.FileNotFoundException: /domain-upgrade-dump/2025-11-12T14:00:23.918979Z-acs-snapshot (Permission denied)\n\tat java.base/java.io.FileOutputStream.open0(Native Method)\n\tat java.base/java.io.FileOutputStream.open(FileOutputStream.java:289)\n\tat java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:230)\n\tat java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:118)\n\tat org.lfdecentralizedtrust.splice.migration.DomainMigrationEncoding$.$anonfun$encode$1(DomainMigrationEncoding.scala:34)\n\tat org.lfdecentralizedtrust.splice.util.BackupDump$.withParentDirectoryFor(BackupDump.scala:56)\n\tat org.lfdecentralizedtrust.splice.migration.DomainMigrationEncoding$.encode(DomainMigrationEncoding.scala:37)\n\tat org.lfdecentralizedtrust.splice.validator.migration.DomainMigrationDump.toHttp(DomainMigrationDump.scala:54)\n\tat org.lfdecentralizedtrust.splice.validator.migration.DomainMigrationDump$.$anonfun$codec$1(DomainMigrationDump.scala:74)\n\tat io.circe.Encoder$$anon$1.apply(Encoder.scala:84)\n\tat io.circe.Codec$$anon$4.apply(Codec.scala:93)\n\tat io.circe.syntax.package$EncoderOps$.asJson$extension(package.scala:24)\n\tat org.lfdecentralizedtrust.splice.migration.DomainMigrationTrigger.$anonfun$exportMigrationDump$1(DomainMigrationTrigger.scala:119)\n\tat org.lfdecentralizedtrust.splice.migration.DomainMigrationTrigger.$anonfun$exportMigrationDump$1$adapted(DomainMigrationTrigger.scala:116)\n\tat scala.concurrent.impl.Promise$Transformation.run(Promise.scala:467)\n\tat com.daml.executors.QueueAwareExecutorService$TrackingRunnable.run(QueueAwareExecutorService.scala:98)\n\tat com.daml.metrics.InstrumentedExecutorServiceMetrics$InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorServiceMetrics.scala:190)\n\tat java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423)\n\tat java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)\n\tat java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)\n\tat java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)\n\tat java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)\n\tat java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)\n",
  "trace-id": "6baa73022a4a7b32105cb1ee9e27d743",
  "span-name": "DecentralizedSynchronizerMigrationTrigger",
  "span-id": "44e56e114291bd4c"
}

It seems to be stuck in a loop retrying this at the moment

[moritzkiefer-da]

Make sure that you also use the 0.4.25 docker compose files and not older compose files with IMAGE_TAG=0.4.25. Those must always match

[dave-axymos]

Make sure that you also use the 0.4.25 docker compose files and not older compose files with IMAGE_TAG=0.4.25. Those must always match

thanks @moritzkiefer-da, that was it!

I'd some config changes in an old file and I ended up copying in an old one, so was missing the chown-domain-upgrade-dump set-up. Sorry about that!

I'll make sure to do a proper diff from now when upgrading so I don't lose changes

[Junyih]

After confirming the migration dump has been created, should I restart the validator with latest version directly, or the same version creates the dump, to consume the dump file and migrate?

[martinflorian-da]

The former, i.e., "restart the validator with latest version directly". Will clarify this in the next iteration on these docs, thanks for bringing it up!

[Junyih]

Thank you! I did restart with same version and the validator didn't start up. great to clarity this on the doc.

[monsieurleberre]

Hello, I am reading the migration guide in here

The validator app will initialize the participant from a clean slate based on the migration ID configured in the validator app. A fresh participant is needed in order to upgrade across non-backwards-compatible changes to the Canton software.

I would also like to migrate our participant to use and external KMS (aws).
According to this document, and our tests in DevNet and Testnet, the only thing required is to change the overrides on the participant YAML value file.

==> It seems like the migration would be the perfect moment to migrate our production participant from a non KMS one to a KMS-enabled one.

Can someone please let me know if I missed something and this is not the case?

[martinflorian-da]

Hey @monsieurleberre! Thanks again for asking here so we have a better record!

During a synchronizer migration you retain all of your current cryptographic keys; the "fresh participant" mentioned in the docs is initialized from 0 but with the same keys you had on the old synchronizer.

If you had a KMS validator before, it will export the key IDs your participant used on the lower migration ID and will then configure your participant on the higher migration ID to use those keys.

If you had a validator without an external KMS before, it will export the keys from your participant on the lower migration ID and will then import them into your participant on the higher migration ID.

There is sadly nothing helpful in this procedure for migrating from non-KMS to KMS. To migrate to KMS in a secure way, you need to rotate your keys, and doing this at the same time with a synchronizer migration has no benefit (...but the downside of compounding operational complexity...).

Note to team: Let's consider adding this to the main docs.

[monsieurleberre]

Thanks a lot for the response, we will definitely not do that at the same time as the upgrade then, I agree that the added complexity is very unnecessary here!

[tomwassing]

Hi team! I wanted to clarify one step in the backup procedure. We're using a KMS in combination with a wrapper-key. Can you confirm that when exporting the identities via /api/validator/v0/admin/participant/identities it exports the unencrypted key pair?

[martinflorian-da]

@tomwassing the KMS mode we're currently able to officially support for Splice is External Key Storage (see https://docs.dev.sync.global/validator_operator/validator_security.html).

For Encrypted Private Key Storage / wrapper-key...

it exports the unencrypted key pair?

My best guess is that the answer is "yes", but I can't confirm to you with certainty, or even guarantee that this will work at all.

(When using external key storage, the dump contains only key IDs, not actual keys.)

[martinflorian-da]

A few clarifications on "Deploying the Validator App and Participant".

This section is about deploying on the new migration ID.

Only start with the steps there once you have confirmed that a migration dump has been written. (See Catching Up Before the Migration.)

k8s/Helm

Follow the steps in https://docs.dev.sync.global/validator_operator/validator_major_upgrades.html#deploying-the-validator-app-and-participant-kubernetes

Clarifications:

Please make sure to pick the correct (incremented) MIGRATION_ID when following the steps.

So:

• pick a new persistence.databaseName for the participant based on the new migration ID: https://github.com/hyperledger-labs/splice/blob/41a8961071cf1575d8f7db1f62b44d006b053694/apps/app/src/pack/examples/sv-helm/participant-values.yaml#L5
• set migration.id to the new migration ID on the validator app: https://github.com/hyperledger-labs/splice/blob/41a8961071cf1575d8f7db1f62b44d006b053694/apps/app/src/pack/examples/sv-helm/standalone-validator-values.yaml#L17
• don't forget to also set migration.migrating: true for the first deployment, as per the guide!

Notably: No changes to the validator app database are necessary; this one is reused by the validator app across all upgrades, major or minor.

For the upcoming MainNet upgrade, all required changes, with the expected correct values (pending final confirmation from the SVs), are:

• persistence.databaseName: participant_4
• migration.id: "4"
• migration.migrating: true (revert back to false after first successful deployment on new migration ID)

Docker compose

Follow the steps in https://docs.dev.sync.global/validator_operator/validator_major_upgrades.html#deploying-the-validator-app-and-participant-docker-compose

Restart the validator, while updating the migration ID in the -m argument

This already takes care of picking a fresh database for the participant, so no further manual config changes are necessary.

the old participant database (participant=-<OLD_MIGRATION_ID>) is no longer used

There is a typo here; the = should be removed.

So for the upcoming MainNet upgrade, the expected correct values (pending final confirmation from the SVs) are:

• ./start.sh (...) -m 4 (...)
• participant-3 (Only relevant some time after the successful migration, in case you want to prune this database.)

Don't forget to also set -M for the first deployment, as per the guide!

Note on old participant database

The old participant database (participant_3 / participant-3) will not be deleted automatically!