sv_operator/sv_helm

[giscus[bot]]

sv_operator/sv_helm

https://docs.dev.sync.global/sv_operator/sv_helm.html

[simonletort-da]

Feedback from Kaiko (Stef):
It would be great to document the OIDC providers that have been tested: Auth0, Okta, Keycloak

maybe in https://docs.dev.sync.global/sv_operator/sv_helm.html#oidc-provider-requirements

[cathyjung-da]

@simonletort-da I've actually created a doc around Okta and Keycloak, and I'm currently waiting for @moritzkiefer-da to verify the steps (CC. @mikesjewett-da).

[cathyjung-da]

The document is now publicly available in the Community-Contributed Documentation section of the Splice doc site. Documentation updates are published alongside releases, which currently occur every two weeks and are propagated sequentially to DevNet, TestNet, and MainNet. There's no need to request access to the Google Doc.

[gyorgy-balazsi-da]

The following route is in the wrong place on the architecture diagram:

global-domain-<MIGRATION_ID>-cometbft.sv.<YOUR_HOSTNAME>:26<MIGRATION_ID>56 should be routed to port 26656 of service global-domain-<MIGRATION_ID>-cometbft-cometbft-p2p in the sv namespace using the TCP protocol.

It should be on the right side of the "CometBft" box.

[gyorgy-balazsi-da]

There is a mismatch between the hostname and URL conventions vs the splice-cluster-ingress-runbook helm chart, so a workaround is needed. See in the docs above:

The SV operators have decided to follow the following convention for all SV hostnames and URLs:
...
Note that the reference ingress charts provided below do not fully satisfy these requirements, as they omit the enumerator part of the hostname.

This is the workaround I have applied:

Step #1: I printed the template:

NAME_SERVICE_DOMAIN="cns"
  
helm template \
  oci://ghcr.io/digital-asset/decentralized-canton-sync/helm/splice-cluster-ingress-runbook \
  --version ${CHART_VERSION} \
  -f - <<EOF \
  > to-be-fixed-splice-cluster-ingress-runbook-manifest.yaml
cluster:
  hostname: "${YOUR_HOSTNAME}"
  svNamespace: "sv"
withSvIngress: true
spliceDomainNames:
  nameServiceDomain: "${NAME_SERVICE_DOMAIN}"
EOF

Step #2: In the cluster-ingress-runbook-manifest.yaml file, I changed the external URLs like this:
"sequencer-0.sv.sv2.scratch.support.network.canton.global" --> "sequencer-0.sv2.scratch.support.network.canton.global"

Step #3: Applied the fixed manifest with kubectl:

kubectl apply -f cluster-ingress-runbook-manifest.yaml

[waynecollier-da]

Here's a general overview of traffic management that you might find useful:

Traffic management has two main aspects:

Making sure the node has enough CC to purchase more traffic
Making sure the node doesn't run out of traffic before it has time to purchase more.
You might imagine that you can handle this by setting up a lower and upper bound for traffic, so that any time a node reaches the lower bound it immediately buys enough traffic to reach the upper bound; and then making sure that you always have enough CC in the node's wallet to buy at the upper bound for, say, several days or weeks. But traffic topups work don't quite work this way, because the lower bound does not automatically trigger a new topup.

Instead, the node operator sets

a target throughput: how much traffic (in bytes) they think they will consume per second
a minimum topup interval: How long to wait until topping up again
Multiplying these together gives the topup amount. So if you set 20 MB/sec as your throughput and 100 seconds as your interval, your topup amount is 2000 MB.

So when does your node top up in this example? The node tops up traffic when its balance falls below 2000 MB, and it has been at least 100 seconds since the last topup.

This second condition--the time elapsed since the last topup--is where a node can run into trouble: a node gets a burst of activity that drains its traffic balance to zero before the minimum topup interval is complete, and the node goes silent.

Why the time delay, instead of simply topping up immediately? To help the node operator avoid spending down the node's CC without a chance to intervene, in the case of a bug (or an attack) that causes runaway submissions.

This is fundamentally a defensive stance for you as a node operator: you operator need to monitor the node actively to get these settings right. Set the topup interval too short, and you could run out of Canton Coin before you notice it. Set it to long, and your node could run out of traffic before the next topup.

Both problems do come up: a node with a long topup window gets a burst of new user activity, and stops transacting. Or a node has too little Canton Coin in its operator wallet, runs out, and fails to purchase new traffic, which also stops the node from transacting.

The Splice team has provided a sample traffic monitoring dashboard here:

https://github.com/hyperledger-labs/splice/blob/main/cluster/pulumi/infra/grafana-dashboards/canton-network/synchronizer-fees-validator.json

If your node does run completely out of traffic it's possible to top up its balance through another node, either using Denex Gas Station, or by using direct traffic purchase commands via party hosted on a node other than the one that ran into trouble:

https://docs.dev.sync.global/app_dev/validator_api/index.html#buying-traffic