Connect secret key mismatch

[smokedlinq]

Recreated the connect deployment and the pods won't start because of the following error:

Error: Unable to initialize the database: Secret key mismatch between database and file; the database was previously used with a different secret key that began with: 400cec1cef079926

I don't see a way to set this key like in the other products, what am I missing?

[colearendt]

Oh no 😞 It looks like you may not be using persistent storage? It is important that the Connect secret key is persistent across deployments. Unfortunately, we have not exposed a way to set it in the helm chart yet. It is stored by default in the data directory, which is ephemeral unless you have enabled persistent storage.

I discussed generically on #252 , but there are two ways to tackle this. Also, we have unfortunately been long delayed on getting a formal doc up on this topic 😅 :

If you want to let Connect manage the key:

• ensure that you have persistent storage enabled (canonical way is through sharedStorage.create = true)

If you want to proactively set the key is likely to:

• use Postgres as your database (Database.Provider=Postgres, and Postgres.URL=..., etc. in Connect config - i.e. the config.Database.Provider value)
• use https://github.com/rstudio/rskey to generate a key
• create a secret with the key in it
• use volumes and volumeMounts to mount the key to /some/path/db/secret.key
• set Database.Dir=/some/path/db/ in Connect config

[smokedlinq]

I am using sharedStorage, this is happening when I am dropping / recreating the namespace. I'll see if I can inject the key file to resolve this.

[colearendt]

Ahhh that makes sense. The reason is that dropping / recreating the namespace presumably does not change the database. So you have a database sticking around from an old installation, but the filesystem / persistent storage has been nuked.

The correct solution here will be to purge the database that you are using. Using an existing database with a new filesystem could definitely cause some very undesirable issues (files Connect expects to exist will not, etc.). I definitely recommend making sure that database cleanup is a part of your "clean install" process!

[dbkegley]

Closing this since the solution falls outside of the helm chart. The best practice here is to use a kubernetes Secret to manage these credentials. If they need to be encrypted then you should enable a secret manager in the cluster

https://docs.posit.co/connect/admin/getting-started/off-host-install/kubernetes-cluster-preparation/#step-3-create-a-secret-containing-a-postgresql-database-password

https://docs.posit.co/connect/admin/getting-started/off-host-install/configure-helm-chart/#step-1-create-your-initial-values.yaml-file