Can't start JetStream: could not create storage directory - mkdir /var/lib/ocis/nats: permission denied

[dschmidt]

I updated oCIS today and after I cleared all my pvcs for testing, the nats service refuses to come up:

2025-03-27T20:34:50Z INF Starting nats-server line=github.com/owncloud/ocis/v2/services/nats/pkg/logging/nats.go:21 service=nats │
2025-03-27T20:34:50Z INF Version: 2.10.22 line=github.com/owncloud/ocis/v2/services/nats/pkg/logging/nats.go:21 service=nats ││ 2025-03-27T20:34:50Z INF Git: [1b37ad3] line=github.com/owncloud/ocis/v2/services/nats/pkg/logging/nats.go:21 service=nats │
2025-03-27T20:34:50Z DBG Go build: go1.22.12 line=github.com/owncloud/ocis/v2/services/nats/pkg/logging/nats.go:45 service=nats │
2025-03-27T20:34:50Z INF Cluster: ocis-cluster line=github.com/owncloud/ocis/v2/services/nats/pkg/logging/nats.go:21 service=nats │
2025-03-27T20:34:50Z INF Name: NB2SWWUZWZ6FGBJQNBPIQUKKEAO6GDJPYQ3KDLJOUN6VQ5W463LRYIYR line=github.com/owncloud/ocis/v2/services/nats/pkg/logging/nats.go:21 service=nats │
2025-03-27T20:34:50Z INF Node: 520V1e5t line=github.com/owncloud/ocis/v2/services/nats/pkg/logging/nats.go:21 service=nats │
2025-03-27T20:34:50Z INF ID: NB2SWWUZWZ6FGBJQNBPIQUKKEAO6GDJPYQ3KDLJOUN6VQ5W463LRYIYR line=github.com/owncloud/ocis/v2/services/nats/pkg/logging/nats.go:21 service=nats │
2025-03-27T20:34:50Z DBG Created system account: "$SYS" line=github.com/owncloud/ocis/v2/services/nats/pkg/logging/nats.go:45 service=nats │
2025-03-27T20:34:50Z INF Starting JetStream line=github.com/owncloud/ocis/v2/services/nats/pkg/logging/nats.go:21 service=nats │
2025-03-27T20:34:50Z DBG JetStream creating dynamic configuration - 35.28 GB memory, 1.00 TB disk line=github.com/owncloud/ocis/v2/services/nats/pkg/logging/nats.go:45 service=nats │
2025-03-27T20:34:50Z FTL Can't start JetStream: could not create storage directory - mkdir /var/lib/ocis/nats: permission denied line=github.com/owncloud/ocis/v2/services/nats/pkg/logging/nats.go:33 service=nats │
stream closed EOF for ocis/nats-75c6fdddfc-fgj8j (nats)

[dschmidt]

Hm - I could workaround it with chownInitContainer: true but that hadn't been necessary in this cluster before, so I'm a little confused on what changed here.

[wkloucek]

Can't start JetStream: could not create storage directory - mkdir /var/lib/ocis/nats: permission denied

I'm not sure what the oCIS chart should do about it.

chownInitContainer: true

is only there for CSI drivers that do not use the fsGroup from the securityContext, see also:
https://github.com/owncloud/ocis-
charts/blob/9614fb4cbc9f47e756dffcd28a217369fb12bb5f/charts/ocis/values.yaml#L3049-L3053

Furthermore I highly recommend you to not use the builtin NATS except for quick testing setups. If it is a testing setup only, you could just start from scratch and see if the issue persists.

[wkloucek]

If it is a testing setup only, you could just start from scratch and see if the issue persists.

Was it reproducible?

[dschmidt]

I will try to reproduce this or next week. The cluster wasn't ... operable ... for some time 😅