Autoscaling / Downsaling results in Bad Gateway 502 Errors

[tiran133]

Hi,

I'm running this latest chart version 0.7.0 with ocis 7.1.1 and following values - see below.
External NATS and user management keyloackl and ldap.

I deployed the NATS from this example https://github.com/owncloud/ocis-charts/tree/main/deployments/ocis-nats

Every time the autoscaler scales down/or I manually kill all storageusers pods and uploading a file at the same time, it results in a Bad Gateway 502 Error

Edit:
The same happens with the proxy deployment/pods. owncloud/ocis#11170

[Uppy] [12:58:40] tus: unexpected response while uploading chunk, originated from request (method: PATCH, url: https://cloud.kube.domain.io/data/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOlsicmV2YSJdLCJleHAiOjE3NDMyMDg4NjIsImlhdCI6MTc0MzEyMjQ2MiwidGFyZ2V0IjoiaHR0cDovL3N0b3JhZ2V1c2Vyczo5MTU4L2RhdGEvdHVzLzgyMjU4Y2U3LTNjNGEtNGQxMS1iNzcxLWRlZWJjMmM4M2NhNiJ9.GXxihlhBuRTT4T6iD6fsVpj5i7l5P0f0XLLW9nef9bQ, response code: 500, response text: , request id: 72ccffe2-a8fc-42fe-870d-f0e6b55fe723)

The frontend service logs following error

2025-03-28T02:58:34Z DBG sending request to internal data server line=github.com/cs3org/reva/v2@v2.27.7/internal/http/services/datagateway/datagateway.go:187 pkg=rhttp request-id=72ccffe2-a8fc-42fe-870d-f0e6b55fe723 service=frontend target=http://storageusers:9158/data/tus/82258ce7-3c4a-4d11-b771-deebc2c83ca6 traceid=c7dd50c8ec14f38d84cf13ee15e466c3
2025-03-28T02:58:40Z ERR error doing PATCH request to data service error="Patch \"http://storageusers:9158/data/tus/82258ce7-3c4a-4d11-b771-deebc2c83ca6\": read tcp 10.42.7.158:58142->10.43.32.181:9158: read: connection reset by peer" line=github.com/cs3org/reva/v2@v2.27.7/internal/http/services/datagateway/datagateway.go:200 pkg=rhttp request-id=72ccffe2-a8fc-42fe-870d-f0e6b55fe723 service=frontend traceid=c7dd50c8ec14f38d84cf13ee15e466c3

Proxy

2025-03-28T02:58:40Z INF access-log bytes=0 duration=5909.739159 line=github.com/owncloud/ocis/v2/services/proxy/pkg/middleware/accesslog.go:34 method=PATCH path=/data/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOlsicmV2YSJdLCJleHAiOjE3NDMyMDg4NjIsImlhdCI6MTc0MzEyMjQ2MiwidGFyZ2V0IjoiaHR0cDovL3N0b3JhZ2V1c2Vyczo5MTU4L2RhdGEvdHVzLzgyMjU4Y2U3LTNjNGEtNGQxMS1iNzcxLWRlZWJjMmM4M2NhNiJ9.GXxihlhBuRTT4T6iD6fsVpj5i7l5P0f0XLLW9nef9bQ proto=HTTP/1.1 remote-addr=115.70.78.44 request-id=72ccffe2-a8fc-42fe-870d-f0e6b55fe723 service=proxy status=500 traceid=2a7294961371d75f889a9f4f07833c4a

It looks like the pod still receives traffic even in terminating state.
I'm using traefik as my ingress, is that a Problem should I use nginx?

Any ideas what can be done about it?

values.yaml

  values:
    tracing:
      enabled: false
      type: jaeger
      endpoint: jaeger.jaeger.svc.cluster.local:6831
      collector: jaeger.jaeger.svc.cluster.local:14268/api/traces
    resources:
       requests:
        cpu: 100m
        memory: 128Mi
    replicas: 1
    autoscaling:
      enabled: true
      minReplicas: 2
      maxReplicas: 10
    logging:
      level: "debug"
      pretty: true
      color: true
    externalDomain: cloud.kube.domain.io
    ingress:
      enabled: true
      ingressClassName: traefik
      annotations:
        traefik.ingress.kubernetes.io/router.tls: "true"
        traefik.ingress.kubernetes.io/router.entrypoints: websecure
      tls:
        - secretName: tls-ocis-ingress
          hosts:
            - cloud.kube.domain.io
    insecure:
      oidcIdpInsecure: true
      ocisHttpApiInsecure: true
    secretRefs:
      ldapSecretRef: ldap-bind-secrets
      s3CredentialsSecretRef: ocis-s3secret
    messagingSystem:
      external:
        enabled: true
        cluster: "ocis-cluster"
        endpoint: nats.ocis-nats.svc.cluster.local:4222
        tls:
          insecure: true
          enabled: false

    registry:
      type: nats-js-kv
      nodes:
        - nats.ocis-nats.svc.cluster.local:4222

    store:
      type: nats-js-kv
      nodes:
        - nats.ocis-nats.svc.cluster.local:4222

    cache:
      type: nats-js-kv
      nodes:
        - nats.ocis-nats.svc.cluster.local:4222

    features:
      externalUserManagement:
        enabled: true
        adminUUID: "ddc2004c-0977-11eb-9d3f-a793888cd0f8"
        autoprovisionAccounts:
          enabled: true
          claimEmail: email
          claimDisplayname: name
          claimGroups: groups
          claimUserName: preferred_username
        oidc:
          issuerURI: https://keycloak.kube.domain.io/realms/oCIS
          userIDClaim: sub
          userIDClaimAttributeMapping: userid
          roleAssignment:
            enabled: true
            claim: roles
        ldap:
          writeable: true
          uri: ldap://openldap.openldap.svc.cluster.local:389
          insecure: true
          bindDN: cn=admin,dc=domain,dc=io
          user:
            schema:
              id: ownCloudUUID
            baseDN: ou=users,dc=domain,dc=io
          group:
            schema:
              id: ownCloudUUID
            baseDN: ou=groups,dc=domain,dc=io
    services:
      search:
        persistence:
          chownInitContainer: true
          enabled: true
      storagesystem:
        persistence:
          enabled: true
          size: 5Gi
          accessModes:
            - ReadWriteMany
          storageClassName: nfs-client
      storageusers:
        persistence:
          enabled: true
          size: 20Gi
          accessModes:
            - ReadWriteMany
          storageClassName: nfs-client
        storageBackend:
          driver: s3ng
          driverConfig:
            s3ng:
              endpoint: https://s3
              bucket: ocis-kube
              putObject:
                disableMultipart: true
      thumbnails:
        resources:
          limits:
            memory: 1Gi
          requests:
            cpu: 100m
            memory: 1Gi
        persistence:
          enabled: true
          size: 5Gi
          accessModes:
           - ReadWriteMany
          storageClassName: nfs-client
      web:
        persistence:
          enabled: true
          size: 1Gi
          accessModes:
            - ReadWriteMany
          storageClassName: nfs-client
        config:
          oidc:
            webClientID: web

[wkloucek]

I think we need to differentiate those two errors:

Bad Gateway 502 Error

This looks indeed like requests are forwarded to something that is no longer accepting requests.

Please note that oCIS is not using Kubernetes Services and thus does not benefit from ReadinessProbes and so on.

See also eg. owncloud/ocis#8589 (comment) where I raised this to the oCIS DEV team. Nothing the oCIS chart can fix this with the current state of oCIS.

tus: unexpected response while uploading chunk, originated from request (method: PATCH, url: xxx, response code: 500

This is an effect of the shutdown behaviour. oCIS doesn't do a graceful shutdown as you and me wished it would do it. In a perfect way a shutdown would look like this:

• receive shutdown signal
• set readiness status to NotReady, so that the pod doesn't get any more requests forwarded
• finish all pending requests
• shutdown

But currently this is not the case! Firstly the NotReady status would needed to be respected by the oCIS service registry or we would need to switch to Kubernetes Service (after oCIS supports this as a configuration)

TUS itself should be able to recover from such errors. I still agree that those error logs on the server side are not nice.

[tiran133]

Thanks for your answer.

Yes, that very unfortunate. So in the current state of oCIS it won't autoscale on Kubernetes.

I just wonder how the best approach would be to leverage Kubernetes without having too much impact on the user/frontend side. Currently, my cluster is set up to auto update and reboot nodes. I might have to turn that off or move it to a specific time.

Are you aware of any other service that does not downscale, properly? At the current stage, killing the proxy and or storageusers service results in a failed request.

I might just add a fixed replica set and turn off autoscale for these two services.

Any other ideas on how to run it stable on Kubernetes?

[wkloucek]

So in the current state of oCIS it won't autoscale on Kubernetes.

You can do autoscaling. But scaling down will cause the issues you reported.

my cluster is set up to auto update and reboot nodes.

Unfortunately this will have similar impact IF pods need to be descheduled. Control plane / in-place kubelet updates don't necessarily cause it because Pods should keep running.

killing the proxy and or storageusers service results in a failed request.

I'd like to differentiate between "killing" as eg. in OOM kill and ordered shutdown (eg. Node drain). I think killing will never be handled that graceful as we'd wish. But ordered shutdown can be made very graceful.

What I wrote in owncloud/ocis#8589 (comment) basically affects ALL oCIS services 😢

Any other ideas on how to run it stable on Kubernetes?

To have it more stable, you'd need to ensure that Pods are not going away. But this is unfortunately against the nature of Kubernetes.

You could go with static count of replicas or forbid downscaling and only allow upscaling. Plus you could enforce zero voluntary evictions via PDB (https://kubernetes.io/docs/tasks/run-application/configure-pdb/#unhealthy-pod-eviction-policy), but that's ugly, too:

If you set maxUnavailable to 0% or 0, or you set minAvailable to 100% or the number of replicas, you are requiring zero voluntary evictions. When you set zero voluntary evictions for a workload object such as ReplicaSet, then you cannot successfully drain a Node running one of those Pods. If you try to drain a Node where an unevictable Pod is running, the drain never completes. This is permitted as per the semantics of PodDisruptionBudget.

[tiran133]

Thank you for the detailed answer.

That's a shame really, I was always the impression that it was designed to work "well" on Kubernetes, and I was surprised to see that it behaves that way. I'm sure there are reasons for it, but it would be nice to see this issues addressed to make it work properly.

Don't get me wrong it works just not as well as it could be, oCIS it is still an awesome software.

Also, the GitHub Issue where you linked to your comment is 5 months without activity. Do you know if this is being addressed or actively worked on?