k8s

Istio

Prometheus scraping with mTLS

Prometheus scrapes endpoints, so IP addresses, Istio deals with services. This means that if you have mTLS enabled for your workload, the proxy will intercept prometheus requests and reject them due to the missing client certificate.

At this point there is 2 things you can do:

1. provide prometheus with the client certificate from the sidecar, instructions can be found here.
2. disable mTLS for the metrics port.

option 1 only works if the application pod exposes the metrics over plaintext. If the metrics are TLS encrypted you need option 2 since we can't make prometheus encrypt twice.

Example disabling mTLS for port 9000 on a keycloak pod:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: keycloak-metrics
spec:
  mtls:
    # inherit mTLS settings if set
    mode: UNSET
  portLevelMtls:
    # disable mTLS on port 9000
    '9000':
      mode: DISABLE
  selector:
    matchLabels:
      app.kubernetes.io/name: keycloak

Statefullset resize volume

Statefullsets don't update the existing PVC if you change the template.

https://github.com/kubernetes/enhancements/pull/4651

To actually resize the PVC (and as an effect the PV):

• Change the size in the statefullset template.
  This won't change anything but will ensure the state is consistant later.
• Scale down the operator owning the statefullset (prometheus-operator, postgres-operator, ...).
  This is so the statefullset won't be immediately recreated.
• Delete the statefullset with propagation orphan.
  This will keep the Pod running while we do the next step
• Edit the PVC to match the statefullset template size.
  This will trigger the resize, if the StorageClass supports it.
• Scale up the operator owning the statefullset.
  This will recreate the statefullset, which will take ownership of the orphaned Pod and PVC.