Add monitoring to redis and other k8s services

[auto-mausx]

Currently there is no monitoring on Redis, which has caused issues in the past. Goal is to add a grafana agent to all of the k8s clusters and to add new dasboards and alerts for Redis and other services within k8s.

[volovyks]

What can we do on Mainnet? Redis is "baked" inside of the node image.

[auto-mausx]

I think grafana can emit metrics at the container level, I might be able to expose them on a different port. But we should probably implement some kind of cleanup logic on the application side. For example removing old entries after x amount of months/weeks would suffice.

[jakmeier]

But we should probably implement some kind of cleanup logic on the application side. For example removing old entries after x amount of months/weeks would suffice.

We use redis in persistent storage mode, not just as an in-memory cache as it was traditionally designed. I don't think cleaning up old entries makes sense in persistent mode. If data grows indefinitely, I would suggest we manually look at what keys these are and check if we need to add explicit clean up logic for those entries. If there is a way to flag old entries, that could be helpful.

[volovyks]

Here is data for Redis Disk usage over the last 6 months. It does grow, almost without drops.

If we look at Total pre-signatures and Total triples in the same time-frame, we will see that the number of stored triples and pre-signatures goes up and down a lot.

It means that our storage is misconfigured, Redis can work in "append-only" mode. Probably we are using some form of it.

What is weird is that I do not see the correlation between the number of stored triples and storage growth. Especially in the last 2 months.

[auto-mausx]

Where exactly are you looking at that chart? I have a feeling that's not specific to the PVC that redis uses because it is not that large of a disk.

[volovyks]

Chain Signatures Dev -> Workloads -> redis-master.
There is also redis-slave and redis-client-deployment, but they do not seam relevant.
Where should I look for the correct data?

[auto-mausx]

So redis slave nodes are just replicas that are Readonly to speed up query time, and redis-client-deployment is a pod deployed with redis-cli so you can interact with redis directly. I am working on setting up grafana alloy for the cluster, this will help us out in determining what pod resources are being used directly so I can put them into grafana. The problem with google metrics, is that it's looking at total node (K8s node in this case) Disk space, not individual pod resources disk space. Once I finish this I can create dashboards in the SRE folder so we can see exact workload metrics.