High CPU usage with buffer path that does not have wildcard

[wadeharrell]

Ran into an issue after a kube upgrade last week where our fluentd was using high CPU, we were able to resolve it today by changing our buffer path from
/var/log/fluentd-buffers/kubernetes.containers.buffer
to
/var/log/fluentd-buffers/kubernetes.containers.*.buffer
Apparently the wildcard helps with threading.

After we resolved it we noticed that stackdriver's fluentd-gcp-v3.1.1 in the kube-system namespace is also using a lot of CPU, double what it was before the kube upgrade.

https://github.com/Stackdriver/kubernetes-configs/blob/master/logging-agent.yaml#L600
is
/var/log/k8s-fluentd-buffers/kubernetes.system.buffer
perhaps
/var/log/k8s-fluentd-buffers/kubernetes.system.*.buffer
could help. Different plugin, so maybe the threading wildcard would not be the same, but in our case the CPU usage impact was dramatic.

[qingling128]

Interesting. Thanks for the heads up! That's very helpful information. We'll take a look.