-
Notifications
You must be signed in to change notification settings - Fork 7
Description
Some DPUs can be very verbose. This combined with situations where many intermediate files are processed makes the log files grow quickly to tens of GB.
ODN is often deployed on VMs where large disk space is not expected to be necessary (certainly not for logs).
In this settings, it happens that pipelines fail because no space is left on device. Just making a DPU too talkative makes the whole system becomes vulnerable. This should never be allowed.
I suggest to make sure that log files can never grow larger than a (configurable) size. I would also suggest not to implement this as a scheduled log rotation mechanism, because that may be not sufficient (one single pipeline can already fill up the disk space before the log rotation comes in action)