High CPU overhead caused by console output in running containers

[ananos]

When running workloads under Urunc, enabling or producing console output (stdout/stderr) results in significant and sustained CPU usage on the host, even when the application itself is otherwise idle or lightly loaded.

This behavior suggests that console output handling (e.g., logging, polling, forwarding, or buffering) introduces non-trivial overhead that scales with runtime duration and/or output frequency, negatively impacting performance and efficiency.

Steps to Reproduce

• Launch a container or unikernel using Urunc with console output enabled.
• Run an application that:
  • Periodically prints to stdout/stderr (e.g., logging every few milliseconds), or
  • Produces moderate continuous output.
• Observe host CPU usage using top, htop, or perf.

Expected Behavior

• Console output handling should incur minimal CPU overhead.
• CPU usage should primarily reflect the application’s actual workload.
• Idle or low-output applications should not cause noticeable host CPU utilization.

Actual Behavior

• Host CPU usage increases when console output is enabled.
• CPU overhead persists even for lightweight applications.
• Disabling console output drastically reduces CPU consumption.

[codesmith25103]

@ananos
I was trying to reproduced this issue.

To test this, I ran an Alpine container using urunc with a simple loop that prints to stdout every 1ms.

According to htop, even for this lightweight workload, the containerd-shim-urunc-v2 process consistently consumes about 2.7% - 3.0% CPU. The client process (ctr) consumes an additional ~2.0%.

[codesmith25103]

I was thinking to solve this problem using BufferReader which will reduces the number of context switches between the kernel and userspace by reading larger chunks at once.
I was also read about io.copy which can automatically handles buffering and takes advantage of efficient system calls (but I do not have much idea about io.copy but we can explore this)

[codesmith25103]

I believe high CPU usage in cmd/urunc/log_forward.go is caused by bufio.Scanner scanning every byte and processEntry attempting json.Unmarshal on every line.