adding support to forward containers output to splunk via hec connector

[dbloms]

We do currently use docker on Debian 11 and let the containers log on stdout through the
Splunk logging driver (https://docs.docker.com/config/containers/logging/splunk/) via HEC (https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector) into Splunk.

In term of docker-compose this gives us the possibility to configure Splunk-logging on a per deployment basis, which is very comfortable as well as independent of any central configuration.

We would like to move to Podman on RHEL8 servers. Everything works fine so far, but we didn't find a way to log stdout of the containers via the HEC interface into Splunk, as the currently available podman version 4 does not provide such a splunk logging driver.

Is there a solution on the part of Podman to log the output of the containers into Splunk.

If not, is it possible to commission a corresponding development against payment?

I opened a feature request in the podman repo and I was told to open the request in this repo.

[chillout2k]

@dbloms: we do also use the the splunk logging driver in our docker deployments and we would be very pleased if podman would support it too. From my point of view this logging driver is an important feature to make podman a real drop-in-replacement for docker in (splunk-specific) production environments.

Thanks in advance :)

[andreasschulze]

this would be valueable also for me

[rhatdan]

This seams reasonable to me. Do we do this in conmon or conmon-rs is the question?
@haircommander @vrothberg @mheon @wdyt?

[haircommander]

I think adding in conmon-rs would be easier (assuming podman integration happens in a timely manner).

[chillout2k]

@haircommander, @rhatdan: ping ;-)

Is it possible to push this feature request? @dbloms offered to pay for it if possible 👍

This might be interesting too: moby/moby#16207

[rhatdan]

You should open the issue in conmon-rs

[mehdik2023]

As mentioned in conmon-rs issue, am trying to raise this feature request in the list.

[Petaris]

This is also an issue for us. We currently use this feature in Docker and would not like to loose it.

[netFLOw95]

Any news on that?

[portante]

It seems as though the base conmon process is the exact wrong place to add more and varied log forwarders. Instead, let's consider conmon reading and saving log data in an efficient raw form, as proposed in #265, and then allowing readers of that raw format to run separately to handle the appropriate forwarding.

We would want conmon to be good at capturing stdout and stderr well and efficiently, and leaving interpretation and forwarding off of the log capture path (see #262 for what happens when you spend time interpreting the byte stream).