Multiline messages

[atombender]

If you log a message containing linebreaks, ain will only prefix the first line with the "preamble". This might be correct, but it poses a problem with most syslog servers which don't deal with multiline messages over UDP/Unix socket, including Rsyslog. (It has a hacky setting called EscapeControlCharactersOnReceive which can be used to ignore line breaks, but it's not good enough. Rsyslog is such a mess.)

I think the better solution is to let ain optionally split the message into lines, and prefix each line with the preamble. Thoughts?

[phuesler]

I think I have used syslog clients that would split multiline strings into individual lines with syslog prefixes (was a Ruby syslog lib if I remember correctly). I see a problem with the order of lines, especially in the case of UDP, but also in a distributed setup. Imagine a stacktrace that shows up in random order in your log file.

I'll have to think about potential solutions. So fare I have:

• Let the client deal with it
• Split the string at newline (what about windows?) and create individual syslog messages
• Concatenate (with what character?) the newlines and turn it into one long line

[phuesler]

I quickly checked how papertrail does it and it looks like their recommended library generates multiple messages from multiple lines

[atombender]

I see your point about out-of-order messages, but is that actually a problem with UDP on localhost (which is pretty much 100% reliable, as far as I know)?

Also, as long as the splitting behaviour is an optional flag, it means you can still get the non-splitting behaviour if you do use UDP with a remote host. (But wouldn't you want to use TCP in that case, given that UDP is lossy? That's what the newest syslog protocol uses, which coincidentally also has "proper" message framing.) (Of course, on a local host there's also /dev/log.)

As an additional (contrary) data point, I've had to set up a couple of Java-based things recently (Elasticsearch and Graylog), and annoyingly, neither of the syslog adapters they use (Log4j and Logback) do the line splitting.

[phuesler]

I had some more time to think about this. As soon as several processes write to the same target, one will run into ordering issues eventually.

I'm leaning towards implementing this with the option to opt-in/opt-out. I'm thinking:

• Split on newline for the current platform (osx, windows, unix)
• Make it configurable
• Verify and document the behavior for both cases

[phuesler]

Would this work for you?