Native JSON serialization support for log entries

[jonsequitur]

A write-only built-in JSON serializer that can reduce the memory pressure from writing out strings, and provide some capabilities specific to logging (swallow and/or write out exception details rather than fail serialization) and Its.Log (control over recursion depth, control over iteration, follow Its.Log's Formatter settings).

[nathanboktae]

So you want LogEntry to be serializable and Its.Log to use the built-in JSON serialize in .NET to serialize them?

[jonsequitur]

LogEntry can already be serialized using a JSON serializer. There's an example of this in the unit tests, using JSON.NET. There are a few issues with general-purpose JSON serializers for this scenario, though:

• They don't understand Its.Log's formatting configurations, e.g. if you want to omit certain fields from your logs, limit recursion depth, or limit now many items in a list will be expanded
• They fail on a serialization error, whereas for diagnostics, we should make a best attempt to serialize as much of the object as possible, as well as document any failures. For example a serialization exception on a certain node could replace that node with the exception itself. The goal here is diagnostics, not fidelity on deserialization.

[jzabroski]

"limit how many items in a list will be expanded"...

Are you referring to how, in Visual Studio's Immediate Window, if I do collection.ToList() on a collection containing more than 100 items, it displays the first 100 items and then warns me there is more data?

"formatting configurations" ...

Calling this formatting isn't correct. You shouldn't conflate serialization with formatting to begin with.

What would be nice is if we could use Mono.Cecil and Its.Log configuration side by side, and generate an "Expected Log Growth" report based on sample waffle data.

As for serialization errors -> again, this shouldn't be conflated with formatting. If serialization fails on a sub-tree of the object graph, I feel like Its.Log should be able to provide its consumers some symbolic execution environment to auto-detect these problems.

Basically:

1. Load Mono.Cecil
2. Load Its.Log assembly and all assemblies in a directory referencing Its.Log
3. Crawl DLLs searching for objects that Its.Log wants to serialize
4. Write "Dig Class" function that digs out the type of the object being serialized
5. Automatically generate sample object trees for the dug-out class.

We can probably also whitelist certain safe classes - like string, known System.Exception derived classes in most of .NET Core, etc.

[jonsequitur]

Let me clarify the goal a little first. I'd agree that this entire category is "formatting" and not serialization. We want to format log output, and a JSON serializer is one tool we sometimes use for that. The goal of Its.Log's built-in formatter is to provide human-readable logs that are as informative as possible even in partial failure scenarios, and not to allow exceptions or excessive iteration or recursion, hence Formatter.RecursionLimit and Formatter.ListExpansionLimit. Its.Log's formatter also has the capability to specify which fields are interesting for any given type. Agreed that this is not serialization from Its.Log's perspective. The point here is that it would be useful to have JSON output follow these same rules.

Now, sending Its.Log's output to a document database that allows searching on JSON makes JSON "formatting" very useful, but all of the JSON serializers I know of have deserializability as a primary requirement, e.g. if they can't successfully serialize, then they throw. This is not as informative as possible, since a failure in a subtree fails the whole log entry.

And in practice, System.Exception-derived classes are actually the most common source of exceptions when using a JSON serializer, because they often have cyclical references (e.g. many types in System.Reflection) or types that don't show themselves during unit testing.

The static analysis approach you're suggesting sounds like it would miss some cases:

• Inputs to Its.Log can be dynamic (including collections containing diverse types)
• Inputs can be of types that are not known at build time

But it's interesting to think of how to detect problems during testing, especially code changes that cause log explosions. I got bitten by one of these very recently.

The strategy has generally been that since runtime behaviors can be unpredictable, we protect the application from exceptions that are thrown anywhere in logging code, and we try to provide as much information about those exceptions when they happen.

[jzabroski]

This may be our gateway into doing your idea: https://github.com/kevin-montrose/Jil

I still dislike how you are commingling formatting with serialization, but this may be an essential problem with OO paradigm.