Serialise to JSON

[torgeirsh]

Is JSON encoding for serialisation supported? I've found some JSON related code in the library, but it seems mostly related to schemas. If I understand correctly, the JSON encoding is similar for serialisation and field default values, so maybe there's a way to reuse that? There doesn't seem to be a way to get a DefaultValue via ToAvro however.

[AlexeyRaga]

This library does not provide encoding to JSON. Schemas - yes, because Avro schema is defined in JSON as per specification.
You probably don't even need this library support for JSON encoding, right? You can write/derive FromJSON/ToJSON instances for your types?

[torgeirsh]

Avro defines a specific JSON encoding. While it's possible to write matching JSON instances by hand, it's error-prone, and you have to keep them in sync with the Avro schema, much like writing FromAvro/ToAvro instances by hand. I understand that generating FromJSON/ToJSON based on the Avro schema is not currently supported, but since the library already generates FromAvro/ToAvro based on the Avro schema, I think it would make sense. If I understand correctly, they are both "official" Avro encodings.

[AlexeyRaga]

Yes, you are right, this library does not currently provide support for JSON encoding.
It could be added (PRs are always welcome).

The biggest pain point here is probably de-conflicting machinery. De-conflicting per se is done when constructing Value type, but it still needs to be taken into account when converting Value to the final types, similar to this and according to Avro specification rules.

[torgeirsh]

I'm not familiar enough with the details of Avro to know what de-conflicting is. Is it a viable approach to make deriveAvroWithOptions' generate FromJSON/ToJSON instances, or would that impose an Aeson dependency on users? Is it better to make a separate deriving function that generates the JSON instances?

[AlexeyRaga]

Dependency is not a problem, we already depend on Aeson.
However, I deliberately stayed away from having FromJSON/ToJSON for any of the types that this library generates. The reason is that users may want their own FromJSON/ToJSON for their own reasons. And they should have this flexibility.

But even then, it would probably be impossible (or really hard) to generate FromJSON for an Avro type. The reason is, again, de-conflicting. If you look at the spec then you'll see that in Avro reader's and writers's schemas can be different. And the specification defines all possible rules for converting from writer's schema to reader's one when they don't match. This is done at run time.

Therefore your decoding function is not only Value -> Either Error a. Granted, you can require something like HasSchema a and get reader's schema, but you'd still need to see the writer's schema in this decoding function.

This library already does de-conflicting between reader's and writer's schema (here), but you'd then need a function like:

Value -> ReadSchema -> Either Error a

Which is not what FromJSON class expects. And then, recursively traversing Value you should also recursively traverse ReadSchema (since each element of an object will have its own schema).

I am not saying that decoding from JSON is not doable. I am saying that FromJSON machinery would probably not work here, or at least I don't see how it can de used.

Maybe one way of doing something like that would be to implement a function like:

jsonToAvro :: Aeson.Value -> ReadSchema -> Either Error Avro.Value

And when we have Avro.Value we can just use existing FromAvro to convert it to Haskell types like:

decodeFromJSON = jsonToAvro json schema >>= fromAvro

This looks doable and even quite trivial from the first glance...

Converting to JSON can be a bit easier and can indeed be generated because the schema is statically known for the write operation (no de-conflicting etc.).
So theoretically we could generate ToJSON, but:

a) having it without a matching FromJSON instance (which we cannot write) would be weird
b) it would deny users from writing their own ToJSON instances.

[torgeirsh]

Thank you for the insight, now I have a better understanding of what the challenges are. Even if the library doesn't provide FromJSON/ToJSON, functions like jsonToAvro/avroToJson would be helpful. Maybe a way forward is for the library to provide typeclasses AvroToJson and JsonToAvro, and then deriveAvroWithOptions' can generate instances for them? It sounds like that would overcome the incompatibility with FromJSON.

[AlexeyRaga]

Maybe we can have ToAvroJSON and FromAvroJSON typeclasses, yes.
Feel free to sketch a PR, it would be interesting to see how it turns out :)

[torgeirsh]

I started to investigate, but the context in which this issue showed up at work has changed, so for now I will leave it. Maybe the issue can remain open for future reference?