Add schemer version to schema

[bminer]

Adding the version number of schemer itself to the encoded schemas will allow even more compatibility as schemer evolves

[BenjaminPritchard]

I always thought adding a small frame to the schemer data make sense. such as:
[uniqueID][schemerVersion][payload length][schemer payload][checksum]

that would only add several bytes, and would allow us to verify that some bytes actually represent schemer data

it is just one idea that maybe makes sense, as long as we are adding the version number anyway?

[bminer]

I think we should discuss further. IMO payload length and checksum are not strictly needed.

The notion of an unique ID is interesting, but I think it's a separate issue.

The JSON document also needs the schemer version number. Should follow semver (i.e. major.minor.revision)

[BenjaminPritchard]

I was just thinking some way to verify that an arbitrary slice of bytes was actually schemer encoded data
My thinking was that common programmer off by one errors or similar could crash Schemer lib if incomplete data was passed to Schemer . Maybe it doesn't matter. I was thinking some way for schemer decode to verify if passed in data is OK.

A simple verification step [to make sure library users are passing correct byte slices by having simple checksum for schemer data] at our level can help prevent/identify buys for users of this library.

RIght now, according to my understanding, the library will crash and/or return garbage if passed in schemer schema / encoded data passed has issues.

[BenjaminPritchard]

So we will shelf the idea of adding a data frame to the encoded schemer data. (Reason for this is to keep the data as small as possible.) However, our discussions have decided that since schemas themselves are only transmitted over the wire occasionally, it is OK to add extra data to them.

For a JSON encoded scheme, we will need to modify the function DecodeSchemaJSON() and then MarshalJSON() in each datatype. The only change should be to add an additional field called: "schemerversion".

The only problem with this approach is that we support nested structures, so writing the version number in each MarshalJSON() would lead to redundancy. I think the way around this is to use a mutex.

The other type of schema we support is the binary format. For this, I think an easy way to implement this is to add a magic (64-bit) number to the beginning of the byte slice, which is followed by the Schema version number.

Like the JSON schemas, a similar use of a mutex can be used to ensure nested data types do not also add these bytes.

[bminer]

I'd just call the additional field version. The version number will only appear at the root level, so I don't think we need a mutex.

For the binary format, I'm good with adding some framing, magic number, checksum, etc.... although I'm slightly worried it would add too much additional complexity to the encoding/decoding code. Similarly, the magic number, framing, and checksum will only appear at the root level. It should not be nested.

[BenjaminPritchard]

Ya, I might just be thinking wrong. But for example, we could have a schema for an integer [by itself]. So if we did, it seems like we would have to have the version number.

But if we had a struct with an integer in it, how would the integer know not to include the version number?

It seems like we would need a mutex [i.e. global variable] to prevent redundancy.

In any case, I think I have enough to start work on this.

It doesn't seem like too big of a task, and we can double check the implementation to make sure it doesn't add to much complexity.

And if I am not thinking right about the struct fields, I will just leave that out.

[bminer]

The version number would not be stored in the Schema instance itself. In other words, there is no struct field for the schema version. It would only be produced when encoded. I think that the encoding function would have to know when it's encoding the root schema, so that it only produces the version number at the root level.

If, for example, you had a struct with an integer field, when the struct is encoded, the version number would only be added to the struct, not to the individual fields. To be honest, I'm not exactly sure how this would work. Perhaps, every type would encode the version, but then for nested types (i.e. object, array), I believe that the MarshalJSON function can unmarshal the field / element and remove its version field before re-marshalling.

For MarshalSchemer, if the first few bytes are reserved for the version number, these could be dropped before appending to the resulting byte slice.

Thoughts?

[BenjaminPritchard]

I think we are on the same page re: the JSON. But actually I think it will be clearer once I start working with the code and get schemer back into the front of my brain!

Re: your point about MarshalSchemer , yes I see what you mean I think.

But for sure we both are in agreement that somehow we want to make sure the schema version number is only encoded once into a JSON/binary schema.