[NET_HANDLER] Considering replacing Protobuf with JSON

[benliao1]

Currently, Runtime communicates with Dawn and Shepherd over the network using the Google Protobuf protocol. It is a highly optimized protocol to deliver a given amount of information over the network in as few bytes as possible. This is useful for large data centers (as Google has) where network loading is a real issue. However, the amount of data sent between Dawn, Runtime, and Shepherd is not that large, in the grand scheme of things.

PiE started using Google Protobuf at least since 2017 in order to try to lighten the load on the network during competition. We now know that the real issue with our networking was that the software projects (especially Shepherd and Runtime) were simply too unstable at the time.

One of Runtime's last big 3rd-party dependencies is Google Protobuf. The process for using Google Protobuf in C and compiling the C header files from source is quite cumbersome. Learning how to write protobuf code, on top of learning all of Runtime, is quite difficult, especially since there isn't much documentation of the protobuf-c library.

This issue is to consider changing the way that Runtime, Dawn, and Shepherd communicate with each other. Instead of using Google Protobuf, we simply use JSON, which has native support in Python and Javascript. C does not support it natively but the tools used to marshal / unmarshal JSON messages in C are tiny in comparison to those needed to work with Protobuf.

This would remove the last huge third-party dependency from Runtime and make it nearly completely self-contained. However, it would require a lot of coordination between all 3 of PiE's software projects to pull off.

[sberkun]

As someone from Shepherd, I am in favor. Protobufs are overkill for Pie's use case, and caused Shepherd some pain last January when the Python protobuf library was broken.

My only concern is that this will cause the message definitions to become more informal. One advantage of protobuf is that (for example) both Dawn and Runtime members can look at input.proto and see exactly what an input message is supposed to contain. If we switch to JSON there's a temptation to pass the "schema" by word of mouth.

We could create a wiki page that contains "definitions" like this:

// message type: 1
UserInputs = {
    "inputs": [Input, ...]
}
Input = {
    "connected": bool,
    "buttons": double,
    "axes": [double, double, double],
    "source": "gamepad" | "keyboard"
}

but there's nothing that forces such a page to stay up-to-date or accurate.

[benliao1]

@sberkun Do you think we could still maintain a Github repo with all of the message definitions in there, and import it as a submodule into Dawn, Shepherd, and Runtime? That way there's no confusion for what the scheme is--the single source of truth is that repo. I suppose there's no way of ensuring that someone doesn't change the code without updating the repo but at that point they're almost breaking it on purpose, right?

[sberkun]

perhaps; I'm still worried that the submodule might go ignored, and new members start writing code without realizing that it exists.

To make it more enforcable, maybe the serialization/deserialization code for all 3 teams should also be in the submodule. For example, Shepherd's code would look something like:

class UserInputs:
    def __init__(self, msg):
        obj = json.loads(msg)
        self.inputs = [Input(ob) for ob in obj["inputs"]]

class Input:
    GAMEPAD = "gamepad"
    KEYBOARD = "keyboard"
    def __init__(self, obj):
        self.connected = obj["connected"]
        self.buttons = obj["buttons"]
        self.axes = obj["axes"]
        self.source = obj["source"]

Then whenever an update needs to be made, the schema and code for all 3 teams can be updated in a single PR.

[mdhgarcia]

Coming from industry - where protobuf is something of a de facto standard - I can see something of both sides: json is easier for humans to work with and infinitely flexible, while protobuf is easier for machines to consume and provides very tight configuration management via schema versioning. I'm surprised the python protobuf library broke at all!

Have you heard of CBOR? If mature C libraries are available for it, you may get close to the best of both worlds, including (yet another) a way of codifying messages that should look a bit more familiar.