Protocol Buffers, evolved

This project enables generation of .proto files using configuration languages like Jsonnet and Nickel. .proto files serve as critical schema specifications for data interchange, powering an enormous ecosystem of tools including grpc-gateway and protoc-gen-validate.

However, raw .proto files have no functionality for code reuse - they're purely data definitions. Jsonnet and Nickel are data templating languages specifically designed to remove boilerplate when the output is pure data, making them perfect for generating protobufs with reusable components.

Installation

$ go install github.com/tomlinford/sroto/cmd/srotoc@latest

You'll also need the protobuf toolchain installed. The srotoc binary calls protoc directly, so protoc must be in your $PATH.

For Jsonnet: The srotoc binary embeds go-jsonnet and the sroto.libsonnet library, so no separate jsonnet installation needed.

For Nickel: Install the Nickel CLI:

# macOS with Homebrew
brew install nickel

# Or build from source
cargo install nickel-lang-cli

Project status

Feature-complete - all .proto file features are supported. Safe for production use, though checking in generated .proto files is recommended early on to verify changes.

Quick example

Here's the same protobuf schema defined in both Jsonnet and Nickel:

Jsonnet version

// filename: example.jsonnet

local sroto = import "sroto.libsonnet";

sroto.File("example.proto", "example", {
    Priority: sroto.Enum({
        // note the lack of the 0 value here, it'll be auto-generated
        LOW: 1,
        HIGH: 3,
    }) {
        // In jsonnet you can pass "keyword arguments" via object composition
        // The sroto.Enum call returns an object which is merged with this
        // object. This enables extending objects without exhaustive redefinition.
        reserved: [2, [4, "max"], "MEDIUM"],
    },
    EchoRequest: sroto.Message({
        message: sroto.StringField(1),
        importance: sroto.Oneof({
            is_important: sroto.BoolField(2),
            priority: sroto.Field("Priority", 3),
        }),
    }),
    EchoResponse: sroto.Message({
        message: sroto.StringField(1),
    }) {
        // All sroto types have a `help` attribute for comments
        help: |||
            EchoResponse echoes back the initial message in the EchoRequest.

            This is used by EchoService.
        |||
    },
    EchoService: sroto.Service({
        // UnaryMethod is just Method with false for (client|server)_streaming
        Echo: sroto.UnaryMethod("EchoRequest", "EchoResponse"),
        StreamEcho: sroto.Method("EchoRequest", "EchoResponse", true, true)
    }),
    // Enums can also be defined with arrays for explicit ordering
    Quality: sroto.Enum([
        sroto.EnumValue(2) {name: "QUALITY_HIGH"},
        sroto.EnumValue(1) {name: "QUALITY_LOW"},
    ]),
})

Nickel version

# filename: example.ncl

let sroto = import "sroto.ncl" in

sroto.File "example.proto" "example" {
  Priority = sroto.Enum {
    # Note: The 0 value will be auto-generated as PRIORITY_UNSPECIFIED
    LOW = 1,
    HIGH = 3,
  } & {
    # Jsonnet-style reserved shorthand: numbers, ranges, and names
    reserved = [2, [4, "max"], "MEDIUM"],
  },

  EchoRequest = sroto.Message {
    message = sroto.StringField 1,
    importance = sroto.Oneof {
      is_important = sroto.BoolField 2,
      priority = sroto.Field "Priority" 3,
    },
  },

  EchoResponse = sroto.Message {
    message = sroto.StringField 1,
  } & {
    # Using record merging to add help text
    help = m%"
      EchoResponse echoes back the initial message in the EchoRequest.

      This is used by EchoService.
    "%,
  },

  EchoService = sroto.Service {
    # UnaryMethod is just Method with false for both streaming flags
    Echo = sroto.UnaryMethod "EchoRequest" "EchoResponse",
    StreamEcho = sroto.Method "EchoRequest" "EchoResponse" true true,
  },

  # Enums can also be defined with arrays for explicit ordering
  Quality = sroto.Enum [
    sroto.EnumValue 2 & { name = "QUALITY_HIGH" },
    sroto.EnumValue 1 & { name = "QUALITY_LOW" },
  ],
} []

Both generate the same .proto file:

# With Jsonnet
srotoc --proto_out=. example.jsonnet

# With Nickel
srotoc --proto_out=. example.ncl

Output:

// filename: example.proto

// Generated by srotoc. DO NOT EDIT!

syntax = "proto3";

package example;

enum Priority {
    PRIORITY_UNSPECIFIED = 0;
    LOW = 1;
    HIGH = 3;

    reserved 2;
    reserved 4 to max;
    reserved "MEDIUM";
}

enum Quality {
    QUALITY_UNSPECIFIED = 0;
    QUALITY_HIGH = 2;
    QUALITY_LOW = 1;
}

message EchoRequest {
    oneof importance {
        bool is_important = 2;
        Priority priority = 3;
    }
    string message = 1;
}

// EchoResponse echoes back the initial message in the EchoRequest.
//
// This is used by EchoService.
message EchoResponse {
    string message = 1;
}

service EchoService {
    rpc Echo(EchoRequest) returns (EchoResponse);
    rpc StreamEcho(stream EchoRequest) returns (stream EchoResponse);
}

You can also generate code directly:

srotoc --proto_out=. --python_out=. example.jsonnet
# Generates both example.proto and example_pb2.py

Jsonnet vs Nickel

Similarities

• Both generate the same IR format and produce identical .proto output
• Nearly identical APIs with the same constructors and patterns
• Both support all protobuf features

Differences

Feature	Jsonnet	Nickel
Syntax	function(arg)	function arg
String interpolation	"%(name)s" % {name: x}	"%{name}"
Multiline strings	|||...|||	m%"..."%
Record merging	+: for append, :: for hidden	& for merge
Imports	import "file.libsonnet"	import "file.ncl"
Type system	Dynamic only	Optional static + contracts
Standard library	std.*	std.*, %record/*, %array/*
Dependencies	Embedded in srotoc	Requires nickel CLI

When to use Jsonnet

1. No dependencies: Already embedded in srotoc
2. Established ecosystem: More mature tooling
3. Familiar: If you're already using Jsonnet elsewhere
4. Simple needs: Just want to generate protos without types

When to use Nickel

1. Better tooling: Built-in LSP support for IDE integration
2. Type safety: Optional contracts for validation
3. Cleaner syntax: More modern functional style
4. Better errors: More helpful error messages
5. Active development: Nickel is actively maintained by Tweag

More examples

• JSONNET.md: Complete Jsonnet examples including imports, options, custom options, and multi-file generation
• NICKEL.md: Complete Nickel examples and API reference

Project goals

Sroto has the following design goals:

1. Provide composition: Enable code reuse when writing protobuf schemas
2. Seamless integration: Fit into existing protobuf workflows
3. Better options experience: Dramatically improve working with custom options

These goals are achieved by:

1. Leveraging data templating languages (Jsonnet/Nickel) with shared libraries that expose easy-to-use APIs
2. Enabling seamless imports between generated .proto files and hand-written .proto files
3. Providing srotoc, a drop-in replacement for protoc that generates .proto files first, then invokes protoc

Schema-first vs Code-first

This approach enables the benefits of code-first schemas without the drawbacks:

• schema-first (e.g. gRPC): schema → all application code
• code-first (e.g. django-rest-framework): primary service code → schema → other code
• sroto: data templating language → schema → all application code

With sroto, protobuf files better serve as the source of truth. All consumers are first-class citizens since the schema is language-agnostic.

Architecture

Sroto supports multiple frontends through a layered architecture:

1. Frontend (Jsonnet/Nickel) → Sroto IR: Expose user-friendly API, normalize inputs, validate names
2. Sroto IR → Proto AST: Handle imports, generate *_UNSPECIFIED enum values, merge options
3. Proto AST → Proto file: Generate syntactically correct protobuf text

The IR (Intermediate Representation) is defined by Go structs in the sroto_ir package. New frontends only need to target the IR format - for example, a CUE frontend would only need to implement step 1.

Compatibility

Mix and match:

• Combine .jsonnet, .ncl, and .proto files in the same project
• Import from generated .proto files into hand-written .proto files
• Gradually migrate existing .proto files to sroto
• Use srotoc as a drop-in replacement for protoc

Contributing

Contributions welcome! Please open an issue to discuss significant changes.

License

MIT License