From 5c7a5a027c55194b0b897c68d9681bf5c7ad1601 Mon Sep 17 00:00:00 2001 From: oleksandra-kovalenko Date: Tue, 30 Dec 2025 07:39:52 +0200 Subject: [PATCH 01/13] =?UTF-8?q?Added=20blog=20article=20"Protobuf=20?= =?UTF-8?q?=E2=80=94=20Serialization=20and=20Beyond.=20Part=201.=20What?= =?UTF-8?q?=E2=80=99s=20in=20It=20for=20Your=20Project"?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../index.md | 275 ++++++++++++++++++ 1 file changed, 275 insertions(+) create mode 100644 site/content/blog/protobuf-whats-in-it-for-your-project/index.md diff --git a/site/content/blog/protobuf-whats-in-it-for-your-project/index.md b/site/content/blog/protobuf-whats-in-it-for-your-project/index.md new file mode 100644 index 00000000..ff6d23a2 --- /dev/null +++ b/site/content/blog/protobuf-whats-in-it-for-your-project/index.md @@ -0,0 +1,275 @@ +--- +title: Protobuf — Serialization and Beyond. + Part 1. What’s in It for Your Project +description: >- + In this article we look at the key Google Protocol Buffers (aka Protobuf) features which make working with data in software development projects more convenient and effective. +author: Alexander Yevsyukov and Dmytro Dashenkov +publishdate: 2020-11-16 +type: blog/post +header_type: fixed-header +--- + +*This six-part series covers using Google Protocol Buffers (aka [Protobuf](https://developers.google.com/protocol-buffers)) to build software Domain-Driven Design way and to cut on the manual effort when migrating data and syncing interaction protocols in complex systems.* + +*In the first part, we look at the key Protobuf features which make working with data in software development projects more convenient and effective.* + +Like most developers, we at TeamDev first encountered Protobuf, when we needed to transfer the data between different platforms. + +The better we mastered this technology, the more ways besides simple serialization we found to apply it to our software development tasks. + +In this cycle, we provide a summary of our five-year experience of using Protobuf for quite a diverse range of software projects: both the ones we do for our clients, and the products we develop for developers: Spine Event Engine, which is a CQRS/ES framework for building cloud applications based on Domain-Driven Design, and the browser components for desktop apps — [JxBrowser](https://www.teamdev.com/jxbrowser) (Java) and [DotNetBrowser](https://www.teamdev.com/dotnetbrowser) (.NET). + +We use the [Protobuf v3](https://developers.google.com/protocol-buffers/docs/proto3) syntax for all the examples in this series. + +## Starting Off with Protobuf + +### Lost in Transition +Our Protobuf story started at a SaaS project. The system was built based on the event-oriented, or, as it is called lately, reactive Domain-Driven Design. Events and data to display were transferred to the JavaScript browser application as JSON objects. Then the customer decided to add Android and iOS clients. By the time the work on the client applications started, the event model was already formed as a hierarchy of Java classes. And there were quite a few of those. + +So we faced the following questions: + +1. **How do we attain type consistency across different languages, if something needs to be changed or added?** When there are dozens of event classes you would rather not do it manually, as mistakes are highly probable. It would be quite nice to have some tool to handle this. +2. **Can we avoid conversion to JSON and backward?** Our system’s load was moderate, so it was not about improving performance at this point. However, it was intuitive, that it would be nice to skip conversion to text and backward, as it happens only for the sake of transfer to another node. + Abandoning data types and only using JSON for all tasks was not an option. It would turn working with the domain model, which is “the heart and the brain” of the business, into operations with strings and primitive types! We set off searching. + +### Looking for Solution +One of the first options we came into was the Wire library by Square. Back then, it was versioned 1.x, supporting Protobuf v2, while Protobuf v3 was in alpha-3 version. The Wire did not solve all our issues with the platforms’ support, as it was intended only for Android and Java, but it got us aware of the Protobuf technology and of the applied code generation. [Compared to others](https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html), Protobuf looked like the best option. + +Yet another issue arose. In Java we had a hierarchy of classes for events and other types of data. As most of them are transferred between client and server, we were looking for the means to define the similar structure of data in all languages involved. But inheritance in Java works differently from what was available in languages, such as JavaScript. A significant amount of hand-written code was needed to implement and then to maintain the hierarchies of the same types for all platforms. + +Protobuf looked like a solution (just in theory for us at that time). We could adapt inheritance into the composition and code-generate the data types for all languages. But that also meant we had to rewrite a part of our application from scratch. + +We came up with the following solutions: + +1. Describe commands, events, and *all* the other data types in Protobuf and generate code for all the languages of the project. +2. Since Protobuf describes *only* data, we will implement the domain entities as Java classes on top of the data classes generated from the proto-types. + +Bringing these solutions to life we: + +* Created Spine Event Engine — the framework for the projects, developed using the Domain-Driven Design methodology, where Protobuf makes working with the data way easier. It speeds up the development and reduces the labor costs of the software. +* Cut down the amount of the code written manually in our integration libraries, [JxBrowser](https://www.teamdev.com/jxbrowser) and [DotNetBrowser](https://www.teamdev.com/dotnetbrowser), where C++ code of Chromium couples with Java and C#. + We will talk in detail about these in the next articles of this series. Meanwhile, let’s take a look at the Protobuf features, which make it useful for many projects, regardless of their nature. + +## What Makes Protobuf So Handy +### Support of Many Languages +The latest versions of Protobuf [support](https://developers.google.com/protocol-buffers/docs/reference/overview) C#, C++, Dart, Go, Java, JavaScript, Objective-C, Python, PHP, Ruby. There is a Swift implementation [by Apple itself](https://github.com/apple/swift-protobuf/). If you need one for Closure, Erlang или Haskell, the list of the [third-party libraries for different languages](https://github.com/protocolbuffers/protobuf/blob/master/docs/third_party.md) is extensive. + +As the name of the article implies, the Protobuf-based code can be used for all the operations with data, not only serialization. And this is the approach we recommend. However, serialization is also worth discussing. It is where it all usually starts. + +### Binary Serialization +Protobuf employs the binary serialization mechanism to transfer data between the nodes effectively, to write and to read from the different languages without additional efforts, and to introduce format changes without breaking compatibility. + +Let’s assume we have a data type `Task` defined as follows: + +```.proto +message Task { + string name = 1; + string description = 2; +} +``` +Then in Java, you can get such object in binary form this way: + +```java +byte[] bytes = task.toByteArray(); +``` + +Or this: + +```java +ByteString bytes = task.toByteString(); +``` + +The `ByteString` class, provided by the library, helps to transform Java String and to work with `ByteBuffer`, `InputStream`, and `OutputStream`. + +In JavaScript the object of the `Task` type can be transformed into bytes using the `serializeBinary()` method: + +```js +const bytes = task.serializeBinary(); +``` + +As a result, we get an array of 8-bit numbers. For reverse transformation, the `deserializeBinary()` function is generated. It is called in a way similar to static methods in Java: + +```java +const task = myprotos.Task.deserializeBinary(bytes); +``` +The situation is alike for Dart. The `writeToBuffer()` method with the common ancestor of the generated classes returns a list of unsigned 8-bit integers: + +```dart +var bytes = task.writeToBuffer(); +``` +And the `fromBuffer()` constructor performs the reverse transformation: + +```dart +var task = Task.fromBuffer(bytes); +``` +### Resilience to Type Change +Writing code in modern IDEs makes it much easier to modify existing data structures. For example, for Java, there are many types of automatic code refactoring available. However, instances of domain types are often persisted to some storage and then read back over and over. Again, with Java, it takes quite an effort — in terms of both time and cost — to craft a fine-grained serialization mechanism to keep the data from years back alive today. + +Protobuf is designed to ensure the backward compatibility of the serialized data. It is flexible enough when it is necessary to change an existing data type. We will not describe [all the possibilities](https://developers.google.com/protocol-buffers/docs/proto3#updating) but only focus on the most interesting. + +#### Adding Fields + +This one is quite simple. You add a field and the updated type will be binary compatible with the old values. The presence of the new field will be transparent for the new code, but it [will be taken into account](https://developers.google.com/protocol-buffers/docs/proto3#unknowns) during serialization. + +#### Deleting Fields + +This one should be handled more delicately. You can simply delete the field and everything will just keep working. But such a cavalry charge comes with risks. + +During the next type modification, someone might add a field with the same index as the recently deleted one. It will ruin the binary compatibility. If in the future someone will add the field with the same name, but of a different type, it will break compatibility with the calling code. + +The authors of Protobuf recommend renaming such fields by adding the `OBSOLETE_` prefix or to delete them, marking the index of this field with the [reserved](https://developers.google.com/protocol-buffers/docs/proto3#reserved) instruction. + +To avoid unpleasant surprises we recommend the following cycle to process the deletion: + +**Step 1.** Mark the field with the `deprecated` option: + +```.proto +message MyMessage { + ... + int32 old_field = 6 [deprecated = true]; +} +``` +**Step 2.** Generate the code for the updated type. + +**Step 3.** Update the calling code, getting rid of the `@Deprecated` method calls. + +**Step 4.** Delete the field, marking its index and name as `reserved`: + + ```.proto + message MyMessage { + ... + reserved 6; + reserved “old_field”; + } + ``` +By doing this we insure ourselves from accidental usage of the old index or name and do not have to keep the outdated code. Renaming fields is not much harder. + +#### Renaming Fields + +Serialization is based on field indices, not on field names, so if something was named poorly, renaming would mean the need to convert data and upgrade software on all nodes. If the field is renamed, the updated type will be binary compatible. + +Such renaming better be done in reverse order. First, you should rename the methods, which work with this field in the generated code. This, in turn, will update the links from the code of the project to these methods. And only after that, the field should be renamed in the proto-type. + +Let’s say our `Task` type has a `name` field and we want to have it named `title` instead of `name`. + +To avoid manual correction of all of the method calls, we should perform the following sequence: + +**Step 1.** Rename the `Task` class method in Java from `getName()` into `getTitle()`. + +**Step 2.** Rename the `Task.Builder` methods `getName()`, `setName()` accordingly. + +**Step 3** Perform these steps for the other languages of your project. And only after this… + +**Step 4.** Rename the field itself in the `Task` proto-type. + +Obviously, it is less convenient compared to ordinary renaming in the environment. But it is worth noting, that: + +* We spend almost no effort to generate the code. +* The renamings might be numerous but not vast. Given due initial attention to the domain language, the number of such issues can be reduced. + +Also note, that if you use serialization in JSON, it is better not to rename a lot because updating the clients to the new version requires additional effort. + +### Output to JSON +As we’ve mentioned earlier, JSON is not the most effective way to exchange data. This string-based protocol doesn’t have an official schema format, and the techniques for supporting any kind of data types with JSON vary immensely. + +In most cases when you need serialization, Protobuf will do well. Being a serialization protocol on its own, it converts into a concise binary representation, which is supported by all the platforms which support Protobuf. + +However, in some cases, JSON is irreplaceable. For example, when debugging dev server requests, it’s handy to have requests and responses in a readable format. Also, some databases support JSON structures, which helps their users avoid the tedious job of defining extra data schemas on top of an existing type model. For such cases, Protobuf supports JSON serialization. + +#### In Java + +To work with JSON in Java we use the utility class `JsonFormat`, which is included in the `protobuf-java-util` library. The `JsonFormat.Printer` class is responsible for the output. A simple case looks as follows: + +```java +var printer = JsonFormat.printer(); +try { + var json = printer.print(myMessage); +} catch (InvalidProtocolBufferException e) { + // Unknown type was encountered in an `Any` field. +} +``` + +As you can see from this example, the `Printer.print()` method can throw an `InvalidProtocolBufferException`. + +It happens, when the message printed contains the `Any` type field which has the type unknown for the printer packed in it. + +If you do not use `Any` in the messages, which are to be converted to JSON, no actions are needed. If you do use it, then you should equip the `Printer` with an object of the `TypeRegistry` type with all the types you expect to be used inside `Any` added to it: + +```java +var registry = + TypeRegistry.newBuilder() + .add(TypeA.getDescriptor()) + .add(TypeB.getDescriptor()) + .add(TypeC.getDescriptor()) + .build(); +var printer = JsonFormat.printer() + .usingTypeRegistry(registry); +``` + +By default the `Printer` output is easy-to-read. However, you can create a version, which will print in the compact format: + +```java +var compactPrinter = printer.omittingInsignificantWhitespace(); +``` + +`JsonFormat.Parser` is used for reverse transition. It also needs the `TypeRegistry` to understand the contents of the fields of `Any` type: + +```java +var parser = JsonFormat.parser() + .usingTypeRegistry(typeRegistry); +var builder = MyMessage.newBuilder(); +parser.merge(json, builder); +MyMessage message = builder.build(); +``` + +#### In JavaScript + +The situation is less appealing for JavaScript. Its out-of-the-box implementation does not allow to serialize to and from JSON. Support of conversion from and to “simple” objects is available with `toObject()` method. But in some cases, e.g., for `google.protobuf.Timestamp`, `google.protobuf.Any`, and other built-in types, the output of the `toObject()` will not match the one printed by the Java library. The user of the Protobuf for JavaScript is only left with extending the generated API, which is what we did when first faced this flaw ourselves. + +#### In Dart + +When it comes to building client-side applications, Dart looks like a viable new alternative to JavaScript. That’s why we opted to support it in the Spine framework, and use Protobuf to generate code for this language as well. + +For Dart, conversion to JSON is very similar to the one used in Java. + +Serialization: + +```dart +var jsonStr = task.toProto3Json(); +``` +Deserialization: + +```dart +var task = Task(); +task.mergeFromProto3Json(jsonStr); +``` + +Both methods have an optional parameter of the `TypeRegistry` type, required to process the `google.protobuf.Any`. We create a `TypeRegistry`, transferring the exemplars of the empty messages to it: + +```dart +var registry = TypeRegistry([Task(), TaskLabel(), User()]); +``` + +And add the parameter to the method: + +```dart +task.mergeFromProto3Json(jsonStr, typeRegistry: registry); +``` + +### Immutability +Immutable types make the developer’s life way better. For instance, have a look at [this talk](https://www.youtube.com/watch?v=pLvrZPSzHxo&feature=youtu.be) on how immutability helps to build great UIs. + +Protobuf objects in Java are immutable. And this is convenient. It is also convenient to create a new object based on the previous one: + +```java +Task updated = task.toBuilder() + .setDescription(“...”) + .build(); +``` + +But it’s not that bright for JavaScript and Dart. + +We’ll talk more about the desire to keep things unchanged and the urge to modify them in the article on immutability, which comes next in this series. + +## Summary +This article begins a tale of our journey with Protobuf. Now, as we’ve discussed the basics and general advantages of using this technology, we will dive into how it can be applied to particular projects. Next up we will talk about the way Protobuf helps you improve your code quality and architecture style. \ No newline at end of file From e895560dbc00583905be580133fb4dceec41b885 Mon Sep 17 00:00:00 2001 From: oleksandra-kovalenko Date: Tue, 30 Dec 2025 12:46:20 +0200 Subject: [PATCH 02/13] =?UTF-8?q?Added=20blog=20article=20"Protobuf=20?= =?UTF-8?q?=E2=80=94=20Serialization=20and=20Beyond.=20Part=202.=20Immutab?= =?UTF-8?q?ility"?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../blog/protobuf-immutability/index.md | 78 +++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100644 site/content/blog/protobuf-immutability/index.md diff --git a/site/content/blog/protobuf-immutability/index.md b/site/content/blog/protobuf-immutability/index.md new file mode 100644 index 00000000..7fd9d273 --- /dev/null +++ b/site/content/blog/protobuf-immutability/index.md @@ -0,0 +1,78 @@ +--- +title: Protobuf — Serialization and Beyond. + Part 2. Immutability +description: >- + In this chapter, we are going to address the elephant in the room which comes with Protobuf in Java in particular — immutable data types. +author: Alexander Yevsyukov and Dmytro Dashenkov +publishdate: 2020-12-02 +type: blog/post +header_type: fixed-header +--- +*In [part 1](blog/protobuf-whats-in-it-for-your-project/) of this series, we have shared how we discovered Protobuf, discussed its advantages over the prior art, and described what it takes to standardize data types in a complex multi-platform software system. In this chapter, we are going to address the elephant in the room which comes with Protobuf in Java in particular — immutable data types.* + +## To Change or Not to Change +When first facing the idea of immutable types, one might say: “This will never work! Not only is the concept overcomplicated, but it also leads to inferior performance”. We can sympathize with this emotion. When forced to write extra code to copy the value of an object just to introduce a single change, one should seriously question their goals. If you find yourself asking this question, this article is for you. + +First things first, let’s go through the motivation behind forbidding to change stuff and why has this concept been [gaining traction](https://elizarov.medium.com/immutability-we-can-afford-10c0dcb8351d) lately. + +### Consistency +Stability feels nice. We function better in a well-known environment. So, we use abstractions and analogies, to make the object model as close as we can to the real world we’re used to. An immutable object has much more in common with an item in real life than a mutable one does. Imagine that you are holding a book. On the other side of the planet, the author decides that they don’t like chapter 5 anymore, so they delete it. And your book gets a bit thinner. This is, at best, a nuisance. It would be fair to sue the author for robbing you of the part of the book. Even electronic copies are never updated like that. Instead, a book can only be updated in a single straightforward way: a copy is made, and changes are made to the new copy before distributing it. This is called a new edition. Why do we like our books to stay the same and never “upgrade” them, even if, like in the case with an e-book, it would be relatively easy to do? Well, it makes it convenient to refer to the book in the future. If we’d like to discuss chapter 5 with our colleagues, we would rather be talking about the same thing. Otherwise, we would get an aliasing bug, which is nicely described in this [blog post](https://martinfowler.com/bliki/AliasingBug.html) by Martin Fowler. + +### Asynchronous Safety +Another feature of immutable types — safety when working in an asynchronous environment. This gets mentioned a lot in many technical articles, as well as software developer job interviews. The gist is as follows: atomic operations on references instead of the object’s fields are the only way to ensure the integrity of shared data. This gives us control over how the data is changed and how to resolve conflicting changes. + +### In-memory Storage +More often than not, software, on both server and client sides, uses some kind of in-memory storage for data. Most likely, data structures used for that rely on objects’ hash codes and equality. Such structures are a gold standard for most platforms. For example, in Java, `HashMap` and `HashSet` will not function properly if elements placed in the map have no consistent `hashCode()` and `equals()` implementations. + +For a mutable type, it is impossible to provide such implementations. Consider a mutable `User` class, which is stored in a `HashSet`. At some point, we may need to iterate through all the users in the set and change each second one of them. After such a mutation, the set is full of objects which cannot be discovered by the `contains()` method anymore, since the hash of the original object is not the same as the hash of the new object. + +Such a problem would never occur with an immutable object. If fields never change, hash code won’t change either. + +## Is Mutating Things Still OK? +Absolutely. There are many cases when immutability has too high of a price. + +The biggest reason to use mutable types over immutable ones is the cost of copying data. At the end of the day, data in a system needs to be changed over time. Sometimes, this change happens very often. For example, creating an extra `Builder` object and a new copy of the original object make up for thrice total object references in memory over a single conventional mutable object. One should also consider the CPU time taken to create a fresh copy of the original object. + +Some systems do not allow for this kind of resource usage. A great example of such a system is software which aggregates data from multiple sensors. A sensor is a small device that does not have much memory of its own, so it pushes the collected data right away in a hope that the receiving end will be able to cope with it. Such software is often run on not very powerful hardware, which poses tight performance constraints. + +Another example of a system that could benefit from data mutability is an automated stock market broker, which is designed to handle requests from the market faster than the competitors. Such systems are specifically written to optimize for speed over all. + +Developers should use their best judgment when considering immutability. If the tradeoff is too high, it is OK not to go all-in on immutability and use it only in cases where the performance restrictions are less harsh. + +## Java API With Protobuf +In Java, generated Protobuf classes are always immutable. Each class has a nested `Builder` class, which has all the same fields but is mutable. To create an instance of such a class, we create a builder, assign the fields, and build the message. + +```java +Task task = Task.newBuilder() +.setName(“…”) +.setDescription(“…”) +.build(); +``` + +In order to change a value, we create a new message based on an existing one. + +```java +Task updated = task.toBuilder() +.setDescription(“…”) +.build(); +``` + +The way Protobuf generates code represents a typical immutable API in Java, excluding some Protobuf-specific extras, such as reflection, etc. + +However, an immutable data type can be of any shape and form you want, or, to be precise, any shape and form the API user needs. For example, types such as `java.lang.String`, `java.io.File`, `java.time.LocalDateTime`, etc. are all immutable data types. + +## What About Other Platforms? +Unfortunately, Protobuf does not generate immutable code for JavaScript and Dart. In fact, some aspects of the message objects in JS are mutable while others [are not](https://developers.google.com/protocol-buffers/docs/reference/javascript-generated#repeated). + +Such a difference between Java and client-side languages can be explained by a desire to make client-specific Protobuf objects as lightweight and performant as possible. However, is the tradeoff really worth it in the world with more and more efficient computers? In our opinion, it is not. + +Nevertheless, in Dart, there are a few tools, as well as [language structures](https://dart.academy/immutable-data-patterns-in-dart-and-flutter/), which can help you with immutability. For instance, `built_value` and `freezed` are two libraries that allow their users to create immutable data types with [almost no extra effort](https://levelup.gitconnected.com/flutter-dart-immutable-objects-and-values-5e321c4c654e). But in order to achieve immutability for data types that originate in Protobuf, we’d have to arrange the conversion from mutable to immutable types, which is a lot of manual work. And in the end, we’d get two classes for one Protobuf type. This is a suboptimal solution, to say the least. + +We have tried creating a symbiosis between `built_value` and Protobuf to achieve immutable “front-facing” types, which would rely on standard Protobuf Dart code for serialization. This attempt has failed as we have found ourselves juggling too many noncoherent abstractions. At the time of writing, we [plan](https://github.com/SpineEventEngine/core-java/issues/1334) to implement custom code generation to support Dart, JavaScript, and Java, which addresses the limitations we face and provides immutability. + +## What’s Next? +To summarize, immutability is a helpful concept for many reasons. Firstly, it helps people understand the lifecycle of an object by building direct analogies with items from the real world. Secondly, it has technical advantages, such as clearer relations with data structures and asynchronous safety. Although there are cases when mutable types are still preferable for performance reasons, most systems may enjoy the benefits of immutability without extra concerns. + +In Java, Protobuf helps us create immutable data types with ease by generating all the code required for building and copying objects. There are also tools that can be used for this task in other languages, such as Dart. + +Simplicity and technical advantages aside, immutable data types are important for a greater reason. They assist us in building projects based on Domain-Driven Design. In the following parts of this series, we are going to talk about DDD and how Protobuf helps us build Domain Models, starting with Value Objects, a simple yet powerful pattern for translating domain concepts into the machine language. \ No newline at end of file From 469d6c7d513a5d330dcc4ae8f938a58747bf3daa Mon Sep 17 00:00:00 2001 From: oleksandra-kovalenko Date: Tue, 30 Dec 2025 19:26:09 +0200 Subject: [PATCH 03/13] =?UTF-8?q?Added=20blog=20article=20"Protobuf=20?= =?UTF-8?q?=E2=80=94=20Serialization=20and=20Beyond.=203.=20Value=20object?= =?UTF-8?q?s"?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../blog/protobuf-value-objects/index.md | 138 ++++++++++++++++++ 1 file changed, 138 insertions(+) create mode 100644 site/content/blog/protobuf-value-objects/index.md diff --git a/site/content/blog/protobuf-value-objects/index.md b/site/content/blog/protobuf-value-objects/index.md new file mode 100644 index 00000000..2bec0fdb --- /dev/null +++ b/site/content/blog/protobuf-value-objects/index.md @@ -0,0 +1,138 @@ +--- +title: Protobuf — serialization and beyond. + Part 3. Value objects +description: >- + In the following few chapters, we will explore the specifics of using Protobuf in systems based on Domain-Driven Design. This one describes a simple yet powerful concept of Value Objects. +author: Alexander Yevsyukov and Dmytro Dashenkov +publishdate: 2021-03-17 +type: blog/post +header_type: fixed-header +--- +*This article continues the series on the practical aspects of using Google Protocol Buffers. In the following few chapters, we will explore the specifics of using Protobuf in systems based on Domain-Driven Design. This one describes a simple yet powerful concept of Value Objects.* + +We have been using Protobuf as more than just a serialization mechanism from the very beginning. It provides a language to model the domain. This ability is especially useful in projects based on the [Domain-Driven Design](https://martinfowler.com/bliki/DomainDrivenDesign.html) methodology, aka DDD. + +The defining feature of a Domain Model is the [Ubiquitous Language](https://martinfowler.com/bliki/UbiquitousLanguage.htmls) — a language that is used and agreed upon throughout the domain: by the domain experts, programmers, and in the code. A Domain Model consists of many components, but the language lays the base for all of them. Value Objects help up transform the language into code. + +## What are Value Objects? +A Value Object is a simple type that represents any kind of logically indivisible domain information, such as `EmailAddress`, `PhoneNumber`, `LocalDateTime`, `Money`, etc. Unlike Entities, Value Objects are only identified by the value itself, i.e. they do not have a designated ID. These objects are immutable and, typically, not too big. Apart from the encapsulation of the data itself, such types can hold the validation logic, basic operations on the data, the string representation, etc. For example, the type `LocalDateTime` may provide methods such as `add(Duration)` which calculates the local date and time after a given duration passes. The result is another instance of `LocalDateTime`, while the original object is unchanged. + +As mentioned earlier, the Value Objects are primarily used to integrate the language of the domain into the application’s system of types. It is a common misconception to treat Value Objects as dummy data structures. In practice, a Value Object is the way to escape all the problems related to the data structures, such as data inconsistencies and [anemic models](https://www.martinfowler.com/bliki/AnemicDomainModel.html). + +## Why Protobuf? +Creating Value Objects in Protobuf is convenient because: + +1. The domain types are created fast and for the many target languages, officially [supported by Google](https://developers.google.com/protocol-buffers/docs/tutorials) and [third-party](https://chromium.googlesource.com/chromium/src/+/master/third_party/protobuf/docs/third_party.md). + Let’s see how the Value Object can be implemented in Protobuf. + +```.proto +message EmailAddress { +string value = 1 [ +(required) = true, +(pattern).regex = ".+@.+\..+" +]; +} +``` + +Out of this declaration, the Protobuf compiler will generate a type, instances of which are compared by the value field. For example, in Java, the class would have a generated `equals()` method. + +Note that the `(required)` and `(pattern)` options are extensions to Protobuf provided by the [Validation library](https://spine.io/docs/guides/validation) (which is a part of the Spine Event Engine framework). + +In addition, in some target languages, such as Java, generated Protobuf types are immutable by default. Unfortunately, some other languages, such as JavaScript and Dart, only support [mutable types](https://github.com/dart-lang/protobuf/issues/413). For Dart, however, a community-driven solution seems to be on the way. The [immutable_proto](https://pub.dev/packages/immutable_proto) package implements code generation for immutable types from Protobuf. We have not tried it out yet, as the library is still in its earliest form, but the notion that other engineers feel our pain and try to do something about it as well warms our hearts. + +For further reading on immutability with regards to Protobuf, see our [previous article](blog/immutability/). + +## Validation +The `EmailAddress` type as declared above has one string field with naive validation via a regular expression. Also, the value field should be filled. This validation API is a part of our efforts on improving the code generation with Protobuf. Right now, we generate validation code for Java and Dart, in order to cover both backend and frontend. Later, other target languages might join the club. + +The validation rules are determined based on the options, such as `(required)` and `(pattern)`. At build time, we add the extra code which validates the message values based on those rules. The code is triggered automatically when a message is constructed. No more extra easy-to-forget steps for validation! + +We will discuss the capabilities and the internals of our validation mechanism in more detail in the next articles of this series. + +## Adding Behavior with the (is) Option +An important part of the Value Object is its behavior. OOP greatly influences the mindset of a programmer, and the need to create utility classes and methods for every little thing at the same time annoys and complicates writing and understanding the code. The ability to create domain types quickly is nice, but we also want them to be convenient to “talk” about in the code. Instead of `user.getAddress().getCountry()`, we would like to be able to write `user.country()`. + +In Java, Protobuf generates non-extensible classes, which makes it hard to add behavior to them. We have solved this problem by defining the option (is). It takes the names of Java interfaces with which we want to mark the Protobuf message. Such interfaces may include default methods, adding behavior to the Value Object. Our custom Protobuf compiler plugin modifies the generated code so that the Proto-types implement the assigned interfaces. Here is how this works. + +``` .proto +message User { +option (is).java_type = "UserMixin"; +UserId id = 1 [(required) = true]; +// The primary billing address. +Address address = 2 [(required) = true]; +... +} +``` + +And here is the `UserMixin` declaration: + + +```java +public interface UserMixin extends UserOrBuilder { +/** +* Obtains the residence country of this user. +*/ +default Country country() { +return getAddress().getCountry(); +} +... +} +``` + +Our Protobuf Compiler plugin sees the `(is).java_type` option and adds the specified interface to the list of implemented interfaces of the generated class. Note that the mixin interface extends the `OrBuilder` interface generated by the Protobuf compiler by default. This little trick allows us to use the getter methods generated for the message fields, such as `getAddress()`. + +## Typed Identifiers +A special case of a Value Object is an identifier. + +The Domain-Driven Design experts [recommend](https://buildplease.com/pages/vo-ids/) having a separate ID type for each of the entity types. It is particularly important in the modern, reactive edition of DDD, where entities process messages (commands or events). Using primitive types or strings for identifiers can lead to a mix-up with parameters if their types are the same. For example, if we have `customerId`, `orderId`, `userId`, and all of them are `long` or `String`, it is easy to mess up with their order mechanically: + +```java +completeOrder(String userId, String customerId, String orderId); +``` + +Moreover, using primitive types for IDs might possibly bring a set of unpleasant side-effects. For instance, `long` IDs, when used on fastly growing entity types (think domain events, user sessions, etc.) may overflow. + +Type-safe IDs offer a solution to these issues. First of all, type-safe IDs cannot be accidentally mixed up, as the compiler will catch such errors easily. Apart from this, the code which works with the typed identifiers is more compact. It is easier to read and understand. For instance, let’s compare the latter to the same call, but with typed IDs: + +```java +completeOrder(UserId user, CustomerId customer, OrderId order); +``` + +Having said the `Id` once in a type name, we can easily drop the `Id` prefix from each of the parameter names. And we can even write this way: + +```java +completeOrder(UserId u, CustomerId c, OrderId o); +``` + +Another benefit to type-safe identifiers lies in their structure. In the basic case, an ID type for an entity looks as follows: + +```.proto +message CustomerId { +string uuid = 1; +} +``` + +However, by hiding the type of identifier implementation, we gain the ability to expand it if necessary. For example, to integrate the data of different vendors. We can do it with `oneof` construction: + +```.proto +message CustomerId { +oneof kind { +uint64 code = 1; +EmailAddress email = 2; +string phone = 3; +} +} +``` + +Since Protobuf is specifically designed to allow additive changes to types without any migration hustle, changing ID structure might just be as easy as adding extra fields to the ID type. + +## Conclusion +Value Objects as a whole is a great concept. It helps the developers bring the ubiquitous language into the code and avoid errors by forming a strongly typed model, instead of one based on primitives. + +Protobuf helps make the creation and maintenance of Value Objects easier. Simple Value Objects which introduce domain clarity into the code are a great improvement already. Coupled with typed identifiers, they bring extra benefits. Thanks to Protobuf, such types can be declared once and distributed all around the system, bridging the language barriers between different components. + +Adding an extra layer of bespoke code generation, including validation rules, behaviour, and type grouping via Java interfaces, we get a powerful mechanism with a strongly typed model able to maintain simple domain invariants. + +Lastly, coupled with the Domain-Driven approach to software architecture, we get a system that, from the ground up, helps the developer tackle only one problem at a time, from simple mechanical issues, on the Value Object level, to however complex business requirements on higher levels. + +In the next parts of this series, we will explore those higher levels of the Domain-Driven Design adoption, and see how Protobuf can help us besides Value Objects. \ No newline at end of file From 33ca95e1469b2381a32eae64a92d8855a7fee0c7 Mon Sep 17 00:00:00 2001 From: oleksandra-kovalenko Date: Tue, 30 Dec 2025 19:26:40 +0200 Subject: [PATCH 04/13] =?UTF-8?q?Added=20blog=20article=20"Protobuf=20?= =?UTF-8?q?=E2=80=94=20Serialization=20and=20Beyond.=204.=20Data=20validat?= =?UTF-8?q?ion"?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../blog/protobuf-data-validation/index.md | 308 ++++++++++++++++++ 1 file changed, 308 insertions(+) create mode 100644 site/content/blog/protobuf-data-validation/index.md diff --git a/site/content/blog/protobuf-data-validation/index.md b/site/content/blog/protobuf-data-validation/index.md new file mode 100644 index 00000000..cd4da42e --- /dev/null +++ b/site/content/blog/protobuf-data-validation/index.md @@ -0,0 +1,308 @@ +--- +title: Protobuf — serialization and beyond. + Part 4. Data validation +description: >- + In this part, we take a look at how to validate data with Protobuf and what can be learned from the mechanisms behind data validation. +author: Alexander Yevsyukov and Dmytro Dashenkov +publishdate: 2021-10-06 +type: blog/post +header_type: fixed-header +--- +*In this series, we explore the features and advantages of using Protocol Buffers as a model-building tool. In this part, we take a look at how to validate data with Protobuf and what can be learned from the mechanisms behind data validation.* + +## What is Validation? +When developing a domain model, one often faces a need to enforce certain rules upon the data objects. Strongly-typed languages, such as Java, help us order up data into neat structures and then build [Value Objects](https://medium.com/teamdev-engineering/protobuf-serialization-and-beyond-part-3-value-objects-e3dc7b935ac) upon those structures. However, none of the programming languages is expressive enough to form the whole model and enforce all the known rules. This is not a failure of the language designers, but a required tradeoff for creating any sort of a general-purpose programming language. + +If not solved in a general way, the need to check if the data fits the domain rules leads to conditional statements and exceptions scattered across the codebase. Such a chaotic approach allows errors to pop up once in a while, rendering the system unreliable. + +### Validation for the rescue +A well-integrated validation mechanism sieves off corrupted (or, often, mistyped) data early in the flow and gives the developers a safety net against small errors. + +{{% note-block class="note" %}} +When talking about validation, we mean the integrity of the individual objects, rather than the whole system. Consider a user account model: +{{% /note-block %}} + +```.proto +message User { + + // ... + + EmailAddress primary_email = 7; +} +``` + +Checking if the `email` is set and is a valid email address is a part of the data object validation. Verifying whether the email is unique in the system or such an address already exists, is not a part of the data object validation and is out of scope for this article. + +Validation solutions come in many shapes and sizes, but all of them share a common structure. Validation rules, a.k.a. validation constraints, define the axiomatic facts about the data in the system. An evaluator for the constraints checks input data and reports any violations. Typically, the constraints are defined in a declarative DSL, while the evaluator is separated from the constraints for simplicity. + +## Java Validation Framework +In Java, there is the Bean Validation 2.0, also known as [JSR 380](https://beanvalidation.org/). This JSR is a specification that constitutes a set of validation annotations, which allow the programmers to declare validation constraints, and an API for manual invocation of the checks. For example, consider a user account object implemented in Java. + +```java +import javax.validation.constraints.Email; +import javax.validation.constraints.NonNull; +import javax.validation.constraints.NotBlank; + +class User { + + @NotBlank + @NonNull + private String login; + + @Email + @NonNull + private String primaryEmail; + + // ... +} +``` + +`String` fields are marked with special annotations, such as `@NotBlank`, `@Email`, `@NonNull` to signify that the value must have some non-whitespace characters, must contain a valid email address, and must not be `null` respectively. + +Such an annotation-based approach gives the API users an ability to choose between explicit invocation of the validation process via the runtime API and implicit invocation via manipulating bytecode and inserting corresponding checks right along with the user code. + +## Validation with Protobuf +We work with Protobuf, a multi-language tool for defining data models. As we use Protobuf for domain modeling, we require a robust built-in mechanism for validation. Unfortunately, Protobuf itself does not supply one. So, we’ve built our own. + +### Why develop our own validation? +{{% note-block class="note" %}} +“Everybody has a logging framework, but mine will be better.” — a naïve software engineer. +{{% /note-block %}} + +Indeed, developing something as fundamental as a validation library might seem a bit silly. Aren’t there people who’ve done that before? + +There are indeed other validation solutions for Protobuf. Many of them are similar to what we do. Some, just like us, generate validation code. Others rely on runtime validators. What makes the difference is the smooth integration with the generated code itself. + +### How it works +First things first. The features described in this article apply primarily to the generated Java code. Some features are also implemented for Dart. In the future, we are planning to also cover other languages, such as JavaScript and C/C++. Fortunately, the cost of adding another language is lower than the cost of developing the whole validation tool from the ground up. + +Protobuf provides options as a native way to declare the meta-data for each file, message, or field. The options API resembles the JSR 380 annotation API. Such cohesion allows developers familiar with the Java API to get a grip of our options API faster. + +Consider the following message: + +```.proto +message LocalDate { + int32 year = 1; + Month month = 2; + int32 day = 3; +} +``` + +It’s up to users to define their custom options and then provide some mechanism to deal with them. On top of this feature, we have built an infrastructure for validating Protobuf messages. Validation rules are defined using custom options for fields and messages: + +```.proto +message LocalDate { + int32 year = 1; + Month month = 2 [(required) = true]; + int32 day = 3 [(min).value = "1", (max).value = "31"]; +} +``` + +This example is rather simplistic, as, for example, the max number of days depends on the month and year values in real life. We ignore this fact for now, since there is no easy and robust way of adding complex logic code to Protobuf definitions. In future, we are planning to allow more complicated scenarios for validation rules. We, however, have not figured out the syntax for such constructs yet. + +Our options, such as `(required)`, `(pattern)`, `(min)`, `(max)`, etc., are transpiled into executable code during Protobuf compilation. We embed this validation code directly into the code generated by the Protobuf compiler for the target language. + +In the example above, the Protobuf compiler generates a Java class from `LocalDate`. We add the `validate()` method into that class. Here is the simplified version of it: + +```java +public ImmutableList validate() { + var violations = ImmutableList.builder(); + if (!isMonthSet(msg)) { + violations.add(ConstraintViolation.newBuilder() + .setMsgFormat("A value must be set.") + .setTypeName("spine.time.LocalDate") + .setFieldPath(path("month"))) + .build()); + } + if (msg.getDay() > 31) { + violations.add(ConstraintViolation.newBuilder() + .setMsgFormat("The number must be ≤ 31.") + .setTypeName("spine.time.LocalDate") + .setFieldValue(msg.getDay()) + .setFieldPath(path("day")) + .build()); + } + if (msg.getDay() < 1) { + violations.add(ConstraintViolation.newBuilder() + .setMsgFormat("The number must be > 1.") + .setTypeName("spine.time.LocalDate") + .setFieldValue(msg.getDay()) + .setFieldPath(path("day")) + .build()); + } + violations.addAll(violationsOfCustomConstraints(msg)); + return violations.build(); +} +``` + +Note that we intentionally chose the “eager” validation approach, i.e. we try to collect all possible violations instead of quitting on the first found error. + +Users of our Validation library can also extend the standard set of options with the custom definitions. This feature is only supported in Java for now. + +Consider the following message: + +```.proto +message Project { + // ... + ZonedDateTime when_created = 4 [(required) = true]; +} +``` + +The `when_created` field stores the info about the timestamp of the project creation. It is true that the field must be set, i.e. it always exists in the domain, hence an absence of this value is a technical error. And there’s more to it. We’ll assume that the described project already exists in real life. So, its creation time has to be in the past from now. Let’s introduce an option to signify that. To do so, we extend the standard set of field options with a new one: + +```.proto +import “google/protobuf/descriptor.proto”; + +extend google.protobuf.FieldOptions { + + WhenOption when = 73819; +} + +message WhenOption { + + TimeDirection in = 1; + + enum TimeDirection { + WOTD_UNDEFINED = 0; + PAST = 1; + FUTURE = 2; + + } +} +``` + +{{% note-block class="note" %}} +The field number for the `when` option is 73819. It is due to how Protobuf distributes field numbers and extensions that we have to use such obscure constants. Read more about field numbers for extension types in [this guide](https://developers.google.com/protocol-buffers/docs/proto#choosing_extension_numbers) and [this comment](https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/descriptor.proto#L311) in `google/protobuf/descriptor.proto`. +{{% /note-block %}} + + +Now, we just use the option in the domain: + +```.proto + +import “example/validation_options.proto”; + +message Project { + // ... + ZonedDateTime when_created = 4 [ + (example.when).in = PAST, (required) = true + ]; +} +``` + +The declaration part is ready. It’s time to move to the implementation. + +First, we define a constraint: + +```java +final class WhenConstraint + extends FieldConstraint + implements CustomConstraint { + + WhenConstraint(TimeOption optionValue, + FieldDeclaration field) { + super(optionValue, field); + } + + @Override + public ImmutableList + validate(MessageValue message) { + // Extract and validate the field value. + } +} +``` + +Then, we define the Java wrapper for the Protobuf option. This class is responsible for locating the option and creating the associated constraint. + +```java +final class When extends FieldValidatingOption { + + When() { + super(TimeOptionsProto.when); + } + + @Override + public Constraint constraintFor(FieldContext value) { + return new WhenConstraint( + optionValue(value), + value.targetDeclaration() + ); + } +} +``` + +Finally, we implement the `ValidatingOptionFactory` interface, override the methods in it, returning only new options and only for the necessary field types: +final class When extends FieldValidatingOption { + +```java + When() { + super(TimeOptionsProto.when); + } + + @Override + public Constraint constraintFor(FieldContext value) { + return new WhenConstraint( + optionValue(value), + value.targetDeclaration() + ); + } +} +``` + +The class `WhenFactory` has to be exposed to the Java `ServiceLoader` mechanism as an implementation of [ValidatingOptionFactory](https://spine.io/base/reference/base/io/spine/validate/option/ValidatingOptionFactory.html) either manually or via an automatic tool, such as [AutoService](https://github.com/google/auto/tree/master/service). + +Here is the diagram of the classes we’ve just described. +![The diagram of the classes](site/content/blog/protobbuf-data-validation/the-diagram-of-the-classes.webp) + +We expose the `WhenFactory`, which implements `theValidatingOptionFactory`, via the `ServiceLoader` API. `WhenFactory` registers the `When` option, which, when necessary, creates a `WhenConstraint` based on a field definition. + +On the diagram above, the amber components are what the user implements and the blue components are the interfaces provided by the Validation library. + +{{% note-block class="note" %}} +The `(when)` option is already a part of the Spine Validation library. Users don’t have to redefine it on their own. +{{% /note-block %}} + +### Protobuf `required`: Il buono, il brutto, il cattivo +A reader familiar with the differences between Protobuf v2 and v3 may have noticed and been surprised by the `required` term in the example defining the `Project` type shown above. + +In Protobuf version 2, any field could be marked as `required` or `optional`. Later, the library creators [declared](https://github.com/protocolbuffers/protobuf/issues/2497) this approach to be more harmful than good. In Protobuf version 3 and its [analogs](https://capnproto.org/faq.html#how-do-i-make-a-field-required-like-in-protocol-buffers), all fields are always regarded as optional. The reason for this is binary compatibility. The `required` field in Protobuf 2 cannot be made `optional` and vice versa without breaking compatibility with previous versions of the message. For a serialization and data transfer tool, this is a big issue, as different components that communicate with each other using Protobuf messages may use different versions of those messages. Breaking binary compatibility means breaking such communications. + +Seems like the effect of using `required` is a net negative one. Why bring it back? + +In our Validation library, we decided to partially revive the `required` monster. We need to describe the requirements for data values in the domain model. Some fields are required by nature. Of course, it may change with the model development. But, unlike the Protobuf v2’s built-in `required` keyword, our option `(required)` works at the validation level, not at the level of the communication protocol. It means that invalid messages can still be transmitted and serialized/deserialized. It is the programmer’s decision whether to use or ignore validation in each case. + +For example, in Java, to create a non-validated message, we use the `buildPartial()` method instead of the regular `build()`. This method was introduced in Protobuf 2 in order to build messages and skip checking the required fields. We expanded this notion to state that the whole message may not be valid when built using `buildPartial()`. + +{{% note-block class="note" %}} +A technical note. In Protobuf 2, all fields had to be declared as either `required` or `optional`. In Protobuf 3, all fields are `optional`. In the fresh Protobuf 3, in [v3.15](https://github.com/protocolbuffers/protobuf/releases/tag/v3.15.0) to be precise, the keyword `optional` was brought back. Though, the semantics of this keyword is different this time around. The Protobuf 3 `optional` allows users to check if the field is set, even if it’s a number of a `bool` field. Not to be confused with ye olde optional fields of Protobuf 2. +{{% /note-block %}} + + +## Proto Reflection +Under the hood, our Validation library uses the Protobuf reflection API in order to obtain the message metadata. When a `.proto` file is compiled into target languages, the compiler exposes the metadata in a form of Protobuf messages known as descriptors. Descriptors contain information about the entire scope of the Protobuf definitions, from message fields to the documentation. It also includes the options defined on the messages and fields. + +In some target languages, a descriptor can also be obtained at runtime. For example, in Java, every message class “knows” about the associated descriptor: + +```java +Descriptor type = MyType.getDescriptor(); +``` + +Unfortunately, there is no analogous API in other target languages, such as JavaScript and Dart. Fortunately, the descriptors are always available at compile time. Users are welcome to add their own [Protobuf compiler plugins](https://developers.google.com/protocol-buffers/docs/reference/other#plugins) to access the descriptors and generate code based on them. + +## More on Code Generation +As of the time of writing, we are working on a new mechanism for code generation, which will also change the internals of how we generate validation code. We thought we might share it here. + +The new tool we’re working on is called [ProtoData](https://github.com/SpineEventEngine/ProtoData/). It allows reducing the effort of generating code for multiple platforms. Descriptors in the Protobuf compiler are a language-agnostic intermediate representation of the Proto definitions. For our code generation, we also build a language-agnostic model, based on the Protobuf definitions. And then feed those representations to multiple language-specific renderers, which turn them into code. + +The whole tool is built on an event-driven architecture, which allows users to tap into the generation process in a simple and non-intrusive way. + +Right now, we’re approaching the first public release and an API freeze for the tool. If you would like to explore it, visit the [GitHub repo](https://github.com/SpineEventEngine/ProtoData/). + +## Conclusion +Protobuf provides a great variety of choices for how to use it. In its simplest form, Protobuf is a serialization format. But due to the systematic and future-proof approach used by the designers of the technology, it has become much more than that. + +Protobuf reflection API, which is, originally based on Protobuf types itself, allows users to bend and stretch the technology to great extent. Access to the meta-data and entry points for quick code generation enabled us to create an entire validation library based on Protobuf definitions without a need to parse the definitions on our own or to do heavy operations on metadata at runtime. + +A validation library is just another step towards an expressive and resilient domain model. And with our Validation library and Protobuf, we make this step a commodity. + +In the next part of this series, we will build on top of what we have already learned and tried by looking at domain modeling. We will explore different kinds of messages, see how validation integrates into the information flow, and discover the hidden benefits of using Protobuf for a domain model. \ No newline at end of file From 349720b10692899f3ef6880ccd65cd86590bc643 Mon Sep 17 00:00:00 2001 From: oleksandra-kovalenko Date: Sun, 4 Jan 2026 14:17:18 +0200 Subject: [PATCH 05/13] Fixed wording of the series description. Fixed typo in a URL. --- .../the-diagram-of-the-classes.webp | Bin 0 -> 15036 bytes .../content/blog/protobuf-immutability/index.md | 2 +- .../blog/protobuf-value-objects/index.md | 2 +- .../index.md | 6 +++--- 4 files changed, 5 insertions(+), 5 deletions(-) create mode 100644 site/content/blog/protobuf-data-validation/the-diagram-of-the-classes.webp diff --git a/site/content/blog/protobuf-data-validation/the-diagram-of-the-classes.webp b/site/content/blog/protobuf-data-validation/the-diagram-of-the-classes.webp new file mode 100644 index 0000000000000000000000000000000000000000..0c7cd8015c891874f4226e09d44fb242799c478c GIT binary patch literal 15036 zcmZvCb9iOVviFW{+qP{x6Wg5FwkDa_wkCEawvCBxClh;rdCxib-t)&-&$GI#YuB!> zwY&PaYSnHP8A-_|UI0K-LR48pnOh4M003ZrT|r=gG!TG@ys~r(G5`Q_GXeyjm3{y> zRb^;Ol-CkdMR(4|qJ}pjdI+qqK!$_==U{qxu zd`eA6CU@z&=Uy+PqM!kKfFGYvZ`;=aJwRZZUpYGMbM#A?+3>rK42xncVj;?}ohJBw*BVpR7>?umH6ex;-l(G~QVHdUW z-?_4=E)IeKYKi7vNXz55Y?(Fibbu-*p7KA00cMA4 zl&aCP(uFj0Rp=Z2+333}>WL=EvkgAQQtAXn2R_DEiwCsTE1&({%6aK_t%rw?;EpjH zLN~b961`z;u&Oz3fkZM#v|-&QmMAYJ6UiGUx(4o+hKoD{bWc&s5TGZn31)NRSew~; zn2E4mY@UBL`oIC(36@FINtPi3UM<}1Bv9>WGbuXt5ll{o4f96Fi^!o}eoT-*(vC3l zE<>H7hg@6ScwDfBv&p+EZYAd?$Dy?op^S?6aRmB>ItdQDZsddnVs~h!IRIz!X~+#F z%(%UxBk!=#`iJx5pQ6~o%iOqN>Lmq}*w~9f)%-7e%#OQc9+Zu>8t~t;?z}p4CTg$2 zgjaGb0Y>yT0Hj75(yLCWe2O<=r|NmJ*kgcE@kH1%NWRJ^^|gW-7{K$x0D^kil8^ZP zN)9kPW4_D+n9cDc1SGB^LOb3+5l<(8R!vpr6U)S;V)_#mYs^vQbUd3rK+Ukut+tmcn=n+lO~GDs%X+xml0bcNt=X%n_YL^pbo& zpP(s>J@V5(>xtwLpzAO0erhK!lhFKfa^#}z{^s{qQDbJTU70Xy?XOY{`G%jo#A}}k zgak}=c!5Q&3c@2<@}P-1d=+H*QzJ!z74ypO$;Dbq-7WnbH*1zxVd?UvG5AKT9LvsO zT^UM=mRX>^{h$W!YeMCadPRGj5c0cE%I^u|e}zvb+!JG;_5hm=q6Q{^`RWO2EJ4eG zcy?U)wV?p5lTMcp*(_3faMbC)-;WHH&{;9CQzj-!3Cj_Dr6ek${t(c>uIMO_RpY9C zNK8P43K(-zK?<|H(zvva1CW_$Bxz1d#$}xnorwWzmkw7b2ohoeF3YkvO`XN~NPr-_ zNwM#uf9FW_MUj~#K%xETH|Ws+s*nfX7#ta>gXC8fW1owZqZgtxh7mWE>f{&WmR;Wd z8AXlk17Z?awNZM*On#ZQ70rUG^omO~py_ zt=J^Ld(T;aN|)+J0|p7cT4hYvVclp))!DzZ*FrtO_s_ex>j1h9@0-LTfjPCeeH{$cHL|MlygGi`KSSNk^v*wg&N~Oi$%*N2aAdn{2|$7s`WD-%{EwE% zvv%K9zY>`RWR&1@h!U(2-R~ifHhQ;XE6vPKyJnq*#I6?OtgBC}UYj66QyRSv)?~he z7|!YYB}0$?jVMN*=bSN73AX(A$l!4UR<*E>9tUa>x#v==AoLa|7U>|`IS%w>I%&u> z`V>bR`4VUHKx&$PIF8GgxxyiNP7qu|3OE%okdHGwv(HcHokM*9Cq~j~%-cGCt1TxO zZI7WiAv*wW+18PlY~U%8Uwe-1;()%da6Aty#OU|mlvAe+w4kRbGt~$WZLsHiseFPw zIV(mf!SCHZy|#!$33MUFeQkUIxveLqxIy^{ZCM@onU23zius<_r+llcl1BdiV1cO@AVv}*$n-;yfM2V;o*ccyv7keEe6WbPfQ!!oGu8~m?7>Fg z(XO>2D&BUsoJ|GF1zO30LVVPepaH zLVY%@h522UCrc4}s+5e=~8LUt3J zo*;NHia0Sq{?u3_S6ir-!;=)r1!$v8KH(DrxcC%UBW7JMaQ0nd2Q{WTsvqbp2kZr4 zc}M`%@m-RUD`N5YUzm7^*sz!&X+jXlPUn?T!lA{K5MW~$7YMe8=b{qNhudOx87%`; zmtBEao_*f?n$68#BXMSim53LP{{G(BkeG*rFIQKR%}g%cpM^W(b(-HXF@#7Z`E6rC z10|0<^EWG++7v+Y_cI@q5X|}G7r}nur(|*t0D9|QQ!Gt_MvjWBPf?-Si7r%J#wVEK zV8T2M4}DBRkt})WcKzde&TXk|7psuin{s+TdIHNpC1Og*t!-|&l&|nTyTqZ_Qhl2e7b`#U$H6cjdo-oB1C=sZ~pTixYbka>V`K z1N?LwyqwppdAxA&q0Eao;S16Y%QN;Bdp@0t3(G^~Cw|9%iMdhD`@#X~BUqtPxQ-UyyKG#qjRio6tW4-%b7Abfna;WcAG{{Ul_c z9YzSV2Hii=Q|B3G+FW4{iila_8K>_Hr7(Q_zZ4u+G@mo3l|u8<3kzurkp%bAcgv6h zQ2p<@RAZxJvaoOnP@B1{f$=~ny!i49fp7@}C2x5l%ByE z?6D(QJ;ehkbR;r@dOBk_Kdi}J8aUQv^K)Kz?s45kyXh=e>*_;S&}NDaKYP0rWcUQP zx=Sf~j!+anxhKXCs&wcW@k3dtVRlKE-Ffc9y6llx)(*&|s?w-*V-CEcN(b7(ajg-wse5iv zoecOiH+IXb<=>dTJOj%Mm%5Wj`&Yi9-FMe~LSV?%xC}+PV z=q8__?8$VgaQQ_2007sq zZhVF+B*>8{EbkZ=LAyei2~_xsQzo%^(oNP0zYujGERp=ru6VJdUlSHBsMe@7045t8 zYy`yfjWm$=Vn20SaJ|uSL{=t|(0Nj&Vi@O!Kxh3@W{oAoO#r&&RyKcCqwrRf@tBARk||ueTo1k_UCe{Z_fyqBZj|Mw2Tbdx&0w>03c=1mmzc8|}7_ zj&!ewcu4q2mh`G<;*sKwkNAKq=Q2OgRlkJ-P8>R+-r_Y&3LIPn3E}Bui$u#mo6i=L z74K!tr7pvO>+)yW5YXEyXgK!;++g#8>Hu z;TTs=4BV!nkVj|)MF+LJOECt(bt*xAD9bU}f)GQl_>tkaD=?nJ-2)KyDq)Br%c1ir zd_4C_-a2@9V*0U}y4292CZ8Y$r@hdH;Q_}kfayFqs8r|7TIFj*)g}|It5G27rQ;_V zZMaKVp1n|_gD}x?N$d1w`#cf$keGwL@Me|nwS!P5PAHgA&%^M;-CghT@%5~ZSpNJf z_=i{Ch5kCe@LO6-3^e{MEVO~S!a0aqfc^3`28{5z+uIXUE$E+@OOU2yF}>MCjPsvk zfc}G0fVUzfGR3mG=+7x@G*Lbe4KKYKSn#oGkU{$OHXC()0E5s93qX3y-N?=M+`fCj z8Pjz!DrK+ruL0g1)s5=bcgkmBAWfqVoFMdPy4MFd5x9M+sdm0f34jGEU3+qX10$R$ z3d7ZC+-Mh1U>sy3J%jv&*usa!%SZK|uUH=u*#%&|MRU8MMD9i1GRbBs7fcBwTvSBw zyxUyd5G#K?O>T!;VNlv2hDd$>PYzitAmb7Z{3#D zn;MLnkm7WsiUXG+?nu1SScXI|zj`<%t~(t>nSwLlqj{fQFz zd*G`b51#h?iDD1SOZdMT@5_fK?SxmIt7=w(_2b*Mq0|OF+{U8!%g6N-5Ey^}_^MYB zbJ73+vsZu|5ZYj{bPz!|-gFuABB273g)^HQ#7Il~cOy~AWTmNARc}Qc+k8I&m8C7> zhjs+lli-ITv90*K5LH0(_4!HCyWmA0Gg0++)7Fh(VnE0Jly^=*C~)CI=r!$J~Y3%GU_d36x!5&8rIr8dA`xE=$Z zbAJIlZgk$3--V2X4uD(0b70PU+pG0w8qnd5=^ogYy9ty9asfv^q;R>DA0;DgYI4;#P? zih1nbI_6dhsL;`Ad$Q_#aSqMbuy2Fnc$?sglt^ayks-ApEUC1wLP(L1VH0AE*gJ{@ z=l2>?3%DlLL(u9Hc#3Gu`XoW(TQfIoNz)>37;g;J7`+lm#EfZL%}5y0K+{&IS|J(Y z8&y-*@Nq*FqUeQXME5#6)htpm{B)ge4!lS5NrJ|3O}IdB0zYcxjm>E{!5jC`@mej_ zbcjk}ZXw4x`>1MiVD3-bV6`+)Y2YE@zekooC?o%Q&7OJU_DHNQbP&37{9Bx}@roX> z8fZN`Q>HMg`mZ}>jXicg?A(Wv85PLW$hn=*$^;#C*Kf}}o;uM6MaG$~pm>ctg33W~ zJyH74cj&djnECI#Oql#iJ?JUc9VY&Ni=LbV+(tJ(>iCQWEh~bh+=@_F1MlTyuPrCW zysgSb{kO0G6+PzuX>zJ2JlW{U54w%%$v3~n9*#u}TrsA-DyRL^Ia>rfxe?Z@{X_u2 z9r0RqUV}j#cew1EAHaY#E`O)rzv6cPwhU~dB~Vu_6S=s=eVlZbF8Y4&Q?y)fywjE2 zpb;DNmNuUsdvY=rQyPozn8GnCtBy*s%vxrnf65Ix-|hvTFJpVy=K9~^pYpS0pb?&3 zJ8s40XL2L-yfcI>lLI&+b}?&BF!^{O)8vx=zBlvMhS#Fj2f-k&h=?K4s5G;&~y z-fGIDBPt{W?xT(PIc#|*Cnf!?}u zeb8IpE2Xe&t%`iE09-4h?$3X3sZ=buo(j;-1OQA6Rx-b8l{o|i4Lyhn86~f_z&G)= zm^zvBGEt8N0Pk%C7grg@^j8Isvy$Q1GV`kGmIT4(3>sn}oi&IVafV7_t9bx>oi`IXxNP^gHqI=z#l(iJ}OfG#J5a-Ln0li%6&JWb%8YQphsL~2Y z>i+lmzhXdpXuW*pYuT>_UcC(>R5-fyXSPt}taI_al!mH#Cp`<5Ftxk{r08Aq{_R>d z%qOK~@|cW(tGO`F^@w#YzL4l`F6cfWr(v^Ne$#Zi;FsgF^-Vc!A;M9g>LsjT?T^J~ zad_(KVr*yKPFcB_$V5%39cReL=3@;Fcpm==YU=C#TzdqZW*z^M+h{?~Gd*lrs8>Z2 z5e4S3fkr>MhSd4b_?316%ugKc3N}UIJw5tSA704!rmS%9O#c z3|Y9LwAds~}v~yG*%@@J5epS4Ehh`%Ti>>Tlc>d*Wzn8dHA_=)P$Q>ZuSabs zxk7%|x%Z!K2M)Tc0QP^60doN06PQl_^_KzQ>Jwxec&vE?agtnBMwU3UcG?~gTmjj| z>7}Choyyl4a=pSz^(#@YEQGPh2@hA%^{1is~N2bo(QZeem_Q`;yn*&2u45?nqp~w@v zc=PoUsKZ9W$XP`m>W{mK7q@n~ho5Ks%aM85-IrRof!R(13;E=_M1S4BdyK7U-%&&M zh@**b;6&L`9jD4zoDN1E2Y#CqW=}X<;1C7?R9eLFMU!aQsOJX%pawo4WplF0fK(9v zRES;(KaulPs5(ntky6ySzmC8NU)x6AqH;J`*7S|I`P-Xo1#O2jed)f zV43MjR~Bn~*~0x9T0>w)Hbs0}yW@th7O!1+ce~a}c|+^m2Zc)?dZI_9cD@wnna>rL zd+_XX=HL^!2F>{BsDKAP!}n2?!ccuMsT&-^dQ=CD4(D=bR%Z&gSBAZ*+zu%NrmOmy zubTCy8dfh}*in`3E*wMxDsNr;Y@e7_mU9kPIs+p}#W7Dt0dlXMhVu6O1}~GdHv~f!_hTZz4Uy>kv67a4*EC zDssuh43->yOGf=BJeSymLBo9qbNWNzQtx3E@YX}nXorpgA+fa^Lul1Oz{oS)G>6Re zv=2ZvhGU5EKZlRZk44Il0R*}BtEhPT#f~~yR~%_U!PtH8rlg!xuCll?zm?U9!ff!A zuS4Rjyssc7&)n{|acS`>JqnBTBP}?|$ipz!@UAEAonkzo!}oeJ9a@$N zN1D5P=VTmwT^ZWel*1Ef2*mXpj*3g!1+5e_r0Nhm5>$|am;S00=Ax(MRFiINuNi#8 zzYaUWr{_s_(=Q>WLQ8`sE89T1WkOgKmCQSsbuMta)~E?jsDJm1p%clS^Z0v<3CV>K z1Q%Bo6D=Ok31f8!r_aB^pzxR3l=Vs-`N=SJnDM&9tK^Z!->rrW_Xmmfj%YhF$A)%S zm66rM@!&b&{AxZvI!nsXx9A%Zp0$w#{8Ub+1UqPswd&AAJB9xY=z7Y~6(lE}0_aLq zavr4BU$1F^oFGCZQp_t(p)etneebHeII3LC@u&p9>Q`lSkQ0j5%Z?~RVH*`!k0@R8 zgS|hYj-pm+jh-wV*S8vVqjB8GX2cdj)!Kn>!01VXADgh^?&-s2HYZ-glUDk$A_@g% zB6d)f(NZ=x-vsxS(~%`}DZKsJKY1DFTaAO|RdrTAxljwearcql4?2m3%W*X5eu@gZ zb-eSwI-TCvYeqE=kv{c|`i`aS2__%TNQ<;%+dn4#(oGI}wU7rcE#Phy1~sp)g4XvCztuuP?7cInW)lYiie!mDIUZSMi&EGku%PLIoEkwdZK? zeVS%WMY(RzpR|(uHiXzl95~U+5%q?Y{ivJE$C41_#CfDE$~#7CTDZ|Tayxo*;S zwmS3nBl*2zCoi^h=3r)-p?`3be}%=oe@RXpy`+&?M8ZUkK~>>fI{t8&R1aHbHt`?S z=+^<6V1@pZEP59(G~LDukdqm0#AAM?syK7IEdZb|i%FO?qXuH(i4}{1W3cf>fVpkr zgAni>?RegEiRiE9_GUFihl6LYA*QrXfxC_L$ZAM`zN+_7bU9_znq0%TFE8|BuIJ2V z__Q5|!JKzG*F*{OqedxS_=8l6;XWgFqDfK2=#V;c;EJ_JvQUS!rpU7_ma6jM!X5-A z!lmFo6jmIOT2JGRrqVfbPXIi6j0rVp;%;euxf{P0vLZ08;LmC>m|FD}w1+(zFjCGi za_Kd2U)e=UF+`VVT8=*<-+WB}Blea-v@bdvOItRM+iQ<-Gx%b^S)6T5lByO8^@kTD z+OY;F-u-tjMR>U_&f=C5G6;m)TYn4~ja=u;q6zOA*%1X~qDH&kwB;uw9F_*|lgE(=0Hm>R$anhtX=mytP#;bd-c2@Lb0+UC5^ZtO+(;)cuRQYp z=+yr!a3xT=$)?o=SZN1I)?jD9f-q!9!~e~*uUn+B8d{s3y8;d0HEfGPZpmS3FD@s! zZP-9M)s=9H4Co(pqPr?Uve@EoZb7)57&om4AHivVZF6BpMC)^+;xLmw zbgI0sW$$9~r80#61Uy(0x>tXwBDnjYwD|4BlcJ?ZLimx&(ZeMD)X;(}Ye*7CV$E8a zpi8yweY|+qSdwu=a+V;)ZgAMON&UN90VW1URDRhi`wrPDr$(uta6tv#w*vTC= zXwo|px|Ek3l>GjujqDg`htQuG$*iKMoaH+)Xuz6Y|1IeiLlCNcGO%`givcr09kp}% zqfkR^IcP&K;?4#1+jX?`g-A#Vl@o(BP`Z*WhAtbDv+&!zE;kY{p_eLmjPx&2fcY=8 zr7=1j=FZ#1)fooR%jSj>*rgRbfC|t zzxpQ^gk2xgQdCe;H+@?bN%i^4t7k5VQ^BNJ&t@nG{R$Bw zz)Z2~c^{Wqh10Uf$)Fkr^ugpa$9j#04_^KIVKT%LM$6kyv6Ea(mi{Ds`AYpq;DF)= zK$MR*0f~)%JLMV9HAoyz2&yB^jB~0UkIr+^j|!c%p?L_p%yWE1oNRTXuF~gvkf-`j zO(S9Vwhx+l#f$oTLdDcT?u%!CPtjPFvjp7=9E$!Ff2F57IH9_Qo;7Lgp6ES4`aP*yepe91x+}5$1Y_;Ee?#o; z3kHxPLuNu4FhUs3X`kK*HB>{iZ!+9n8;Zwg+<-}8Ng772`m+|~%7HUv%9+Mw2TBug z_grlnDKYt*T;1W&Z=DAmhPsTcafqnl-QU`ca9xKvaEK~*u<>&96!VRpj+N8BvB011sNFxb82`|b4N0^#~8Io5Dw3!H^5moDPC z4jn&g_aO8O$ruvWd0gsQgFefFH@F<-lC*CHF{I?kI(CsS*@C%?OMvPu-GM0?+Pe8Z zDY*OTtDtCb=#vCQItHU#J1y)y34*rP;9{1(86k>7&|DK2Fv^c8{xxA2vm$iQ3B(zb z)F=haNqCb?TbJu+$vY1&#P8aELYaRh)%Bz zD7r?m=xMMoRwd)hN20VncLkMU!hJ5OEF3vYSb>^#Zt>|8(ARp?11H%Z4EaW=1yy1U zyt6PYeXtZ>q@k@L`2fKXT~eQ6q03Rh4&E$u>N=4U?sda&e}0TSgWKgtNV$XWU8igx zmyA6^F1m1kn{9G{BXj8jHS+VT-oOo}O|U2yyQ)4K*_<5nuLx&R zQF04=VgDq`Ln&K6=*$y^LbU!X#gdP}0HYvoru4}u;ke)%Wt8+qBUpCx+-qE{Lt zP8Dc`$2UE=pQ8X380+1e4r0RwOS`Z)?$3c9)|>_6>{;ZZZGRS+D@0a|q=n*g%ljQ` z$1vp@I#pNS3v$UBXi??MND%%xgA>9}RXz3@s3Z%lSFdU-3;7=6(hM6$|9uUu`)-Yfg}E#w9~B8yOYv8oxAG0di0;v5emC|QSbA42UMOV~#$eoaKl zbn==c=iCGy2$(;_v3dm?g+{D6Gev0ZxOZg{Qi8`zic+X~-n+B%niRaV61o*x{~C0e z`xsQ)GZ_)w*dUqS_A`!!8sIIJ_3j`vAJSa8W+zLO@6NBo+evKGV8kS3rDGpzy@oF| z)w8uK93nkFRqaik>*^hB+fXpXeeky~9G3mjYF7aejXyOPqU!K2XyWs(EMTAT7AYFDAW#~wX;T{GFuIM_XV)eMF~FfJ0(5}ef!h&BF@V8q5g}=xnUUfB|->H za&_jYI#NkWb$F*^K|>A&Ug@4armxj56MHg;>6C18jTN7uQD2=NXNl zuSE2rC9Gb>q%ugvsUu-1r0eUgpX*AK7m#)plUgSx1)3Bg6^G8Z{i$oU`CAuCQ-Y|E zX^I7372c4f%y|4>QL?Ih!THpI3;AT!Qt0WvZpPgi^8675thKQra5%8i^QiZfv4hs; z5-OkDr`!Z%g)S{o^?Z$7)5dE%9|so$z7=J_CT(41gIzPOpDg)!)V$Y=6V)W~t}6}i{z~$Ni`@vi zKlI!nHBi$v3@o|=X{wF1$&OA|i$6>un1|9is1mRqi+~7@3T878z?_qq!{rZJMO1x7 zguq|H=jde)ZNdB3RNVXO6EJF(f(7j0WaL=hdOg91E~X45q!cL+u^zSeW(G| zaN(3NTnSCul(VEA(PDE#(Fk>Ll0l=fnsdKnuZjJ`>?F{|^CWBhUd1E>8y$F}v<%Gw z-*bG5VIfOmL#bOtzA3wMw9*!x$`q!*XR1QKmcq6Nj#9!co39R`j=qs$-!x;rZw&I{h(2NU+eFK4f{>wdM6}k z{IEk-vQ!D-x(~QepYX?a#(k6lx`GRoTzt#f(Z#JGg<_*Ihu4TIQ20v2^ldE003Hwr zF!M09Bv2xlnm{tjX_)3d;5}Cuv{#RtNtc*=Aq*G83F`J~f;pU%p1CE2mMR-B{z@J+ z#ffZg+B|P#67zS{2E*_sB~upPgd`tNM%7*^VoINB7jIDY&%yiY1@zA`un`0)5hi>u z2a^5_OG&Ev!1k)T@0d(}`b>!ub`JqTJnL4-1BPwt3_q|O&Z_q2D&I72(66|3k6M2F zAFuJ3JIJYqX{!GLA8PE-^Bwj(1s3(Uk_ci?aC4#VqymjqQNla`CK|^NH zOi^o<{Bgtq_|?XGcF#)Z=l8dFx)O^ar=KsrgU2Jsgb)uzgO%@hIWCL5I!q5U++@RaKu z_4@+j$59xbc_ey+O_FUyha+ig8K%xieo_!wc*mWhaRq7Xoo=p))uj z;ah1NU^Y3ezP(-sQOv>mQvP=BtwIzOL8S0Q9Xy??b(ujA|E|K?0qoFNs6>GZLd{1C z(a%CWT#?XSi1tMSt#;86FlXxb2f*1oR?TjPb$-@PHs6)Kuj+)N-&+~@mF#`h5YO!K zxfDLm4qbi!!j@JVAma6|`uFMtt27D<$JiSSaN+RdQ8Qv5j2>m}y!mhr-uu#;?(pcY zdCc_KTKg?_RGGu#OPeW;SVEFhTW1&g$oMF~J7qZB`JH%v#$Yr=!}O15>rd^I3b~gJ zP(!nuWb*Fa@-L-D$r_;ku0Z54w2NNHX<@UqSnH_*L*LmdC&BQN; zG0frTN}7`A!|@XWeA^72`FkGJG4YxfLpiL|v5tAOuw8e$zAR4Rx@(+yf)*_R4xrFN zeH7s5B(N1D9qXuNq}mt=kt#R_Ow~|xYdLQ)Ch*&b;Fsok9W$3|f%V?a5y$b>Z3rJ^ z=4K=YKD}=A0+2oD4Uo(;BWJ}xc$T6Md5VV``&8(0mZB1phG6)d*!ckfYKC0-XE0L5 zUI1wAtHca&yTB7*9>}lEuXU_M>MBnS{p~eJFu8rxi2FNe%UFNxK~)dgKnFd~7>%~E zTlYnlgvu&<(Y4&4_^iUEKsCTy3d#s}Ft9!_d%-rM$rohc9aj#%VQD6&n&EZxkhPXO zx>~T|(725VjlAlYz)$%@MWVtjM^Uzu+qcd=GiRTh?&u;ld&<%)CEPB--S71nd8xfN z>3X_PL_>pkG4t+01_kn7Q!Wd8Ev?NC*wWrFtu~NLi4mQ~M=+h#vBXxB_`hPa#JP@v zs!iNs6BqRN)CB;;kvOTokqe5$ouy=&)75dYU zit#NQ0q>L774vstdub(JEw*CgLo18N1O)O>7alV2zYDCu21|0&bVY{(b`Jq9USjZp z!iS+Eze%Kxe~tW{j(?X)D~dB8$WNI~Svg8JM$`&6>}t2>6$oSPLClCxp=b3$kX8Y& z--Mqbe3w(v$-dGQ-lYmzzLj5aK9q)uNFLOQ&J?mU8c+R|)%Hu%3B@OTVTyR4W{=>u zU1+lw9M$3Y33*Ss<>)uMk*)QUYujzHwM`qdw`kUtV|cDT|Et-HQeBUj4rZ3?1IeZf z6k&Ln&;fy}AO4+`O-$P3XlP_q_M7j1@;hpFT4nuj{)U`q{y z6&W9|Gcis8ODV@5QpxPKa?_P9%UpEhAShqvj-DB04T-J_?XOh8X)W&fbOeSuU@u0@ zOTOTHI`?V9?hV$ZQa2BmnwzFFg$bR5p{$8FTT&BOAC=>l3O%i@ zc)QPcfF|R5&wbjqnq$9K)c1iJcS4TfruWBJPDLsh-UW zh^tk7v87cK^w;d8MCP34HX%&qf}8vi9B{%b_~W_Dz=6P6@0$C~_wO+=l%wk_j8SYa z8Y5JbJWKn01-|zT1_J(JJ4gG^F-XVR-@Co2PzZIN;H^%smW_J#>=uj9TQ1&{5cK`4 z^j=z^c(HnUzV|+i5o zz#?LMSmO!?49^QZ4{gukVEKPW2-RV2i(+R?wavS}!e$8KQw%YM^V$*6UF1q?V+T-v zTI+h+=(&wT<&y4?ui3E!MaZ9G5(~ph6dz%@-TRj7JY zFwj~WUgP^~>E}IyDJ{}F2KtG1)YhmPou|()ZTK-7nKZ4D1e*vpAr}wFQ^mJz7o3HM zUE!tYj3o*J0l$pM$EPU`f}UgZl%LJrs9oPE`ytxqMJzT_cobIf+2n>5fW~-{_Kk}S zk`aQE>n;^WLO1+#{xj~np{Rypth2i+P`zgE*2Q>;CG?RZjAo3-<97#*Dmwe(0hxL- z@tjtZ#SB-5{qEMlrgtEc4sTK6>tW)|wGpM6sYBP%Q{`fN*94Ce8!WN_EAQ#tNv;5G z${wvoOU0T;EP>Ibr}+EfU5Y)_D^!r67s(G;IFe0^^0(u+-;o#d69ly~$*kM0oJsr^ zv^i|e2m80wOryT1Vse8MuNd);(sPrN7~c}f3D3%W2+IdcB5iehFRMyb4XPiv#C;pq zEeE1EX1X@K$uuI~Gw@0E6PI+~W>nB15ECWsmSn0m$R|4g7U5Tr9oLrlb>44OcPxep zrLS-b(+MHfx_a1jk8fl+j?RHQRZ60)i7*l1o&>g9u}{TF8Inrskhbe4+0y0Wa^IlS zk|sFs^)46)pNRk?YR1VEg1vZ7Sek$OS@1inQQ>@7d-Ae=`d$OMj&a+c=b9yese*#a z%}AR8{lm6hkZiDRINN6k4;!Spvqgwhk;dGbLSAVa`HdmT!wg>pr47p`SGxX99k0<$ z%!bu*ew0M$12>4=?iJ4?m$^`a-Hs9i0>$E|x(`i;RLACA1x}ml`K(&~eSq0sqK9L2H7t5lUw5(XT288^=62<;234iZ%r*?^GZ=e3VvkLye#gYWx=*)CUP z5i`Q8o4){M9huLPq+d_hlh!#I6UZ_ zOF}9uHwh1%nCE&CMv}?G3be34xYpQ`(5D{X*7@ZzF#Se67i#s?^3P$EH5zLERoX3l zzpkq<9-S2VJra)d>nV`tl3CQh@ciLubr74; zaE6(HKFC?4|Az&;T3L7JKq2P=cvt*jK1Nyk(FJt5v@z7Vs|k8=eE`k zF9n;PUT09i0v~gvJSV=B%(y9IqRygBTZs5@>|Bj>_~&eLiW@=w$D2QI2Q+QqP1Ilh zMlym%x0hdd!%J+uWK=k3w=!EwVZ#G9@ChqSraDTFJgo4VwqO2Wc8SF}J&s(w_;nM5%gZ;`i6FxVnMg|r% z!rzv1<_p&~28awMl3pT$427E5(&m0poxn=wB-DSjD8o>p#JU*G2nMP(lFlQJBbwCf z2!W#~_HZ7xfqYV}Wq^q2h}C;>-g40s2W_%}RyW?a?u&p2$$Bj^Fd*qo42K#y->E*Q z{s1|2r=cjD@ODQ05u1mV&2Ob&H2^otR|O<4Tb&Uj4+zMS_GyVMLAFRuVH1ai2U)wW zZwDpHzUeY}6EnLI9D;PLqQ?X>RU_1Qa@7um_ox=g&;+vWyx`(_q( zaVZsIM>CwY?RAF?Rx95`_EiJ#gXQUgd1*gMIa|4Eg9O7+_SgCbRd?^ZGOqe~NB zB#^xT001s{Y%R>kGrbpp_zNJ`pPsl7XX9Toj08b}9S9Xy9Sw7U3c*n)2w8rGoYLyM zbqE&oZG(i4UcSrtO(W8?&i7N&p`fYT^S(CX36r3mgQoIWL1i1V`H_^5i-Vgs@+u9O z%7q6(b}k1+yz_eTJ9~#w2Za2n_q0Jy-T2>_Vfnx!EIANDvHuM{1Q^I zH0Ce&{)Z&AWF*_a1`?ju765>Z3@rfqYXk{E1pxzqd~yF=xF8V!>NUQ2y#Me9Up)1H zbpCmRKm+|hed`GDf9Zc2{1-`owg1@ppB@qbh=&4ze@U2GS(rf*{>_7YSpek!N6x~` d43htEi$LJlD^>o#Ee3z7fc=M0|HtGP{XdG!!#V%} literal 0 HcmV?d00001 diff --git a/site/content/blog/protobuf-immutability/index.md b/site/content/blog/protobuf-immutability/index.md index 7fd9d273..cec35ca1 100644 --- a/site/content/blog/protobuf-immutability/index.md +++ b/site/content/blog/protobuf-immutability/index.md @@ -1,5 +1,5 @@ --- -title: Protobuf — Serialization and Beyond. +title: Protobuf — serialization and bbeyond. Part 2. Immutability description: >- In this chapter, we are going to address the elephant in the room which comes with Protobuf in Java in particular — immutable data types. diff --git a/site/content/blog/protobuf-value-objects/index.md b/site/content/blog/protobuf-value-objects/index.md index 2bec0fdb..6a55b503 100644 --- a/site/content/blog/protobuf-value-objects/index.md +++ b/site/content/blog/protobuf-value-objects/index.md @@ -12,7 +12,7 @@ header_type: fixed-header We have been using Protobuf as more than just a serialization mechanism from the very beginning. It provides a language to model the domain. This ability is especially useful in projects based on the [Domain-Driven Design](https://martinfowler.com/bliki/DomainDrivenDesign.html) methodology, aka DDD. -The defining feature of a Domain Model is the [Ubiquitous Language](https://martinfowler.com/bliki/UbiquitousLanguage.htmls) — a language that is used and agreed upon throughout the domain: by the domain experts, programmers, and in the code. A Domain Model consists of many components, but the language lays the base for all of them. Value Objects help up transform the language into code. +The defining feature of a Domain Model is the [Ubiquitous Language](https://martinfowler.com/bliki/UbiquitousLanguage.html) — a language that is used and agreed upon throughout the domain: by the domain experts, programmers, and in the code. A Domain Model consists of many components, but the language lays the base for all of them. Value Objects help up transform the language into code. ## What are Value Objects? A Value Object is a simple type that represents any kind of logically indivisible domain information, such as `EmailAddress`, `PhoneNumber`, `LocalDateTime`, `Money`, etc. Unlike Entities, Value Objects are only identified by the value itself, i.e. they do not have a designated ID. These objects are immutable and, typically, not too big. Apart from the encapsulation of the data itself, such types can hold the validation logic, basic operations on the data, the string representation, etc. For example, the type `LocalDateTime` may provide methods such as `add(Duration)` which calculates the local date and time after a given duration passes. The result is another instance of `LocalDateTime`, while the original object is unchanged. diff --git a/site/content/blog/protobuf-whats-in-it-for-your-project/index.md b/site/content/blog/protobuf-whats-in-it-for-your-project/index.md index ff6d23a2..cec5b828 100644 --- a/site/content/blog/protobuf-whats-in-it-for-your-project/index.md +++ b/site/content/blog/protobuf-whats-in-it-for-your-project/index.md @@ -1,6 +1,6 @@ --- -title: Protobuf — Serialization and Beyond. - Part 1. What’s in It for Your Project +title: Protobuf — serialization and beyond. + Part 1. What’s in it for your project description: >- In this article we look at the key Google Protocol Buffers (aka Protobuf) features which make working with data in software development projects more convenient and effective. author: Alexander Yevsyukov and Dmytro Dashenkov @@ -9,7 +9,7 @@ type: blog/post header_type: fixed-header --- -*This six-part series covers using Google Protocol Buffers (aka [Protobuf](https://developers.google.com/protocol-buffers)) to build software Domain-Driven Design way and to cut on the manual effort when migrating data and syncing interaction protocols in complex systems.* +*This series covers using Google Protocol Buffers (aka [Protobuf](https://developers.google.com/protocol-buffers)) to build software Domain-Driven Design way and to cut on the manual effort when migrating data and syncing interaction protocols in complex systems.* *In the first part, we look at the key Protobuf features which make working with data in software development projects more convenient and effective.* From e5e83b4a59d5a76af141f8cbd9af73b07407146b Mon Sep 17 00:00:00 2001 From: oleksandra-kovalenko Date: Wed, 7 Jan 2026 05:38:56 +0200 Subject: [PATCH 06/13] Fixed broken URS. --- site/content/blog/protobuf-data-validation/index.md | 6 +++--- site/content/blog/protobuf-immutability/index.md | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/site/content/blog/protobuf-data-validation/index.md b/site/content/blog/protobuf-data-validation/index.md index cd4da42e..b8f0524e 100644 --- a/site/content/blog/protobuf-data-validation/index.md +++ b/site/content/blog/protobuf-data-validation/index.md @@ -11,7 +11,7 @@ header_type: fixed-header *In this series, we explore the features and advantages of using Protocol Buffers as a model-building tool. In this part, we take a look at how to validate data with Protobuf and what can be learned from the mechanisms behind data validation.* ## What is Validation? -When developing a domain model, one often faces a need to enforce certain rules upon the data objects. Strongly-typed languages, such as Java, help us order up data into neat structures and then build [Value Objects](https://medium.com/teamdev-engineering/protobuf-serialization-and-beyond-part-3-value-objects-e3dc7b935ac) upon those structures. However, none of the programming languages is expressive enough to form the whole model and enforce all the known rules. This is not a failure of the language designers, but a required tradeoff for creating any sort of a general-purpose programming language. +When developing a domain model, one often faces a need to enforce certain rules upon the data objects. Strongly-typed languages, such as Java, help us order up data into neat structures and then build [Value Objects](blog/protobuf-value-objects/) upon those structures. However, none of the programming languages is expressive enough to form the whole model and enforce all the known rules. This is not a failure of the language designers, but a required tradeoff for creating any sort of a general-purpose programming language. If not solved in a general way, the need to check if the data fits the domain rules leads to conditional statements and exceptions scattered across the codebase. Such a chaotic approach allows errors to pop up once in a while, rendering the system unreliable. @@ -249,10 +249,10 @@ final class When extends FieldValidatingOption { } ``` -The class `WhenFactory` has to be exposed to the Java `ServiceLoader` mechanism as an implementation of [ValidatingOptionFactory](https://spine.io/base/reference/base/io/spine/validate/option/ValidatingOptionFactory.html) either manually or via an automatic tool, such as [AutoService](https://github.com/google/auto/tree/master/service). +The class `WhenFactory` has to be exposed to the Java `ServiceLoader` mechanism as an implementation of [ValidatingOptionFactory](https://spine.io/base-libraries/dokka-reference/base/base/io.spine.validate.option/-validating-option-factory/index.html) either manually or via an automatic tool, such as [AutoService](https://github.com/google/auto/tree/master/service). Here is the diagram of the classes we’ve just described. -![The diagram of the classes](site/content/blog/protobbuf-data-validation/the-diagram-of-the-classes.webp) +![The diagram of the classes](site/content/blog/protobuf-data-validation/the-diagram-of-the-classes.webp) We expose the `WhenFactory`, which implements `theValidatingOptionFactory`, via the `ServiceLoader` API. `WhenFactory` registers the `When` option, which, when necessary, creates a `WhenConstraint` based on a field definition. diff --git a/site/content/blog/protobuf-immutability/index.md b/site/content/blog/protobuf-immutability/index.md index cec35ca1..00bd45de 100644 --- a/site/content/blog/protobuf-immutability/index.md +++ b/site/content/blog/protobuf-immutability/index.md @@ -62,11 +62,11 @@ The way Protobuf generates code represents a typical immutable API in Java, excl However, an immutable data type can be of any shape and form you want, or, to be precise, any shape and form the API user needs. For example, types such as `java.lang.String`, `java.io.File`, `java.time.LocalDateTime`, etc. are all immutable data types. ## What About Other Platforms? -Unfortunately, Protobuf does not generate immutable code for JavaScript and Dart. In fact, some aspects of the message objects in JS are mutable while others [are not](https://developers.google.com/protocol-buffers/docs/reference/javascript-generated#repeated). +Unfortunately, Protobuf does not generate immutable code for JavaScript and Dart. In fact, some aspects of the message objects in JS are mutable while others [are not](https://protobuf.dev/protobuf-javascript/#repeated-fields). Such a difference between Java and client-side languages can be explained by a desire to make client-specific Protobuf objects as lightweight and performant as possible. However, is the tradeoff really worth it in the world with more and more efficient computers? In our opinion, it is not. -Nevertheless, in Dart, there are a few tools, as well as [language structures](https://dart.academy/immutable-data-patterns-in-dart-and-flutter/), which can help you with immutability. For instance, `built_value` and `freezed` are two libraries that allow their users to create immutable data types with [almost no extra effort](https://levelup.gitconnected.com/flutter-dart-immutable-objects-and-values-5e321c4c654e). But in order to achieve immutability for data types that originate in Protobuf, we’d have to arrange the conversion from mutable to immutable types, which is a lot of manual work. And in the end, we’d get two classes for one Protobuf type. This is a suboptimal solution, to say the least. +Nevertheless, in Dart, there are a few tools, as well as language structures, which can help you with immutability. For instance, `built_value` and `freezed` are two libraries that allow their users to create immutable data types with [almost no extra effort](https://levelup.gitconnected.com/flutter-dart-immutable-objects-and-values-5e321c4c654e). But in order to achieve immutability for data types that originate in Protobuf, we’d have to arrange the conversion from mutable to immutable types, which is a lot of manual work. And in the end, we’d get two classes for one Protobuf type. This is a suboptimal solution, to say the least. We have tried creating a symbiosis between `built_value` and Protobuf to achieve immutable “front-facing” types, which would rely on standard Protobuf Dart code for serialization. This attempt has failed as we have found ourselves juggling too many noncoherent abstractions. At the time of writing, we [plan](https://github.com/SpineEventEngine/core-java/issues/1334) to implement custom code generation to support Dart, JavaScript, and Java, which addresses the limitations we face and provides immutability. From 423f307b120ecf5998ccd58d710e9458dd7a488a Mon Sep 17 00:00:00 2001 From: "julia.evseeva" Date: Mon, 12 Jan 2026 12:26:40 +0100 Subject: [PATCH 07/13] Use `proto` instead of `.proto` as a code language --- site/content/blog/protobuf-data-validation/index.md | 12 ++++++------ site/content/blog/protobuf-value-objects/index.md | 8 ++++---- .../protobuf-whats-in-it-for-your-project/index.md | 8 ++++---- 3 files changed, 14 insertions(+), 14 deletions(-) diff --git a/site/content/blog/protobuf-data-validation/index.md b/site/content/blog/protobuf-data-validation/index.md index b8f0524e..03a3eb31 100644 --- a/site/content/blog/protobuf-data-validation/index.md +++ b/site/content/blog/protobuf-data-validation/index.md @@ -22,7 +22,7 @@ A well-integrated validation mechanism sieves off corrupted (or, often, mistyped When talking about validation, we mean the integrity of the individual objects, rather than the whole system. Consider a user account model: {{% /note-block %}} -```.proto +```proto message User { // ... @@ -80,7 +80,7 @@ Protobuf provides options as a native way to declare the meta-data for each file Consider the following message: -```.proto +```proto message LocalDate { int32 year = 1; Month month = 2; @@ -90,7 +90,7 @@ message LocalDate { It’s up to users to define their custom options and then provide some mechanism to deal with them. On top of this feature, we have built an infrastructure for validating Protobuf messages. Validation rules are defined using custom options for fields and messages: -```.proto +```proto message LocalDate { int32 year = 1; Month month = 2 [(required) = true]; @@ -141,7 +141,7 @@ Users of our Validation library can also extend the standard set of options with Consider the following message: -```.proto +```proto message Project { // ... ZonedDateTime when_created = 4 [(required) = true]; @@ -150,7 +150,7 @@ message Project { The `when_created` field stores the info about the timestamp of the project creation. It is true that the field must be set, i.e. it always exists in the domain, hence an absence of this value is a technical error. And there’s more to it. We’ll assume that the described project already exists in real life. So, its creation time has to be in the past from now. Let’s introduce an option to signify that. To do so, we extend the standard set of field options with a new one: -```.proto +```proto import “google/protobuf/descriptor.proto”; extend google.protobuf.FieldOptions { @@ -178,7 +178,7 @@ The field number for the `when` option is 73819. It is due to how Protobuf distr Now, we just use the option in the domain: -```.proto +```proto import “example/validation_options.proto”; diff --git a/site/content/blog/protobuf-value-objects/index.md b/site/content/blog/protobuf-value-objects/index.md index 6a55b503..e347121f 100644 --- a/site/content/blog/protobuf-value-objects/index.md +++ b/site/content/blog/protobuf-value-objects/index.md @@ -25,7 +25,7 @@ Creating Value Objects in Protobuf is convenient because: 1. The domain types are created fast and for the many target languages, officially [supported by Google](https://developers.google.com/protocol-buffers/docs/tutorials) and [third-party](https://chromium.googlesource.com/chromium/src/+/master/third_party/protobuf/docs/third_party.md). Let’s see how the Value Object can be implemented in Protobuf. -```.proto +```proto message EmailAddress { string value = 1 [ (required) = true, @@ -54,7 +54,7 @@ An important part of the Value Object is its behavior. OOP greatly influences th In Java, Protobuf generates non-extensible classes, which makes it hard to add behavior to them. We have solved this problem by defining the option (is). It takes the names of Java interfaces with which we want to mark the Protobuf message. Such interfaces may include default methods, adding behavior to the Value Object. Our custom Protobuf compiler plugin modifies the generated code so that the Proto-types implement the assigned interfaces. Here is how this works. -``` .proto +```proto message User { option (is).java_type = "UserMixin"; UserId id = 1 [(required) = true]; @@ -106,7 +106,7 @@ completeOrder(UserId u, CustomerId c, OrderId o); Another benefit to type-safe identifiers lies in their structure. In the basic case, an ID type for an entity looks as follows: -```.proto +```proto message CustomerId { string uuid = 1; } @@ -114,7 +114,7 @@ string uuid = 1; However, by hiding the type of identifier implementation, we gain the ability to expand it if necessary. For example, to integrate the data of different vendors. We can do it with `oneof` construction: -```.proto +```proto message CustomerId { oneof kind { uint64 code = 1; diff --git a/site/content/blog/protobuf-whats-in-it-for-your-project/index.md b/site/content/blog/protobuf-whats-in-it-for-your-project/index.md index cec5b828..4663da23 100644 --- a/site/content/blog/protobuf-whats-in-it-for-your-project/index.md +++ b/site/content/blog/protobuf-whats-in-it-for-your-project/index.md @@ -61,7 +61,7 @@ Protobuf employs the binary serialization mechanism to transfer data between the Let’s assume we have a data type `Task` defined as follows: -```.proto +```proto message Task { string name = 1; string description = 2; @@ -123,7 +123,7 @@ To avoid unpleasant surprises we recommend the following cycle to process the de **Step 1.** Mark the field with the `deprecated` option: -```.proto +```proto message MyMessage { ... int32 old_field = 6 [deprecated = true]; @@ -135,8 +135,8 @@ message MyMessage { **Step 4.** Delete the field, marking its index and name as `reserved`: - ```.proto - message MyMessage { +```proto +message MyMessage { ... reserved 6; reserved “old_field”; From 42540ba16c7895e8077c806f570f65373452b2f8 Mon Sep 17 00:00:00 2001 From: "julia.evseeva" Date: Mon, 12 Jan 2026 12:30:36 +0100 Subject: [PATCH 08/13] Fix URLs --- site/content/blog/protobuf-data-validation/index.md | 2 +- site/content/blog/protobuf-value-objects/index.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/site/content/blog/protobuf-data-validation/index.md b/site/content/blog/protobuf-data-validation/index.md index 03a3eb31..879d3ad9 100644 --- a/site/content/blog/protobuf-data-validation/index.md +++ b/site/content/blog/protobuf-data-validation/index.md @@ -252,7 +252,7 @@ final class When extends FieldValidatingOption { The class `WhenFactory` has to be exposed to the Java `ServiceLoader` mechanism as an implementation of [ValidatingOptionFactory](https://spine.io/base-libraries/dokka-reference/base/base/io.spine.validate.option/-validating-option-factory/index.html) either manually or via an automatic tool, such as [AutoService](https://github.com/google/auto/tree/master/service). Here is the diagram of the classes we’ve just described. -![The diagram of the classes](site/content/blog/protobuf-data-validation/the-diagram-of-the-classes.webp) +![The diagram of the classes](the-diagram-of-the-classes.webp) We expose the `WhenFactory`, which implements `theValidatingOptionFactory`, via the `ServiceLoader` API. `WhenFactory` registers the `When` option, which, when necessary, creates a `WhenConstraint` based on a field definition. diff --git a/site/content/blog/protobuf-value-objects/index.md b/site/content/blog/protobuf-value-objects/index.md index e347121f..f5b75ba7 100644 --- a/site/content/blog/protobuf-value-objects/index.md +++ b/site/content/blog/protobuf-value-objects/index.md @@ -40,7 +40,7 @@ Note that the `(required)` and `(pattern)` options are extensions to Protobuf pr In addition, in some target languages, such as Java, generated Protobuf types are immutable by default. Unfortunately, some other languages, such as JavaScript and Dart, only support [mutable types](https://github.com/dart-lang/protobuf/issues/413). For Dart, however, a community-driven solution seems to be on the way. The [immutable_proto](https://pub.dev/packages/immutable_proto) package implements code generation for immutable types from Protobuf. We have not tried it out yet, as the library is still in its earliest form, but the notion that other engineers feel our pain and try to do something about it as well warms our hearts. -For further reading on immutability with regards to Protobuf, see our [previous article](blog/immutability/). +For further reading on immutability with regards to Protobuf, see our [previous article](blog/protobuf-immutability/). ## Validation The `EmailAddress` type as declared above has one string field with naive validation via a regular expression. Also, the value field should be filled. This validation API is a part of our efforts on improving the code generation with Protobuf. Right now, we generate validation code for Java and Dart, in order to cover both backend and frontend. Later, other target languages might join the club. From 82a8a747e300355b3264548c858e070f399cd3d2 Mon Sep 17 00:00:00 2001 From: "julia.evseeva" Date: Mon, 12 Jan 2026 12:34:08 +0100 Subject: [PATCH 09/13] Clean up alignment and add EOF --- .../blog/protobuf-data-validation/index.md | 24 +++++++++----- .../blog/protobuf-immutability/index.md | 17 +++++++--- .../blog/protobuf-value-objects/index.md | 18 +++++++---- .../index.md | 31 ++++++++++++++----- 4 files changed, 64 insertions(+), 26 deletions(-) diff --git a/site/content/blog/protobuf-data-validation/index.md b/site/content/blog/protobuf-data-validation/index.md index 879d3ad9..169ee8bd 100644 --- a/site/content/blog/protobuf-data-validation/index.md +++ b/site/content/blog/protobuf-data-validation/index.md @@ -1,21 +1,24 @@ --- -title: Protobuf — serialization and beyond. - Part 4. Data validation +title: Protobuf — serialization and beyond. Part 4. Data validation description: >- - In this part, we take a look at how to validate data with Protobuf and what can be learned from the mechanisms behind data validation. + In this part, we take a look at how to validate data with Protobuf and what + can be learned from the mechanisms behind data validation. author: Alexander Yevsyukov and Dmytro Dashenkov publishdate: 2021-10-06 type: blog/post header_type: fixed-header --- + *In this series, we explore the features and advantages of using Protocol Buffers as a model-building tool. In this part, we take a look at how to validate data with Protobuf and what can be learned from the mechanisms behind data validation.* ## What is Validation? -When developing a domain model, one often faces a need to enforce certain rules upon the data objects. Strongly-typed languages, such as Java, help us order up data into neat structures and then build [Value Objects](blog/protobuf-value-objects/) upon those structures. However, none of the programming languages is expressive enough to form the whole model and enforce all the known rules. This is not a failure of the language designers, but a required tradeoff for creating any sort of a general-purpose programming language. + +When developing a domain model, one often faces a need to enforce certain rules upon the data objects. Strongly-typed languages, such as Java, help us order up data into neat structures and then build [Value Objects](blog/protobuf-value-objects/) upon those structures. However, none of the programming languages is expressive enough to form the whole model and enforce all the known rules. This is not a failure of the language designers, but a required tradeoff for creating any sort of general-purpose programming language. If not solved in a general way, the need to check if the data fits the domain rules leads to conditional statements and exceptions scattered across the codebase. Such a chaotic approach allows errors to pop up once in a while, rendering the system unreliable. ### Validation for the rescue + A well-integrated validation mechanism sieves off corrupted (or, often, mistyped) data early in the flow and gives the developers a safety net against small errors. {{% note-block class="note" %}} @@ -36,6 +39,7 @@ Checking if the `email` is set and is a valid email address is a part of the dat Validation solutions come in many shapes and sizes, but all of them share a common structure. Validation rules, a.k.a. validation constraints, define the axiomatic facts about the data in the system. An evaluator for the constraints checks input data and reports any violations. Typically, the constraints are defined in a declarative DSL, while the evaluator is separated from the constraints for simplicity. ## Java Validation Framework + In Java, there is the Bean Validation 2.0, also known as [JSR 380](https://beanvalidation.org/). This JSR is a specification that constitutes a set of validation annotations, which allow the programmers to declare validation constraints, and an API for manual invocation of the checks. For example, consider a user account object implemented in Java. ```java @@ -62,9 +66,11 @@ class User { Such an annotation-based approach gives the API users an ability to choose between explicit invocation of the validation process via the runtime API and implicit invocation via manipulating bytecode and inserting corresponding checks right along with the user code. ## Validation with Protobuf + We work with Protobuf, a multi-language tool for defining data models. As we use Protobuf for domain modeling, we require a robust built-in mechanism for validation. Unfortunately, Protobuf itself does not supply one. So, we’ve built our own. ### Why develop our own validation? + {{% note-block class="note" %}} “Everybody has a logging framework, but mine will be better.” — a naïve software engineer. {{% /note-block %}} @@ -74,6 +80,7 @@ Indeed, developing something as fundamental as a validation library might seem a There are indeed other validation solutions for Protobuf. Many of them are similar to what we do. Some, just like us, generate validation code. Others rely on runtime validators. What makes the difference is the smooth integration with the generated code itself. ### How it works + First things first. The features described in this article apply primarily to the generated Java code. Some features are also implemented for Dart. In the future, we are planning to also cover other languages, such as JavaScript and C/C++. Fortunately, the cost of adding another language is lower than the cost of developing the whole validation tool from the ground up. Protobuf provides options as a native way to declare the meta-data for each file, message, or field. The options API resembles the JSR 380 annotation API. Such cohesion allows developers familiar with the Java API to get a grip of our options API faster. @@ -98,7 +105,7 @@ message LocalDate { } ``` -This example is rather simplistic, as, for example, the max number of days depends on the month and year values in real life. We ignore this fact for now, since there is no easy and robust way of adding complex logic code to Protobuf definitions. In future, we are planning to allow more complicated scenarios for validation rules. We, however, have not figured out the syntax for such constructs yet. +This example is rather simplistic, as, for example, the max number of days depends on the month and year values in real life. We ignore this fact for now, since there is no easy and robust way of adding complex logic code to Protobuf definitions. In the future, we are planning to allow more complicated scenarios for validation rules. We, however, have not figured out the syntax for such constructs yet. Our options, such as `(required)`, `(pattern)`, `(min)`, `(max)`, etc., are transpiled into executable code during Protobuf compilation. We embed this validation code directly into the code generated by the Protobuf compiler for the target language. @@ -175,7 +182,6 @@ message WhenOption { The field number for the `when` option is 73819. It is due to how Protobuf distributes field numbers and extensions that we have to use such obscure constants. Read more about field numbers for extension types in [this guide](https://developers.google.com/protocol-buffers/docs/proto#choosing_extension_numbers) and [this comment](https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/descriptor.proto#L311) in `google/protobuf/descriptor.proto`. {{% /note-block %}} - Now, we just use the option in the domain: ```proto @@ -263,6 +269,7 @@ The `(when)` option is already a part of the Spine Validation library. Users don {{% /note-block %}} ### Protobuf `required`: Il buono, il brutto, il cattivo + A reader familiar with the differences between Protobuf v2 and v3 may have noticed and been surprised by the `required` term in the example defining the `Project` type shown above. In Protobuf version 2, any field could be marked as `required` or `optional`. Later, the library creators [declared](https://github.com/protocolbuffers/protobuf/issues/2497) this approach to be more harmful than good. In Protobuf version 3 and its [analogs](https://capnproto.org/faq.html#how-do-i-make-a-field-required-like-in-protocol-buffers), all fields are always regarded as optional. The reason for this is binary compatibility. The `required` field in Protobuf 2 cannot be made `optional` and vice versa without breaking compatibility with previous versions of the message. For a serialization and data transfer tool, this is a big issue, as different components that communicate with each other using Protobuf messages may use different versions of those messages. Breaking binary compatibility means breaking such communications. @@ -277,7 +284,6 @@ For example, in Java, to create a non-validated message, we use the `buildPartia A technical note. In Protobuf 2, all fields had to be declared as either `required` or `optional`. In Protobuf 3, all fields are `optional`. In the fresh Protobuf 3, in [v3.15](https://github.com/protocolbuffers/protobuf/releases/tag/v3.15.0) to be precise, the keyword `optional` was brought back. Though, the semantics of this keyword is different this time around. The Protobuf 3 `optional` allows users to check if the field is set, even if it’s a number of a `bool` field. Not to be confused with ye olde optional fields of Protobuf 2. {{% /note-block %}} - ## Proto Reflection Under the hood, our Validation library uses the Protobuf reflection API in order to obtain the message metadata. When a `.proto` file is compiled into target languages, the compiler exposes the metadata in a form of Protobuf messages known as descriptors. Descriptors contain information about the entire scope of the Protobuf definitions, from message fields to the documentation. It also includes the options defined on the messages and fields. @@ -290,6 +296,7 @@ Descriptor type = MyType.getDescriptor(); Unfortunately, there is no analogous API in other target languages, such as JavaScript and Dart. Fortunately, the descriptors are always available at compile time. Users are welcome to add their own [Protobuf compiler plugins](https://developers.google.com/protocol-buffers/docs/reference/other#plugins) to access the descriptors and generate code based on them. ## More on Code Generation + As of the time of writing, we are working on a new mechanism for code generation, which will also change the internals of how we generate validation code. We thought we might share it here. The new tool we’re working on is called [ProtoData](https://github.com/SpineEventEngine/ProtoData/). It allows reducing the effort of generating code for multiple platforms. Descriptors in the Protobuf compiler are a language-agnostic intermediate representation of the Proto definitions. For our code generation, we also build a language-agnostic model, based on the Protobuf definitions. And then feed those representations to multiple language-specific renderers, which turn them into code. @@ -299,10 +306,11 @@ The whole tool is built on an event-driven architecture, which allows users to t Right now, we’re approaching the first public release and an API freeze for the tool. If you would like to explore it, visit the [GitHub repo](https://github.com/SpineEventEngine/ProtoData/). ## Conclusion + Protobuf provides a great variety of choices for how to use it. In its simplest form, Protobuf is a serialization format. But due to the systematic and future-proof approach used by the designers of the technology, it has become much more than that. Protobuf reflection API, which is, originally based on Protobuf types itself, allows users to bend and stretch the technology to great extent. Access to the meta-data and entry points for quick code generation enabled us to create an entire validation library based on Protobuf definitions without a need to parse the definitions on our own or to do heavy operations on metadata at runtime. A validation library is just another step towards an expressive and resilient domain model. And with our Validation library and Protobuf, we make this step a commodity. -In the next part of this series, we will build on top of what we have already learned and tried by looking at domain modeling. We will explore different kinds of messages, see how validation integrates into the information flow, and discover the hidden benefits of using Protobuf for a domain model. \ No newline at end of file +In the next part of this series, we will build on top of what we have already learned and tried by looking at domain modeling. We will explore different kinds of messages, see how validation integrates into the information flow, and discover the hidden benefits of using Protobuf for a domain model. diff --git a/site/content/blog/protobuf-immutability/index.md b/site/content/blog/protobuf-immutability/index.md index 00bd45de..fbdc7c7b 100644 --- a/site/content/blog/protobuf-immutability/index.md +++ b/site/content/blog/protobuf-immutability/index.md @@ -1,27 +1,32 @@ --- -title: Protobuf — serialization and bbeyond. - Part 2. Immutability +title: Protobuf — serialization and beyond. Part 2. Immutability description: >- - In this chapter, we are going to address the elephant in the room which comes with Protobuf in Java in particular — immutable data types. + In this chapter, we are going to address the elephant in the room which comes + with Protobuf in Java in particular — immutable data types. author: Alexander Yevsyukov and Dmytro Dashenkov publishdate: 2020-12-02 type: blog/post header_type: fixed-header --- + *In [part 1](blog/protobuf-whats-in-it-for-your-project/) of this series, we have shared how we discovered Protobuf, discussed its advantages over the prior art, and described what it takes to standardize data types in a complex multi-platform software system. In this chapter, we are going to address the elephant in the room which comes with Protobuf in Java in particular — immutable data types.* ## To Change or Not to Change + When first facing the idea of immutable types, one might say: “This will never work! Not only is the concept overcomplicated, but it also leads to inferior performance”. We can sympathize with this emotion. When forced to write extra code to copy the value of an object just to introduce a single change, one should seriously question their goals. If you find yourself asking this question, this article is for you. First things first, let’s go through the motivation behind forbidding to change stuff and why has this concept been [gaining traction](https://elizarov.medium.com/immutability-we-can-afford-10c0dcb8351d) lately. ### Consistency + Stability feels nice. We function better in a well-known environment. So, we use abstractions and analogies, to make the object model as close as we can to the real world we’re used to. An immutable object has much more in common with an item in real life than a mutable one does. Imagine that you are holding a book. On the other side of the planet, the author decides that they don’t like chapter 5 anymore, so they delete it. And your book gets a bit thinner. This is, at best, a nuisance. It would be fair to sue the author for robbing you of the part of the book. Even electronic copies are never updated like that. Instead, a book can only be updated in a single straightforward way: a copy is made, and changes are made to the new copy before distributing it. This is called a new edition. Why do we like our books to stay the same and never “upgrade” them, even if, like in the case with an e-book, it would be relatively easy to do? Well, it makes it convenient to refer to the book in the future. If we’d like to discuss chapter 5 with our colleagues, we would rather be talking about the same thing. Otherwise, we would get an aliasing bug, which is nicely described in this [blog post](https://martinfowler.com/bliki/AliasingBug.html) by Martin Fowler. ### Asynchronous Safety + Another feature of immutable types — safety when working in an asynchronous environment. This gets mentioned a lot in many technical articles, as well as software developer job interviews. The gist is as follows: atomic operations on references instead of the object’s fields are the only way to ensure the integrity of shared data. This gives us control over how the data is changed and how to resolve conflicting changes. ### In-memory Storage + More often than not, software, on both server and client sides, uses some kind of in-memory storage for data. Most likely, data structures used for that rely on objects’ hash codes and equality. Such structures are a gold standard for most platforms. For example, in Java, `HashMap` and `HashSet` will not function properly if elements placed in the map have no consistent `hashCode()` and `equals()` implementations. For a mutable type, it is impossible to provide such implementations. Consider a mutable `User` class, which is stored in a `HashSet`. At some point, we may need to iterate through all the users in the set and change each second one of them. After such a mutation, the set is full of objects which cannot be discovered by the `contains()` method anymore, since the hash of the original object is not the same as the hash of the new object. @@ -29,6 +34,7 @@ For a mutable type, it is impossible to provide such implementations. Consider a Such a problem would never occur with an immutable object. If fields never change, hash code won’t change either. ## Is Mutating Things Still OK? + Absolutely. There are many cases when immutability has too high of a price. The biggest reason to use mutable types over immutable ones is the cost of copying data. At the end of the day, data in a system needs to be changed over time. Sometimes, this change happens very often. For example, creating an extra `Builder` object and a new copy of the original object make up for thrice total object references in memory over a single conventional mutable object. One should also consider the CPU time taken to create a fresh copy of the original object. @@ -40,6 +46,7 @@ Another example of a system that could benefit from data mutability is an automa Developers should use their best judgment when considering immutability. If the tradeoff is too high, it is OK not to go all-in on immutability and use it only in cases where the performance restrictions are less harsh. ## Java API With Protobuf + In Java, generated Protobuf classes are always immutable. Each class has a nested `Builder` class, which has all the same fields but is mutable. To create an instance of such a class, we create a builder, assign the fields, and build the message. ```java @@ -62,6 +69,7 @@ The way Protobuf generates code represents a typical immutable API in Java, excl However, an immutable data type can be of any shape and form you want, or, to be precise, any shape and form the API user needs. For example, types such as `java.lang.String`, `java.io.File`, `java.time.LocalDateTime`, etc. are all immutable data types. ## What About Other Platforms? + Unfortunately, Protobuf does not generate immutable code for JavaScript and Dart. In fact, some aspects of the message objects in JS are mutable while others [are not](https://protobuf.dev/protobuf-javascript/#repeated-fields). Such a difference between Java and client-side languages can be explained by a desire to make client-specific Protobuf objects as lightweight and performant as possible. However, is the tradeoff really worth it in the world with more and more efficient computers? In our opinion, it is not. @@ -71,8 +79,9 @@ Nevertheless, in Dart, there are a few tools, as well as language structures, wh We have tried creating a symbiosis between `built_value` and Protobuf to achieve immutable “front-facing” types, which would rely on standard Protobuf Dart code for serialization. This attempt has failed as we have found ourselves juggling too many noncoherent abstractions. At the time of writing, we [plan](https://github.com/SpineEventEngine/core-java/issues/1334) to implement custom code generation to support Dart, JavaScript, and Java, which addresses the limitations we face and provides immutability. ## What’s Next? + To summarize, immutability is a helpful concept for many reasons. Firstly, it helps people understand the lifecycle of an object by building direct analogies with items from the real world. Secondly, it has technical advantages, such as clearer relations with data structures and asynchronous safety. Although there are cases when mutable types are still preferable for performance reasons, most systems may enjoy the benefits of immutability without extra concerns. In Java, Protobuf helps us create immutable data types with ease by generating all the code required for building and copying objects. There are also tools that can be used for this task in other languages, such as Dart. -Simplicity and technical advantages aside, immutable data types are important for a greater reason. They assist us in building projects based on Domain-Driven Design. In the following parts of this series, we are going to talk about DDD and how Protobuf helps us build Domain Models, starting with Value Objects, a simple yet powerful pattern for translating domain concepts into the machine language. \ No newline at end of file +Simplicity and technical advantages aside, immutable data types are important for a greater reason. They assist us in building projects based on Domain-Driven Design. In the following parts of this series, we are going to talk about DDD and how Protobuf helps us build Domain Models, starting with Value Objects, a simple yet powerful pattern for translating domain concepts into the machine language. diff --git a/site/content/blog/protobuf-value-objects/index.md b/site/content/blog/protobuf-value-objects/index.md index f5b75ba7..8d65c6cc 100644 --- a/site/content/blog/protobuf-value-objects/index.md +++ b/site/content/blog/protobuf-value-objects/index.md @@ -1,13 +1,15 @@ --- -title: Protobuf — serialization and beyond. - Part 3. Value objects +title: Protobuf — serialization and beyond. Part 3. Value objects description: >- - In the following few chapters, we will explore the specifics of using Protobuf in systems based on Domain-Driven Design. This one describes a simple yet powerful concept of Value Objects. + In the following few chapters, we will explore the specifics of using Protobuf + in systems based on Domain-Driven Design. This one describes a simple yet powerful + concept of Value Objects. author: Alexander Yevsyukov and Dmytro Dashenkov publishdate: 2021-03-17 type: blog/post header_type: fixed-header --- + *This article continues the series on the practical aspects of using Google Protocol Buffers. In the following few chapters, we will explore the specifics of using Protobuf in systems based on Domain-Driven Design. This one describes a simple yet powerful concept of Value Objects.* We have been using Protobuf as more than just a serialization mechanism from the very beginning. It provides a language to model the domain. This ability is especially useful in projects based on the [Domain-Driven Design](https://martinfowler.com/bliki/DomainDrivenDesign.html) methodology, aka DDD. @@ -15,11 +17,13 @@ We have been using Protobuf as more than just a serialization mechanism from the The defining feature of a Domain Model is the [Ubiquitous Language](https://martinfowler.com/bliki/UbiquitousLanguage.html) — a language that is used and agreed upon throughout the domain: by the domain experts, programmers, and in the code. A Domain Model consists of many components, but the language lays the base for all of them. Value Objects help up transform the language into code. ## What are Value Objects? + A Value Object is a simple type that represents any kind of logically indivisible domain information, such as `EmailAddress`, `PhoneNumber`, `LocalDateTime`, `Money`, etc. Unlike Entities, Value Objects are only identified by the value itself, i.e. they do not have a designated ID. These objects are immutable and, typically, not too big. Apart from the encapsulation of the data itself, such types can hold the validation logic, basic operations on the data, the string representation, etc. For example, the type `LocalDateTime` may provide methods such as `add(Duration)` which calculates the local date and time after a given duration passes. The result is another instance of `LocalDateTime`, while the original object is unchanged. As mentioned earlier, the Value Objects are primarily used to integrate the language of the domain into the application’s system of types. It is a common misconception to treat Value Objects as dummy data structures. In practice, a Value Object is the way to escape all the problems related to the data structures, such as data inconsistencies and [anemic models](https://www.martinfowler.com/bliki/AnemicDomainModel.html). ## Why Protobuf? + Creating Value Objects in Protobuf is convenient because: 1. The domain types are created fast and for the many target languages, officially [supported by Google](https://developers.google.com/protocol-buffers/docs/tutorials) and [third-party](https://chromium.googlesource.com/chromium/src/+/master/third_party/protobuf/docs/third_party.md). @@ -36,13 +40,14 @@ string value = 1 [ Out of this declaration, the Protobuf compiler will generate a type, instances of which are compared by the value field. For example, in Java, the class would have a generated `equals()` method. -Note that the `(required)` and `(pattern)` options are extensions to Protobuf provided by the [Validation library](https://spine.io/docs/guides/validation) (which is a part of the Spine Event Engine framework). +Note that the `(required)` and `(pattern)` options are extensions to Protobuf provided by the [Validation library](docs/guides/validation/) (which is a part of the Spine Event Engine framework). In addition, in some target languages, such as Java, generated Protobuf types are immutable by default. Unfortunately, some other languages, such as JavaScript and Dart, only support [mutable types](https://github.com/dart-lang/protobuf/issues/413). For Dart, however, a community-driven solution seems to be on the way. The [immutable_proto](https://pub.dev/packages/immutable_proto) package implements code generation for immutable types from Protobuf. We have not tried it out yet, as the library is still in its earliest form, but the notion that other engineers feel our pain and try to do something about it as well warms our hearts. For further reading on immutability with regards to Protobuf, see our [previous article](blog/protobuf-immutability/). ## Validation + The `EmailAddress` type as declared above has one string field with naive validation via a regular expression. Also, the value field should be filled. This validation API is a part of our efforts on improving the code generation with Protobuf. Right now, we generate validation code for Java and Dart, in order to cover both backend and frontend. Later, other target languages might join the club. The validation rules are determined based on the options, such as `(required)` and `(pattern)`. At build time, we add the extra code which validates the message values based on those rules. The code is triggered automatically when a message is constructed. No more extra easy-to-forget steps for validation! @@ -50,6 +55,7 @@ The validation rules are determined based on the options, such as `(required)` a We will discuss the capabilities and the internals of our validation mechanism in more detail in the next articles of this series. ## Adding Behavior with the (is) Option + An important part of the Value Object is its behavior. OOP greatly influences the mindset of a programmer, and the need to create utility classes and methods for every little thing at the same time annoys and complicates writing and understanding the code. The ability to create domain types quickly is nice, but we also want them to be convenient to “talk” about in the code. Instead of `user.getAddress().getCountry()`, we would like to be able to write `user.country()`. In Java, Protobuf generates non-extensible classes, which makes it hard to add behavior to them. We have solved this problem by defining the option (is). It takes the names of Java interfaces with which we want to mark the Protobuf message. Such interfaces may include default methods, adding behavior to the Value Object. Our custom Protobuf compiler plugin modifies the generated code so that the Proto-types implement the assigned interfaces. Here is how this works. @@ -66,7 +72,6 @@ Address address = 2 [(required) = true]; And here is the `UserMixin` declaration: - ```java public interface UserMixin extends UserOrBuilder { /** @@ -127,6 +132,7 @@ string phone = 3; Since Protobuf is specifically designed to allow additive changes to types without any migration hustle, changing ID structure might just be as easy as adding extra fields to the ID type. ## Conclusion + Value Objects as a whole is a great concept. It helps the developers bring the ubiquitous language into the code and avoid errors by forming a strongly typed model, instead of one based on primitives. Protobuf helps make the creation and maintenance of Value Objects easier. Simple Value Objects which introduce domain clarity into the code are a great improvement already. Coupled with typed identifiers, they bring extra benefits. Thanks to Protobuf, such types can be declared once and distributed all around the system, bridging the language barriers between different components. @@ -135,4 +141,4 @@ Adding an extra layer of bespoke code generation, including validation rules, be Lastly, coupled with the Domain-Driven approach to software architecture, we get a system that, from the ground up, helps the developer tackle only one problem at a time, from simple mechanical issues, on the Value Object level, to however complex business requirements on higher levels. -In the next parts of this series, we will explore those higher levels of the Domain-Driven Design adoption, and see how Protobuf can help us besides Value Objects. \ No newline at end of file +In the next parts of this series, we will explore those higher levels of the Domain-Driven Design adoption, and see how Protobuf can help us besides Value Objects. diff --git a/site/content/blog/protobuf-whats-in-it-for-your-project/index.md b/site/content/blog/protobuf-whats-in-it-for-your-project/index.md index 4663da23..98f1963a 100644 --- a/site/content/blog/protobuf-whats-in-it-for-your-project/index.md +++ b/site/content/blog/protobuf-whats-in-it-for-your-project/index.md @@ -1,8 +1,9 @@ --- -title: Protobuf — serialization and beyond. - Part 1. What’s in it for your project +title: Protobuf — serialization and beyond. Part 1. What’s in it for your project description: >- - In this article we look at the key Google Protocol Buffers (aka Protobuf) features which make working with data in software development projects more convenient and effective. + In this article we look at the key Google Protocol Buffers (aka Protobuf) + features which make working with data in software development projects more + convenient and effective. author: Alexander Yevsyukov and Dmytro Dashenkov publishdate: 2020-11-16 type: blog/post @@ -24,6 +25,7 @@ We use the [Protobuf v3](https://developers.google.com/protocol-buffers/docs/pro ## Starting Off with Protobuf ### Lost in Transition + Our Protobuf story started at a SaaS project. The system was built based on the event-oriented, or, as it is called lately, reactive Domain-Driven Design. Events and data to display were transferred to the JavaScript browser application as JSON objects. Then the customer decided to add Android and iOS clients. By the time the work on the client applications started, the event model was already formed as a hierarchy of Java classes. And there were quite a few of those. So we faced the following questions: @@ -33,6 +35,7 @@ So we faced the following questions: Abandoning data types and only using JSON for all tasks was not an option. It would turn working with the domain model, which is “the heart and the brain” of the business, into operations with strings and primitive types! We set off searching. ### Looking for Solution + One of the first options we came into was the Wire library by Square. Back then, it was versioned 1.x, supporting Protobuf v2, while Protobuf v3 was in alpha-3 version. The Wire did not solve all our issues with the platforms’ support, as it was intended only for Android and Java, but it got us aware of the Protobuf technology and of the applied code generation. [Compared to others](https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html), Protobuf looked like the best option. Yet another issue arose. In Java we had a hierarchy of classes for events and other types of data. As most of them are transferred between client and server, we were looking for the means to define the similar structure of data in all languages involved. But inheritance in Java works differently from what was available in languages, such as JavaScript. A significant amount of hand-written code was needed to implement and then to maintain the hierarchies of the same types for all platforms. @@ -47,16 +50,19 @@ We came up with the following solutions: Bringing these solutions to life we: * Created Spine Event Engine — the framework for the projects, developed using the Domain-Driven Design methodology, where Protobuf makes working with the data way easier. It speeds up the development and reduces the labor costs of the software. -* Cut down the amount of the code written manually in our integration libraries, [JxBrowser](https://www.teamdev.com/jxbrowser) and [DotNetBrowser](https://www.teamdev.com/dotnetbrowser), where C++ code of Chromium couples with Java and C#. +* Cut down the amount of the code written manually in our integration libraries, [JxBrowser](https://teamdev.com/jxbrowser/) and [DotNetBrowser](https://teamdev.com/dotnetbrowser/), where C++ code of Chromium couples with Java and C#. We will talk in detail about these in the next articles of this series. Meanwhile, let’s take a look at the Protobuf features, which make it useful for many projects, regardless of their nature. ## What Makes Protobuf So Handy + ### Support of Many Languages + The latest versions of Protobuf [support](https://developers.google.com/protocol-buffers/docs/reference/overview) C#, C++, Dart, Go, Java, JavaScript, Objective-C, Python, PHP, Ruby. There is a Swift implementation [by Apple itself](https://github.com/apple/swift-protobuf/). If you need one for Closure, Erlang или Haskell, the list of the [third-party libraries for different languages](https://github.com/protocolbuffers/protobuf/blob/master/docs/third_party.md) is extensive. As the name of the article implies, the Protobuf-based code can be used for all the operations with data, not only serialization. And this is the approach we recommend. However, serialization is also worth discussing. It is where it all usually starts. ### Binary Serialization + Protobuf employs the binary serialization mechanism to transfer data between the nodes effectively, to write and to read from the different languages without additional efforts, and to introduce format changes without breaking compatibility. Let’s assume we have a data type `Task` defined as follows: @@ -67,6 +73,7 @@ message Task { string description = 2; } ``` + Then in Java, you can get such object in binary form this way: ```java @@ -92,6 +99,7 @@ As a result, we get an array of 8-bit numbers. For reverse transformation, the ` ```java const task = myprotos.Task.deserializeBinary(bytes); ``` + The situation is alike for Dart. The `writeToBuffer()` method with the common ancestor of the generated classes returns a list of unsigned 8-bit integers: ```dart @@ -102,7 +110,9 @@ And the `fromBuffer()` constructor performs the reverse transformation: ```dart var task = Task.fromBuffer(bytes); ``` + ### Resilience to Type Change + Writing code in modern IDEs makes it much easier to modify existing data structures. For example, for Java, there are many types of automatic code refactoring available. However, instances of domain types are often persisted to some storage and then read back over and over. Again, with Java, it takes quite an effort — in terms of both time and cost — to craft a fine-grained serialization mechanism to keep the data from years back alive today. Protobuf is designed to ensure the backward compatibility of the serialized data. It is flexible enough when it is necessary to change an existing data type. We will not describe [all the possibilities](https://developers.google.com/protocol-buffers/docs/proto3#updating) but only focus on the most interesting. @@ -129,6 +139,7 @@ message MyMessage { int32 old_field = 6 [deprecated = true]; } ``` + **Step 2.** Generate the code for the updated type. **Step 3.** Update the calling code, getting rid of the `@Deprecated` method calls. @@ -140,8 +151,9 @@ message MyMessage { ... reserved 6; reserved “old_field”; - } - ``` +} +``` + By doing this we insure ourselves from accidental usage of the old index or name and do not have to keep the outdated code. Renaming fields is not much harder. #### Renaming Fields @@ -170,6 +182,7 @@ Obviously, it is less convenient compared to ordinary renaming in the environmen Also note, that if you use serialization in JSON, it is better not to rename a lot because updating the clients to the new version requires additional effort. ### Output to JSON + As we’ve mentioned earlier, JSON is not the most effective way to exchange data. This string-based protocol doesn’t have an official schema format, and the techniques for supporting any kind of data types with JSON vary immensely. In most cases when you need serialization, Protobuf will do well. Being a serialization protocol on its own, it converts into a concise binary representation, which is supported by all the platforms which support Protobuf. @@ -206,7 +219,7 @@ var printer = JsonFormat.printer() .usingTypeRegistry(registry); ``` -By default the `Printer` output is easy-to-read. However, you can create a version, which will print in the compact format: +By default, the `Printer` output is easy-to-read. However, you can create a version, which will print in the compact format: ```java var compactPrinter = printer.omittingInsignificantWhitespace(); @@ -237,6 +250,7 @@ Serialization: ```dart var jsonStr = task.toProto3Json(); ``` + Deserialization: ```dart @@ -257,6 +271,7 @@ task.mergeFromProto3Json(jsonStr, typeRegistry: registry); ``` ### Immutability + Immutable types make the developer’s life way better. For instance, have a look at [this talk](https://www.youtube.com/watch?v=pLvrZPSzHxo&feature=youtu.be) on how immutability helps to build great UIs. Protobuf objects in Java are immutable. And this is convenient. It is also convenient to create a new object based on the previous one: @@ -272,4 +287,4 @@ But it’s not that bright for JavaScript and Dart. We’ll talk more about the desire to keep things unchanged and the urge to modify them in the article on immutability, which comes next in this series. ## Summary -This article begins a tale of our journey with Protobuf. Now, as we’ve discussed the basics and general advantages of using this technology, we will dive into how it can be applied to particular projects. Next up we will talk about the way Protobuf helps you improve your code quality and architecture style. \ No newline at end of file +This article begins a tale of our journey with Protobuf. Now, as we’ve discussed the basics and general advantages of using this technology, we will dive into how it can be applied to particular projects. Next up we will talk about the way Protobuf helps you improve your code quality and architecture style. From fccca452cd5a049df54141f1f312f54a59a272de Mon Sep 17 00:00:00 2001 From: "julia.evseeva" Date: Mon, 12 Jan 2026 12:40:36 +0100 Subject: [PATCH 10/13] Fix the code alignment --- .../blog/protobuf-immutability/index.md | 10 ++--- .../blog/protobuf-value-objects/index.md | 44 +++++++++---------- 2 files changed, 27 insertions(+), 27 deletions(-) diff --git a/site/content/blog/protobuf-immutability/index.md b/site/content/blog/protobuf-immutability/index.md index fbdc7c7b..8705ff3f 100644 --- a/site/content/blog/protobuf-immutability/index.md +++ b/site/content/blog/protobuf-immutability/index.md @@ -51,17 +51,17 @@ In Java, generated Protobuf classes are always immutable. Each class has a neste ```java Task task = Task.newBuilder() -.setName(“…”) -.setDescription(“…”) -.build(); + .setName(“…”) + .setDescription(“…”) + .build(); ``` In order to change a value, we create a new message based on an existing one. ```java Task updated = task.toBuilder() -.setDescription(“…”) -.build(); + .setDescription(“…”) + .build(); ``` The way Protobuf generates code represents a typical immutable API in Java, excluding some Protobuf-specific extras, such as reflection, etc. diff --git a/site/content/blog/protobuf-value-objects/index.md b/site/content/blog/protobuf-value-objects/index.md index 8d65c6cc..6c55c7d3 100644 --- a/site/content/blog/protobuf-value-objects/index.md +++ b/site/content/blog/protobuf-value-objects/index.md @@ -31,10 +31,10 @@ Creating Value Objects in Protobuf is convenient because: ```proto message EmailAddress { -string value = 1 [ -(required) = true, -(pattern).regex = ".+@.+\..+" -]; + string value = 1 [ + (required) = true, + (pattern).regex = ".+@.+\..+" + ]; } ``` @@ -62,11 +62,11 @@ In Java, Protobuf generates non-extensible classes, which makes it hard to add b ```proto message User { -option (is).java_type = "UserMixin"; -UserId id = 1 [(required) = true]; -// The primary billing address. -Address address = 2 [(required) = true]; -... + option (is).java_type = "UserMixin"; + UserId id = 1 [(required) = true]; + // The primary billing address. + Address address = 2 [(required) = true]; + ... } ``` @@ -74,13 +74,13 @@ And here is the `UserMixin` declaration: ```java public interface UserMixin extends UserOrBuilder { -/** -* Obtains the residence country of this user. -*/ -default Country country() { -return getAddress().getCountry(); -} -... + /** + * Obtains the residence country of this user. + */ + default Country country() { + return getAddress().getCountry(); + } + ... } ``` @@ -113,7 +113,7 @@ Another benefit to type-safe identifiers lies in their structure. In the basic c ```proto message CustomerId { -string uuid = 1; + string uuid = 1; } ``` @@ -121,11 +121,11 @@ However, by hiding the type of identifier implementation, we gain the ability to ```proto message CustomerId { -oneof kind { -uint64 code = 1; -EmailAddress email = 2; -string phone = 3; -} + oneof kind { + uint64 code = 1; + EmailAddress email = 2; + string phone = 3; + } } ``` From 6f4ed4bf6af6bd0acc59620b4d11c221a0f55cd7 Mon Sep 17 00:00:00 2001 From: "julia.evseeva" Date: Mon, 12 Jan 2026 12:45:55 +0100 Subject: [PATCH 11/13] Update date style --- site/assets/scss/pages/blog/_post-info.scss | 2 ++ 1 file changed, 2 insertions(+) diff --git a/site/assets/scss/pages/blog/_post-info.scss b/site/assets/scss/pages/blog/_post-info.scss index 3dee23ca..a780ed1b 100644 --- a/site/assets/scss/pages/blog/_post-info.scss +++ b/site/assets/scss/pages/blog/_post-info.scss @@ -31,6 +31,7 @@ margin-bottom: 16px; } + span, .post-author, .date { display: inline-block; @@ -39,6 +40,7 @@ margin: 0; } + span, .date { color: $gray-300; } From b2c9088cb160e041ade65947968c86c3e2573f0b Mon Sep 17 00:00:00 2001 From: "julia.evseeva" Date: Mon, 12 Jan 2026 12:52:31 +0100 Subject: [PATCH 12/13] Fix URLs --- site/content/blog/protobuf-immutability/index.md | 2 +- site/content/blog/protobuf-value-objects/index.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/site/content/blog/protobuf-immutability/index.md b/site/content/blog/protobuf-immutability/index.md index 8705ff3f..94468f0c 100644 --- a/site/content/blog/protobuf-immutability/index.md +++ b/site/content/blog/protobuf-immutability/index.md @@ -76,7 +76,7 @@ Such a difference between Java and client-side languages can be explained by a d Nevertheless, in Dart, there are a few tools, as well as language structures, which can help you with immutability. For instance, `built_value` and `freezed` are two libraries that allow their users to create immutable data types with [almost no extra effort](https://levelup.gitconnected.com/flutter-dart-immutable-objects-and-values-5e321c4c654e). But in order to achieve immutability for data types that originate in Protobuf, we’d have to arrange the conversion from mutable to immutable types, which is a lot of manual work. And in the end, we’d get two classes for one Protobuf type. This is a suboptimal solution, to say the least. -We have tried creating a symbiosis between `built_value` and Protobuf to achieve immutable “front-facing” types, which would rely on standard Protobuf Dart code for serialization. This attempt has failed as we have found ourselves juggling too many noncoherent abstractions. At the time of writing, we [plan](https://github.com/SpineEventEngine/core-java/issues/1334) to implement custom code generation to support Dart, JavaScript, and Java, which addresses the limitations we face and provides immutability. +We have tried creating a symbiosis between `built_value` and Protobuf to achieve immutable “front-facing” types, which would rely on standard Protobuf Dart code for serialization. This attempt has failed as we have found ourselves juggling too many noncoherent abstractions. At the time of writing, we [plan]({{% get-site-data "spine.core_jvm_repo" %}}/issues/1334) to implement custom code generation to support Dart, JavaScript, and Java, which addresses the limitations we face and provides immutability. ## What’s Next? diff --git a/site/content/blog/protobuf-value-objects/index.md b/site/content/blog/protobuf-value-objects/index.md index 6c55c7d3..4a461635 100644 --- a/site/content/blog/protobuf-value-objects/index.md +++ b/site/content/blog/protobuf-value-objects/index.md @@ -42,7 +42,7 @@ Out of this declaration, the Protobuf compiler will generate a type, instances o Note that the `(required)` and `(pattern)` options are extensions to Protobuf provided by the [Validation library](docs/guides/validation/) (which is a part of the Spine Event Engine framework). -In addition, in some target languages, such as Java, generated Protobuf types are immutable by default. Unfortunately, some other languages, such as JavaScript and Dart, only support [mutable types](https://github.com/dart-lang/protobuf/issues/413). For Dart, however, a community-driven solution seems to be on the way. The [immutable_proto](https://pub.dev/packages/immutable_proto) package implements code generation for immutable types from Protobuf. We have not tried it out yet, as the library is still in its earliest form, but the notion that other engineers feel our pain and try to do something about it as well warms our hearts. +In addition, in some target languages, such as Java, generated Protobuf types are immutable by default. Unfortunately, some other languages, such as JavaScript and Dart, only support [mutable types](https://github.com/google/protobuf.dart/issues/413). For Dart, however, a community-driven solution seems to be on the way. The [immutable_proto](https://pub.dev/packages/immutable_proto) package implements code generation for immutable types from Protobuf. We have not tried it out yet, as the library is still in its earliest form, but the notion that other engineers feel our pain and try to do something about it as well warms our hearts. For further reading on immutability with regards to Protobuf, see our [previous article](blog/protobuf-immutability/). From 30c61348266f3a22a0780cec29b0dfea04d9fa6a Mon Sep 17 00:00:00 2001 From: "julia.evseeva" Date: Mon, 12 Jan 2026 12:52:51 +0100 Subject: [PATCH 13/13] Add the `medium.com` to excludes due to the 403 error --- lychee.toml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lychee.toml b/lychee.toml index e74b06fb..06a6e7ac 100644 --- a/lychee.toml +++ b/lychee.toml @@ -13,7 +13,8 @@ exclude = [ "stackoverflow.com/questions/*", "openjdk.org/*", "npmjs.com/*", - "raw.githubusercontent.com/SpineEventEngine/*" + "raw.githubusercontent.com/SpineEventEngine/*", + "medium.com/*" ] # Exclude these filesystem paths from getting checked.