Comprehensive Serialization Framework Comparison Tests

[expanded-for-real]

Update current comparison tests to benchmark Imprint against common serialization frameworks, ranging from prolific general-purpose libraries (Jackson) to high-performance streaming formats (Protobuf, Avro).

Required Frameworks

Core Frameworks (Must Have)

1. Protobuf - Google's high-performance binary format, widely used in microservices
2. Avro - Apache's schema evolution-friendly format, popular in big data ecosystems
3. Kryo - Fast Java serialization library, common in distributed systems
4. Jackson JSON - De facto standard for JSON serialization in Java
5. MessagePack (via Jackson) - Efficient binary JSON-like format
6. FlatBuffers - Google's zero-copy serialization, direct competitor to Imprint's field access model
7. FST - Fast Java serialization, popular drop-in replacement for Java serialization
8. CBOR - Concise Binary Object Representation, standardized binary JSON alternative

Optional Frameworks (Nice to Have)

9. Apache Thrift - Cross-language RPC framework with efficient serialization
10. Apache Fury (formerly Fury) - JIT-optimized serialization framework
11. Jackson with Blackbird - Jackson with bytecode generation for performance

Test Categories

1. Serialization Performance

• Metric: Average time per operation (ns/op)
• Goal: Measure write performance across frameworks
• Expected: Imprint may be slower due to directory construction overhead

2. Deserialization Performance

• Metric: Average time per operation (ns/op)
• Goal: Measure read performance across frameworks
• Expected: Imprint should excel due to O(1) field directory

3. Single Field Access Performance

• Metric: Time to access field near end of record without full deserialization
• Goal: Showcase Imprint's core value proposition
• Expected: Imprint should significantly outperform competitors

4. Serialized Size Comparison

• Metric: Bytes per serialized record
• Goal: Compare storage efficiency
• Expected: Binary formats (Protobuf, Avro) likely smallest

5. Merge/Update Operations

• Metric: Time to merge two records and re-serialize
• Goal: Test partial update scenarios
• Expected: Imprint should be competitive due to selective field access

Test Data Characteristics

Data Characteristics

• Field count: 20 fields total (7 core + 13 extra)
• Record size: ~500-1000 bytes when serialized
• Access pattern: Tests should access field 15+ to simulate "field near end" scenarios
• Complexity: Mix of primitives, strings, collections, and nested data

Production-Like Configuration

Framework Setup Requirements

• Protobuf: Use generated classes with proper schema definition
• Avro: Use generic records with predefined schema, binary encoding
• Kryo: Register classes, configure for production (no auto-registration)
• Jackson: Standard ObjectMapper with default settings
• MessagePack: Jackson with MessagePack factory
• Thrift (optional): Use compact protocol, generated classes
• Fury (optional): Configure for Java-only optimization

Test Environment

• JVM Settings: Standard production JVM settings, sufficient heap
• Warmup: Adequate JMH warmup iterations (3+ iterations, 1+ seconds each)
• Measurement: Consistent measurement iterations (3+ iterations)
• Threading: Single-threaded tests to avoid concurrency overhead

Implementation Notes

Dependencies

• Add protobuf-gradle-plugin for schema compilation
• Include all framework dependencies in JMH scope only (not main library)
• Use compatible versions across frameworks

[expanded-for-real]

Imprint Serialization Framework Benchmark Results

Comprehensive performance comparison between Imprint and other popular Java serialization libraries.

Deserialization Performance

Library	Score (ns/op)	Error	Relative Performance
FlatBuffers	20.48	±4.48	1.0x (baseline)
Imprint	949.59	±84.52	46.4x slower
Protobuf	1,624.15	±116.95	79.3x slower
Kryo	2,432.46	±581.81	118.8x slower
Avro	3,178.98	±1,950.29	155.3x slower
Jackson JSON	3,301.59	±1,255.19	161.3x slower
MessagePack	3,465.13	±763.14	169.2x slower

Single Field Access Performance

Library	Score (ns/op)	Error	Relative Performance
FlatBuffers	56.86	±3.97	1.0x (baseline)
Imprint	1,109.83	±105.04	19.5x slower
Protobuf	1,756.12	±194.33	30.9x slower
Kryo	2,330.56	±643.53	41.0x slower
Avro	2,713.64	±291.58	47.7x slower
Jackson JSON	3,030.92	±303.88	53.3x slower
MessagePack	3,235.35	±492.90	56.9x slower

Serialization Performance

Library	Score (ns/op)	Error	Relative Performance
FlatBuffers	2,312.61	±166.98	1.0x (baseline)
Kryo	2,489.67	±111.61	1.1x slower
Avro	2,574.01	±287.78	1.1x slower
Jackson JSON	3,115.20	±421.58	1.3x slower
Protobuf	3,176.99	±291.45	1.4x slower
MessagePack	3,481.52	±941.89	1.5x slower
Imprint	4,159.82	±611.58	1.8x slower

Merge Operation Performance

Library	Score (ns/op)	Error	Relative Performance
FlatBuffers	8,486.27	±5,182.12	1.0x (baseline)
Kryo	12,443.80	±7,478.14	1.5x slower
Imprint	12,739.71	±1,844.28	1.5x slower
Jackson JSON	12,859.14	±1,785.97	1.5x slower
Protobuf	12,936.36	±2,446.32	1.5x slower
Avro	15,579.03	±10,501.37	1.8x slower
MessagePack	20,969.48	±18,362.38	2.5x slower

Serialized Size Comparison

Library	Score (ns/op)	Error
Avro	0.79	±0.05
Jackson JSON	0.83	±0.05
Protobuf	0.86	±0.08
Kryo	0.87	±0.25
MessagePack	0.89	±0.11
Imprint	1.07	±0.12
FlatBuffers	1.07	±0.17

Lower scores are better for all benchmarks

[expanded-for-real]

@agavra results included here (... and hopefully I can get this kind of report published during the CI pipeline one day lol). The deserialize on Imprint and Flatbuffer needs to be fixed as it isn't quite apples to apples with the others. Aside from that everything looks promising so far except for the fact that flatbuffers is ridiculously fast on single field access and Imprint should be way closer to that. That being said, flatbuffer does have a pretty high overhead in terms of setup and maintenance that we're beating (like at least Imprint doesn't need a separate C compiler).

Right now, deserialize() parses the entire directory upfront, but we could potentially defer that and parse only the header for basic validation, and then store the raw directory bytes without actually parsing individual entries. Then we just use binary search directly on the raw directory buffer whenever fields are accessed. Assuming that actually works, it could reduce the initial deserialization cost, especially if you definitely only need to access a couple of records. The downside is that if you want to get all records in the object then it'd probably be slower than something like protobuf, maybe even closer to Jackson. But I think part of Imprint's design is to be optimized to only want to access a few fields anyways?

Also need to look at why full serialization is so slow

[agavra]

This is awesome, thanks @expanded-for-real! I added some thoughts about overall performance on #9 as well.