A production-ready benchmarking suite for evaluating 13 modern Java serialization frameworks with comprehensive performance metrics, resource monitoring, and real-time visualization.
- Introduction
- Supported Frameworks
- Getting Started
- Architecture
- Benchmarking
- Metrics
- Monitoring
- Performance Results
- Framework Selection
- API Reference
- Configuration
- Troubleshooting
- Contributing
- Resources
This project provides a comprehensive evaluation platform for Java serialization frameworks. Each framework is implemented as an independent Spring Boot microservice with standardized REST APIs, enabling fair and consistent performance comparisons across 13 different serialization technologies.
Choosing the right serialization framework impacts application performance, scalability, and maintainability. However, comparing frameworks fairly requires:
- Identical testing conditions
- Consistent payload structures
- Comprehensive metrics collection
- Real-world scenario simulation
This benchmark solves these challenges by providing:
- Standardized V2 API across all 13 frameworks
- Unified payload generator for fair comparison
- Four-phase metrics collection (network, serialization, resources, transport)
- Production-ready monitoring with Grafana and Prometheus
Measures actual production concerns: network latency, serialization time, resource usage, and transport efficiency.
Single command-line interface for all operations: start, stop, benchmark, analyze.
Pre-configured Grafana dashboards with Prometheus metrics export for real-time analysis.
Standardized V2 API ensures identical testing conditions across all frameworks.
104 test scenarios: 4 payload sizes times 2 configurations times 13 frameworks.
| ID | Framework | Port | Category | Official Site |
|---|---|---|---|---|
| 1 | Jackson JSON | 8081 | Text-based | https://github.com/FasterXML/jackson |
| 2 | Apache Avro | 8083 | Binary Schema | https://avro.apache.org/ |
| 3 | Kryo | 8084 | Binary Schema-less | https://github.com/EsotericSoftware/kryo |
| 4 | MessagePack | 8086 | Binary Schema-less | https://msgpack.org/ |
| 5 | Apache Thrift | 8087 | Binary Schema | https://thrift.apache.org/ |
| 6 | Cap'n Proto | 8088 | Zero-Copy | https://capnproto.org/ |
| 7 | FST | 8090 | Binary Schema-less | https://github.com/RuedigerMoeller/fast-serialization |
| 8 | gRPC | 8092 | RPC Framework | https://grpc.io/ |
| 9 | CBOR | 8093 | Binary Schema-less | https://cbor.io/ |
| 10 | BSON | 8094 | Binary Schema-less | http://bsonspec.org/ |
| 11 | Apache Arrow | 8095 | Columnar | https://arrow.apache.org/ |
| 12 | SBE | 8096 | Binary Schema | https://github.com/real-logic/simple-binary-encoding |
| 13 | Apache Parquet | 8097 | Columnar | https://parquet.apache.org/ |
Description: Human-readable serialization formats
- Jackson JSON: Ubiquitous web API format with excellent tooling
Description: Require schema definition for serialization
- Apache Avro: Schema evolution with backward and forward compatibility
- Apache Thrift: Multi-language RPC with Interface Definition Language
- SBE: Fixed-size encoding for ultra-low latency applications
Description: No schema definition required
- Kryo: Fast Java-only serialization
- FST: Drop-in replacement for Java serialization
- MessagePack: Efficient binary JSON alternative
- CBOR: Concise Binary Object Representation
- BSON: Binary JSON with rich type system
Description: Direct memory access without copying
- Cap'n Proto: Zero-copy reads with schema evolution support
Description: Column-oriented storage for analytics
- Apache Arrow: In-memory columnar format for data exchange
- Apache Parquet: Columnar file format with excellent compression
Description: Remote procedure call support
- gRPC: HTTP/2-based RPC with streaming capabilities
- SBE: 0.89ms average
- Cap'n Proto: 1.23ms average
- FST: 1.45ms average
- Kryo: 1.67ms average
- Apache Arrow: 2.01ms average
- Apache Parquet: 0.25 ratio (75% reduction)
- Apache Avro: 0.32 ratio (68% reduction)
- MessagePack: 0.38 ratio (62% reduction)
- BSON: 0.42 ratio (58% reduction)
- CBOR: 0.45 ratio (55% reduction)
- Java 21 (GraalVM or OpenJDK)
- Maven 3.9 or higher
- Python 3.8 or higher
- Git
- Prometheus (for monitoring)
- Grafana (for visualization)
git clone https://github.com/yourusername/java_serialization_frameworks.git
cd java_serialization_frameworkspip3 install psutilchmod +x manage.sh enhanced_benchmark.py./manage.sh startWait 30-60 seconds for all services to initialize.
./manage.sh statusExpected output shows all 13 services running.
./manage.sh benchmarkDuration: Approximately 15-20 minutes for complete benchmark suite.
./manage.sh analyzeView performance rankings and detailed metrics analysis.
java_serialization_frameworks/
|
|-- README.md (this file)
|
|-- docs/ (complete documentation)
| |-- README.md (documentation index)
| |-- frameworks/ (framework deep-dives)
| | |-- jackson.md
| | |-- avro.md
| | |-- [11 more frameworks]
| |-- guides/ (comprehensive guides)
| |-- BENCHMARK_SUMMARY.md
| |-- SCREENSHOT_GUIDE.md
| |-- VISUAL_DOCUMENTATION.md
|
|-- manage.sh (unified CLI)
|-- enhanced_benchmark.py (metrics collection)
|-- analyze_metrics.py (results analysis)
|
|-- dashboards/ (Grafana configs)
| |-- serialization-performance.json
| |-- resource-utilization.json
| |-- README.md
|
|-- common-payload/ (shared models)
| |-- pom.xml
| |-- src/main/java/org/techishthoughts/payload/
| |-- generator/
| |-- model/
| |-- service/
|
|-- jackson-poc/ (framework implementations)
|-- avro-poc/
|-- kryo-poc/
|-- msgpack-poc/
|-- thrift-poc/
|-- capnproto-poc/
|-- fst-poc/
|-- grpc-poc/
|-- cbor-poc/
|-- bson-poc/
|-- arrow-poc/
|-- sbe-poc/
|-- parquet-poc/
|
|-- results/ (benchmark outputs)
|-- logs/ (service logs)
|-- screenshots/ (visual documentation)
Ensures fair comparison by using identical payloads across all frameworks.
- User profiles with demographics
- Order history with line items
- Social connections and skills
- Tracking events with timestamps
| Level | User Count | Approximate Size | Primary Use Case |
|---|---|---|---|
| SMALL | 10 | 1 KB | Latency measurement |
| MEDIUM | 100 | 10 KB | Standard benchmarking |
| LARGE | 1,000 | 100 KB | Throughput testing |
| HUGE | 10,000 | 1 MB | Stress testing |
All frameworks implement: POST /api/{framework}/v2/benchmark
{
"complexity": "MEDIUM",
"iterations": 50,
"enableWarmup": true,
"enableCompression": true,
"enableRoundtrip": true,
"enableMemoryMonitoring": true
}{
"success": true,
"framework": "Kryo",
"complexity": "MEDIUM",
"iterations": 50,
"totalDurationMs": 845.23,
"averageSerializationTimeMs": 16.90,
"averageSerializedSizeBytes": 8234,
"averageCompressionRatio": 0.72,
"successRate": 100.0,
"roundtripSuccess": true,
"memoryMetrics": {
"peakMemoryMb": 245.8,
"memoryDeltaMb": 8.4
}
}./manage.sh start # Start all 13 services
./manage.sh stop # Stop all services
./manage.sh restart # Restart all services
./manage.sh status # Check service health./manage.sh benchmark # Run comprehensive benchmark
./manage.sh analyze # Analyze latest results./manage.sh logs <framework> # View framework logs
./manage.sh clean # Clean logs and temporary files./manage.sh help # Display detailed usageCommand: ./manage.sh benchmark
What It Tests:
- 13 frameworks
- 4 payload sizes (SMALL, MEDIUM, LARGE, HUGE)
- 2 configurations (baseline, with compression)
- All 4 metrics phases
Duration: 15-20 minutes
Command: python3 enhanced_benchmark.py
Customization: Modify SCENARIOS and BENCHMARK_CONFIGS variables in the script.
curl -X POST http://localhost:8081/api/jackson/v2/benchmark \
-H "Content-Type: application/json" \
-d '{
"complexity": "SMALL",
"iterations": 100,
"enableWarmup": true,
"enableCompression": false,
"enableRoundtrip": true,
"enableMemoryMonitoring": true
}'- Run benchmarks on idle system
- Close unnecessary applications
- Use consistent hardware
- Disable power-saving modes
- Always enable warm-up for JVM optimization
- Run benchmarks multiple times (minimum 3)
- Average results for consistency
- Monitor system resources during execution
- Verify all services are healthy before starting
- Check for errors in logs
- Validate roundtrip success rates
- Compare results across runs for consistency
Purpose: Measure connection establishment overhead
Metrics Collected:
- DNS lookup time: Name resolution latency
- TCP connect time: TCP handshake duration
- TLS handshake time: SSL/TLS negotiation time
- Total connection time: Complete connection establishment
Purpose: Measure serialization and deserialization speed
Metrics Collected:
- Average time: Mean serialization time
- Minimum time: Best case performance
- Maximum time: Worst case performance
- P50 latency: Median performance
- P95 latency: 95th percentile
- P99 latency: 99th percentile
- Throughput: Operations per second
Purpose: Measure system resource consumption
Metrics Collected:
- CPU usage: Processor utilization percentage
- Memory usage: Heap memory consumption in MB
- Memory delta: Memory increase during operation
- Thread count: Active thread count
Purpose: Measure data transmission efficiency
Metrics Collected:
- Payload size: Serialized data size in bytes
- Compression ratio: Size reduction percentage
- Network throughput: Data transfer rate in Mbps
- Overhead: Protocol overhead percentage
Location: results/enhanced_benchmark_YYYYMMDD_HHMMSS.json
Contents:
- Complete test results
- All metrics for all test scenarios
- Metadata and timestamps
Location: results/metrics_YYYYMMDD_HHMMSS.prom
Contents:
- Time-series metrics
- Grafana-compatible format
- Real-time monitoring support
Contents:
- Real-time progress updates
- Summary statistics
- Performance rankings
Purpose: Comprehensive performance analysis and comparison
Visualizations:
- Serialization time line chart
- Throughput comparison
- Payload size bar gauge
- Compression ratio statistics
- Memory and CPU usage trends
- Performance heatmap
- Framework rankings table
Filters:
- Framework selection (multi-select)
- Payload size (SMALL, MEDIUM, LARGE, HUGE)
- Configuration (baseline, with compression)
Purpose: Deep dive into resource consumption
Visualizations:
- Memory usage time series
- CPU usage by framework
- Memory efficiency (MB per 1000 operations)
- CPU efficiency (percentage per 1000 operations)
- Resource utilization heatmap
- Top resource consumers table
- Alert statistics
brew install prometheusEdit /opt/homebrew/etc/prometheus.yml:
scrape_configs:
- job_name: 'serialization-benchmarks'
file_sd_configs:
- files:
- '/path/to/java_serialization_frameworks/results/*.prom'
refresh_interval: 30sbrew services start prometheusNavigate to: http://localhost:9090
brew install grafana
brew services start grafanaNavigate to: http://localhost:3000 Default credentials: admin/admin
- Navigate to Configuration > Data Sources
- Select Prometheus
- Set URL: http://localhost:9090
- Click Save and Test
curl -X POST http://admin:admin@localhost:3000/api/dashboards/db \
-H "Content-Type: application/json" \
-d @dashboards/serialization-performance.json| Rank | Framework | Average Time | Payload Size | Throughput |
|---|---|---|---|---|
| 1 | SBE | 0.89ms | 1.2KB | 1,124 ops/s |
| 2 | Cap'n Proto | 1.23ms | 1.5KB | 813 ops/s |
| 3 | FST | 1.45ms | 2.3KB | 690 ops/s |
| 4 | Kryo | 1.67ms | 2.3KB | 599 ops/s |
| 5 | Apache Arrow | 2.01ms | 3.4KB | 498 ops/s |
| Rank | Framework | Compression Ratio | Size Reduction |
|---|---|---|---|
| 1 | Apache Parquet | 0.25 | 75% |
| 2 | Apache Avro | 0.32 | 68% |
| 3 | MessagePack | 0.38 | 62% |
| 4 | BSON | 0.42 | 58% |
| 5 | CBOR | 0.45 | 55% |
| Category | Average Time | Best Framework | Primary Use Case |
|---|---|---|---|
| Binary Schema | 1.45ms | SBE (0.89ms) | Ultra-low latency systems |
| Binary Schema-less | 2.12ms | FST (1.45ms) | High-performance Java applications |
| Zero-Copy | 1.23ms | Cap'n Proto | Low-latency inter-process communication |
| Columnar | 3.45ms | Arrow (2.01ms) | Analytics and data processing |
| RPC Framework | 4.56ms | gRPC | Microservices communication |
| Text-based | 5.67ms | Jackson | Web APIs and human-readable data |
Recommended: Jackson JSON Alternative: CBOR, MessagePack
Rationale: Human-readable format, universal HTTP client support, easy debugging.
Recommended: gRPC Alternative: Apache Avro with Kafka
Rationale: HTTP/2 support, streaming capabilities, strong typing with Protocol Buffers.
Recommended: Kryo Alternative: FST, SBE
Rationale: Fastest serialization for Java-only environments, minimal overhead.
Recommended: SBE Alternative: Cap'n Proto, Kryo
Rationale: Fixed-size encoding, zero-allocation design, sub-millisecond performance.
Recommended: Apache Arrow Alternative: Apache Parquet
Rationale: Columnar format, zero-copy IPC, excellent for in-memory analytics.
Recommended: CBOR Alternative: MessagePack, Apache Avro
Rationale: Compact binary format, low bandwidth requirements, simple implementation.
Recommended: Apache Avro Alternative: Apache Thrift, Cap'n Proto
Rationale: Built-in schema evolution, backward and forward compatibility.
| Requirement | Primary Choice | Alternative | Reason |
|---|---|---|---|
| Fastest serialization | SBE | Cap'n Proto, FST | Sub-millisecond performance |
| Best compression | Apache Parquet | Apache Avro | 75% size reduction |
| Schema evolution | Apache Avro | Thrift, Cap'n Proto | Forward and backward compatibility |
| Ease of use | Jackson JSON | Kryo, MessagePack | Familiar format, extensive tooling |
| Cross-language support | Apache Thrift | Avro, gRPC | Multi-language IDL |
| Zero-copy operations | Cap'n Proto | Apache Arrow | Direct memory access |
| Streaming support | gRPC | Apache Avro with Kafka | Built-in streaming capabilities |
| Analytics workloads | Apache Arrow | Apache Parquet | Columnar format |
Syntax: ./manage.sh <command> [options]
Available Commands:
- start: Start all 13 framework services
- stop: Stop all running services
- restart: Restart all services
- status: Display service health status
- benchmark: Run comprehensive benchmark
- analyze: Analyze latest results
- logs: Tail logs for specific framework
- clean: Remove logs and temporary files
- help: Display help information
Method: GET
URL: http://localhost:{port}/actuator/health
Response:
{
"status": "UP"
}Method: GET
URL: http://localhost:{port}/api/{framework}/v2/info
Response:
{
"framework": "Kryo",
"version": "2.0",
"supportedCompressionAlgorithms": ["GZIP", "SNAPPY"],
"supportsSchemaEvolution": false,
"typicalUseCase": "High-performance Java-only serialization"
}Method: POST
URL: http://localhost:{port}/api/{framework}/v2/benchmark
Content-Type: application/json
Request Parameters:
- complexity: SMALL, MEDIUM, LARGE, or HUGE
- iterations: Number of test iterations (1-1000)
- enableWarmup: Boolean for JVM warm-up
- enableCompression: Boolean for compression testing
- enableRoundtrip: Boolean for serialization verification
- enableMemoryMonitoring: Boolean for memory tracking
| Framework | Port | Health Check URL |
|---|---|---|
| Jackson JSON | 8081 | http://localhost:8081/actuator/health |
| Apache Avro | 8083 | http://localhost:8083/actuator/health |
| Kryo | 8084 | http://localhost:8084/actuator/health |
| MessagePack | 8086 | http://localhost:8086/actuator/health |
| Apache Thrift | 8087 | http://localhost:8087/actuator/health |
| Cap'n Proto | 8088 | http://localhost:8088/actuator/health |
| FST | 8090 | http://localhost:8090/actuator/health |
| gRPC | 8092 | http://localhost:8092/actuator/health |
| CBOR | 8093 | http://localhost:8093/actuator/health |
| BSON | 8094 | http://localhost:8094/actuator/health |
| Apache Arrow | 8095 | http://localhost:8095/actuator/health |
| SBE | 8096 | http://localhost:8096/actuator/health |
| Apache Parquet | 8097 | http://localhost:8097/actuator/health |
Location: {framework}-poc/pom.xml
Example:
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<jvmArguments>
-Xms512m
-Xmx2048m
-XX:+UseG1GC
-XX:MaxGCPauseMillis=200
-XX:+UseStringDeduplication
</jvmArguments>
</configuration>
</plugin>Location: {framework}-poc/src/main/resources/application.yml
Example:
server:
port: 8081
tomcat:
threads:
max: 200
min-spare: 10
max-connections: 8192
spring:
application:
name: jackson-serialization-service
logging:
level:
org.techishthoughts: INFOLocation: enhanced_benchmark.py
Scenarios Configuration:
SCENARIOS = [
{'complexity': 'SMALL', 'iterations': 100},
{'complexity': 'MEDIUM', 'iterations': 50},
{'complexity': 'LARGE', 'iterations': 20},
{'complexity': 'HUGE', 'iterations': 5}
]Benchmark Configurations:
BENCHMARK_CONFIGS = [
{'name': 'baseline', 'enableCompression': False},
{'name': 'compressed', 'enableCompression': True}
]Problem: Services fail to start or show as STOPPED
Diagnosis Steps:
- Check Java version:
java -version(should be Java 21) - Check port conflicts:
lsof -i :8081 - View logs:
./manage.sh logs jacksonortail -f logs/jackson.log
Solutions:
- Restart services:
./manage.sh restart - Kill conflicting processes
- Verify Java 21 installation
Problem: Benchmark fails with timeout or connection errors
Diagnosis Steps:
- Verify service health:
./manage.sh status - Check individual service:
curl http://localhost:8081/actuator/health - Verify Python dependencies:
pip3 list | grep psutil
Solutions:
- Increase timeout in enhanced_benchmark.py (line 169: change 180 to 300)
- Install missing dependencies:
pip3 install psutil - Reduce iterations for initial testing
Problem: OutOfMemoryError or high memory usage
Diagnosis Steps:
- List Java processes:
jps - Check GC statistics:
jstat -gc <pid> 1000 - Monitor heap usage:
jmap -heap <pid>
Solutions:
- Increase JVM heap in pom.xml:
-Xmx4096m - Reduce test iterations
- Enable garbage collection logging
- Monitor memory trends in Grafana
Problem: Grafana dashboards show no data
Diagnosis Steps:
- Verify Prometheus running:
open http://localhost:9090 - Check metrics files exist:
ls -l results/*.prom - Verify Prometheus configuration:
cat /opt/homebrew/etc/prometheus.yml
Solutions:
- Restart Prometheus:
brew services restart prometheus - Re-import dashboards
- Check file permissions on results directory
- Verify data source configuration in Grafana
| Error Message | Cause | Solution |
|---|---|---|
| Connection refused | Service not running | Run ./manage.sh start |
| Port already in use | Port conflict | Kill conflicting process or change port |
| JSON parse error | Malformed response | Check service logs for errors |
| Timeout (180s) | Service overloaded | Reduce iterations or increase timeout |
| ModuleNotFoundError: psutil | Missing Python dependency | Run pip3 install psutil |
Use GitHub Issues for bug reports and feature requests.
- System information (OS, Java version, Maven version)
- Steps to reproduce the issue
- Expected behavior
- Actual behavior
- Relevant logs
- Fork the repository
- Create feature branch:
git checkout -b feature/amazing-feature - Follow existing code style
- Add tests for new features
- Update documentation
- Commit with clear messages
- Push to your fork
- Submit pull request
- Wait for review
- Address feedback
- Create
{framework}-pocmodule based on existing structure - Implement V2 API endpoints
- Add framework to
FRAMEWORKSinmanage.sh - Update
enhanced_benchmark.pywith framework configuration - Test thoroughly with all payload sizes
- Update documentation
- All payload sizes (SMALL, MEDIUM, LARGE, HUGE)
- Both configurations (baseline, with compression)
- Roundtrip verification
- Memory monitoring
- Health check endpoint
Location: docs/README.md
Contents:
- Framework deep-dive navigation
- Quick navigation by use case
- Quick navigation by performance
- Framework comparison matrix
- Reading guide for different audiences
Location: docs/frameworks/
Available:
- Jackson JSON - Complete (442 lines)
- Apache Arrow - Complete (600+ lines)
- Apache Avro - Complete (600+ lines)
- BSON - Complete (740+ lines)
- Cap'n Proto - Complete (700+ lines)
- CBOR - Complete (700+ lines)
- FST - Complete (740+ lines)
- gRPC - Complete (1,260+ lines)
- Kryo - Complete (750+ lines)
- MessagePack - Complete (860+ lines)
- Apache Parquet - Complete (800+ lines)
- SBE - Complete (700+ lines)
- Apache Thrift - Complete (900+ lines)
Location: docs/guides/
Available:
- Benchmark Results - Performance analysis
- Visual Documentation - ASCII diagrams
- Screenshot Guide - Documentation capture
Location: dashboards/README.md
Contents:
- Grafana configuration
- Prometheus setup
- Dashboard import instructions
Issues: GitHub Issues Discussions: GitHub Discussions Email: your-email@example.com
This project is licensed under the MIT License. See the LICENSE file for details.
- Framework maintainers for excellent serialization libraries
- Spring Boot team for the application framework
- Prometheus and Grafana teams for monitoring tools
- All contributors to this project
Last Updated: 2025-10-22 Version: 2.0.0