This repository was archived by the owner on Jan 26, 2026. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 1
Use Unsafe memory buffer instead of byte buffer - https://github.com/imprint-serde/imprint-java/issues/21 #24
Merged
Merged
Changes from all commits
Commits
Show all changes
71 commits
Select commit
Hold shift + click to select a range
5cb33c9
Update .gitignore
expanded-for-real 63313e4
initial commit for imprint-java
dd4fdbc
initial commit for imprint-java
expanded-for-real bce1d13
Add GitHub Actions CI workflow for automated testing
f5d90b5
Merge remote-tracking branch 'origin/dev' into dev
72c468f
Update GitHub Actions workflow to use upload-artifact@v4
468d682
Add Gradle wrapper validation to CI workflow
cf05b13
Fix gitignore to include gradle-wrapper.jar for CI
d0d7983
Force add gradle-wrapper.jar to repository
f2cdd1b
Update wrapper validation action to v3
57c8249
Fix Javadoc syntax errors and disable strict Javadoc checking
edb3057
Add JMH benchmark .bat and .sh for full suite benchmarking and perfor…
2853e3f
fix map serialization error in benchmark test and streamline ci file …
3a5a113
Add execute permissions back for gradlew in CI
50a288b
Add some more string based performance benchmarks and try to make str…
ea1c4c4
Merge pull request #2 from imprint-serde/faster-strings
expanded-for-real 43cab28
second main commit to address initial commits
expanded-for-real fdb8a56
additional cleanup to address concerns in https://github.com/imprint-…
2e56688
minor style fixes
9353388
minor style fixes again
09d0377
minor style fixes on benchmark tests and supress unused
6209bb1
minor reordering
ace7c67
Merge branch 'main' into dev
4632e01
Full comprehensive comparison tests with a lot of other libraries + s…
3738861
replace deprecated gradle methods with latest
12d2823
Merge Comparisons into dev branch (#8)
expanded-for-real f7a6e8e
Lazy load of directory and header data
2834dbb
Merge remote-tracking branch 'origin/main' into dev
83ed961
minor cleanup
a605b65
minor cleanup
aacddeb
minor cleanup
3bf81ad
Actually fixes offsets and read Byte Values for Maps and Arrays even …
7eaa6e9
change CI file to use JMH plugin to respect iteration and warmup valu…
32640cd
ok plugin didn't work apparently so reverting that and just reducing …
2d882c2
Merge branch 'dev' into lazy-directory
880aeb0
trying to update github ci to make jmh actually work correctly
8831922
lazy directory deserialization
e361cf0
Merge branch 'main' into dev
73eade6
remove extra comments
02866d5
remove extra comments
6278665
Merge branch 'refs/heads/main' into dev
09443eb
Add merge and project APIs; optimize/simplify ImprintBuffers with Tre…
0c7b237
Optimize serialization path and remove ImprintWriter code in favor of…
574323e
Allow for record creation path from builder to bypass extra TreeMapping
b2bebee
Calculate estimated size as fields are added instead of deferring it
f1df8d7
Use idiomatic Directory interface and optimize builder
7420b7f
add large object profiling and refactor tests
4d86447
add Thrift competitor and fix framework issues
a722e45
Add single-field access test
9d0f2c8
correct benchmark methodology for fairness
4b2664c
micro-optiomize and attempt to make ComparisonBenchmark tests a littl…
b4cf85d
final optimization and reorganization into better project structure
cce8994
final optimization and reorganization into better project structure
f06ad98
Merge branch 'main' into zero-copy
50c8a4b
track custom map
eb40310
delete extra operations file because I moved it
b8449c8
adding comments and TODOs
96fbc20
various micro-optimizations
686a855
make serilaizers static/final; begin to refactor to avoid virtual dis…
ffc7918
add new Imprint specific benchmark
e562ac3
Remove Value and TypeHandler to significantly reduce dynamic dispatch
999f48c
add static serializers
3b0a0d1
Add Unsafe direct buffer wrapper class
8c54405
Try to get merge to use the new direct/unsafe/growable imprint buffer
1148079
update comments
541eddd
convert operations to use expandable buffer
0f7b52b
finalize using unsafe buffer everywhere
60aff65
fix formatting
8a8fbc5
fix comparison tests
69f66d9
Merge branch 'main' into custom-buffers
d3369cf
remove extra comments and debug outputs
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -16,8 +16,15 @@ | |
| @OutputTimeUnit(TimeUnit.NANOSECONDS) | ||
| @State(Scope.Benchmark) | ||
| @Warmup(iterations = 3, time = 1) | ||
| @Measurement(iterations = 25, time = 1) | ||
| @Fork(value = 1, jvmArgs = {"-Xms4g", "-Xmx4g"}) | ||
| @Measurement(iterations = 10, time = 1) | ||
| @Fork(value = 1, jvmArgs = {"-Xms4g", "-Xmx4g", | ||
| "--illegal-access=permit", | ||
|
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ChronicleWire apparently needs all this |
||
| "--add-exports=java.base/jdk.internal.ref=ALL-UNNAMED", | ||
| "--add-opens=java.base/java.lang=ALL-UNNAMED", | ||
| "--add-opens=java.base/java.lang.reflect=ALL-UNNAMED", | ||
| "--add-opens=java.base/java.util=ALL-UNNAMED", | ||
| "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED", | ||
| "-Dimprint.buffer.bounds.check=false"}) | ||
| public class ComparisonBenchmark { | ||
|
|
||
| private static final List<SerializingBenchmark> FRAMEWORKS = List.of( | ||
|
|
@@ -28,9 +35,10 @@ public class ComparisonBenchmark { | |
| new AvroSerializingBenchmark(), | ||
| new ThriftSerializingBenchmark(), | ||
| new KryoSerializingBenchmark(), | ||
| new MessagePackSerializingBenchmark()); | ||
| new MessagePackSerializingBenchmark(), | ||
| new ChronicleWireSerializingBenchmark()); | ||
|
|
||
| @Param({"Imprint"}) | ||
| @Param({"Imprint", "Jackson-JSON", "Protobuf", "FlatBuffers", "Avro-Generic", "Thrift", "Kryo", "MessagePack", "Chronicle-Wire"}) | ||
| public String framework; | ||
|
|
||
| private SerializingBenchmark serializingBenchmark; | ||
|
|
@@ -51,27 +59,27 @@ public void setup() { | |
| } | ||
|
|
||
| @Benchmark | ||
| public void serialize(Blackhole bh) { | ||
| public void serializeRecord(Blackhole bh) { | ||
| serializingBenchmark.serialize(bh); | ||
| } | ||
|
|
||
| //@Benchmark | ||
| public void deserialize(Blackhole bh) { | ||
| @Benchmark | ||
| public void deserializeRecord(Blackhole bh) { | ||
| serializingBenchmark.deserialize(bh); | ||
| } | ||
|
|
||
| //@Benchmark | ||
| public void projectAndSerialize(Blackhole bh) { | ||
| @Benchmark | ||
| public void projectThenSerialize(Blackhole bh) { | ||
| serializingBenchmark.projectAndSerialize(bh); | ||
| } | ||
|
|
||
| //@Benchmark | ||
| public void mergeAndSerialize(Blackhole bh) { | ||
| @Benchmark | ||
| public void mergeThenSerialize(Blackhole bh) { | ||
| serializingBenchmark.mergeAndSerialize(bh); | ||
| } | ||
|
|
||
| //@Benchmark | ||
| public void accessField(Blackhole bh) { | ||
| @Benchmark | ||
| public void accessSingleField(Blackhole bh) { | ||
| serializingBenchmark.accessField(bh); | ||
| } | ||
|
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
163 changes: 163 additions & 0 deletions
163
src/jmh/java/com/imprint/benchmark/serializers/ChronicleWireSerializingBenchmark.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,163 @@ | ||
| package com.imprint.benchmark.serializers; | ||
|
|
||
| import com.imprint.benchmark.DataGenerator; | ||
| import net.openhft.chronicle.bytes.Bytes; | ||
| import net.openhft.chronicle.wire.BinaryWire; | ||
| import net.openhft.chronicle.wire.Wire; | ||
| import net.openhft.chronicle.wire.WireType; | ||
| import org.openjdk.jmh.infra.Blackhole; | ||
|
|
||
| import java.util.List; | ||
| import java.util.Map; | ||
|
|
||
| public class ChronicleWireSerializingBenchmark extends AbstractSerializingBenchmark { | ||
|
|
||
| private byte[] serializedRecord1; | ||
|
|
||
| public ChronicleWireSerializingBenchmark() { | ||
| super("Chronicle-Wire"); | ||
| } | ||
|
|
||
| @Override | ||
| public void setup(DataGenerator.TestRecord record1, DataGenerator.TestRecord record2) { | ||
| super.setup(record1, record2); | ||
|
|
||
| // Pre-serialize for deserialize benchmarks | ||
| this.serializedRecord1 = serializeRecord(record1); | ||
| byte[] serializedRecord2 = serializeRecord(record2); | ||
| } | ||
|
|
||
| @Override | ||
| public void serialize(Blackhole bh) { | ||
| byte[] serialized = serializeRecord(testData); | ||
| bh.consume(serialized); | ||
| } | ||
|
|
||
| @Override | ||
| public void deserialize(Blackhole bh) { | ||
| DataGenerator.TestRecord deserialized = deserializeRecord(serializedRecord1); | ||
| bh.consume(deserialized); | ||
| } | ||
|
|
||
| @Override | ||
| public void projectAndSerialize(Blackhole bh) { | ||
| // Full round trip: deserialize, project to a new object, re-serialize | ||
| DataGenerator.TestRecord original = deserializeRecord(serializedRecord1); | ||
|
|
||
| // Simulate projection by creating projected object | ||
| DataGenerator.ProjectedRecord projected = new DataGenerator.ProjectedRecord(); | ||
| projected.id = original.id; | ||
| projected.timestamp = original.timestamp; | ||
| projected.tags = original.tags.subList(0, Math.min(5, original.tags.size())); | ||
|
|
||
| byte[] serialized = serializeProjectedRecord(projected); | ||
| bh.consume(serialized); | ||
| } | ||
|
|
||
| @Override | ||
| public void mergeAndSerialize(Blackhole bh) { | ||
| // Deserialize both records, merge them, and serialize the result | ||
| DataGenerator.TestRecord r1 = deserializeRecord(serializedRecord1); | ||
| DataGenerator.TestRecord r2 = testData2; // Use second record directly | ||
|
|
||
| // Create merged record following the pattern from other implementations | ||
| DataGenerator.TestRecord merged = new DataGenerator.TestRecord(); | ||
| merged.id = r1.id; | ||
| merged.timestamp = System.currentTimeMillis(); // new value | ||
| merged.flags = r1.flags; | ||
| merged.active = false; // new value | ||
| merged.value = r1.value; | ||
| merged.data = r1.data; | ||
| merged.tags = r2.tags; | ||
| merged.metadata = r2.metadata; | ||
|
|
||
| byte[] serialized = serializeRecord(merged); | ||
| bh.consume(serialized); | ||
| } | ||
|
|
||
| @Override | ||
| public void accessField(Blackhole bh) { | ||
| DataGenerator.TestRecord deserialized = deserializeRecord(serializedRecord1); | ||
| long timestamp = deserialized.timestamp; | ||
| bh.consume(timestamp); | ||
| } | ||
|
|
||
| private byte[] serializeRecord(DataGenerator.TestRecord record) { | ||
| Bytes<?> bytes = Bytes.elasticByteBuffer(); | ||
| try { | ||
| Wire wire = WireType.BINARY.apply(bytes); | ||
|
|
||
| wire.writeDocument(false, w -> { | ||
| if (record.id != null) w.write("id").text(record.id); | ||
| w.write("timestamp").int64(record.timestamp) | ||
| .write("flags").int32(record.flags) | ||
| .write("active").bool(record.active) | ||
| .write("value").float64(record.value); | ||
|
|
||
| if (record.data != null) { | ||
| w.write("data").bytes(record.data); | ||
| } | ||
| if (record.tags != null) { | ||
| w.write("tags").object(record.tags); | ||
| } | ||
| if (record.metadata != null) { | ||
| w.write("metadata").marshallable(m -> { | ||
| for (Map.Entry<String, String> entry : record.metadata.entrySet()) { | ||
| m.write(entry.getKey()).text(entry.getValue()); | ||
| } | ||
| }); | ||
| } | ||
| }); | ||
|
|
||
| byte[] result = new byte[(int) bytes.readRemaining()]; | ||
| bytes.read(result); | ||
| return result; | ||
| } finally { | ||
| bytes.releaseLast(); | ||
| } | ||
| } | ||
|
|
||
| private byte[] serializeProjectedRecord(DataGenerator.ProjectedRecord record) { | ||
| Bytes<?> bytes = Bytes.elasticByteBuffer(); | ||
| try { | ||
| Wire wire = WireType.BINARY.apply(bytes); | ||
|
|
||
| wire.writeDocument(false, w -> { | ||
| if (record.id != null) w.write("id").text(record.id); | ||
| w.write("timestamp").int64(record.timestamp); | ||
| if (record.tags != null) { | ||
| w.write("tags").object(record.tags); | ||
| } | ||
| }); | ||
|
|
||
| byte[] result = new byte[(int) bytes.readRemaining()]; | ||
| bytes.read(result); | ||
| return result; | ||
| } finally { | ||
| bytes.releaseLast(); | ||
| } | ||
| } | ||
|
|
||
| private DataGenerator.TestRecord deserializeRecord(byte[] data) { | ||
| Bytes<?> bytes = Bytes.wrapForRead(data); | ||
| try { | ||
| Wire wire = new BinaryWire(bytes); | ||
| DataGenerator.TestRecord record = new DataGenerator.TestRecord(); | ||
|
|
||
| wire.readDocument(null, w -> { | ||
| record.id = w.read("id").text(); | ||
| record.timestamp = w.read("timestamp").int64(); | ||
| record.flags = w.read("flags").int32(); | ||
| record.active = w.read("active").bool(); | ||
| record.value = w.read("value").float64(); | ||
| record.data = w.read("data").bytes(); | ||
| record.tags = (List<Integer>) w.read("tags").object(); | ||
| record.metadata = w.read("metadata").marshallableAsMap(String.class, String.class); | ||
| }); | ||
|
|
||
| return record; | ||
| } finally { | ||
| bytes.releaseLast(); | ||
| } | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Got frustrated halfway through the effort and decided to add ChronicleWire as a comparison