This repository was archived by the owner on Jan 26, 2026. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 1
Zero copy to main #19
Merged
Merged
Changes from all commits
Commits
Show all changes
57 commits
Select commit
Hold shift + click to select a range
5cb33c9
Update .gitignore
expanded-for-real 63313e4
initial commit for imprint-java
dd4fdbc
initial commit for imprint-java
expanded-for-real bce1d13
Add GitHub Actions CI workflow for automated testing
f5d90b5
Merge remote-tracking branch 'origin/dev' into dev
72c468f
Update GitHub Actions workflow to use upload-artifact@v4
468d682
Add Gradle wrapper validation to CI workflow
cf05b13
Fix gitignore to include gradle-wrapper.jar for CI
d0d7983
Force add gradle-wrapper.jar to repository
f2cdd1b
Update wrapper validation action to v3
57c8249
Fix Javadoc syntax errors and disable strict Javadoc checking
edb3057
Add JMH benchmark .bat and .sh for full suite benchmarking and perfor…
2853e3f
fix map serialization error in benchmark test and streamline ci file …
3a5a113
Add execute permissions back for gradlew in CI
50a288b
Add some more string based performance benchmarks and try to make str…
ea1c4c4
Merge pull request #2 from imprint-serde/faster-strings
expanded-for-real 43cab28
second main commit to address initial commits
expanded-for-real fdb8a56
additional cleanup to address concerns in https://github.com/imprint-…
2e56688
minor style fixes
9353388
minor style fixes again
09d0377
minor style fixes on benchmark tests and supress unused
6209bb1
minor reordering
ace7c67
Merge branch 'main' into dev
4632e01
Full comprehensive comparison tests with a lot of other libraries + s…
3738861
replace deprecated gradle methods with latest
12d2823
Merge Comparisons into dev branch (#8)
expanded-for-real f7a6e8e
Lazy load of directory and header data
2834dbb
Merge remote-tracking branch 'origin/main' into dev
83ed961
minor cleanup
a605b65
minor cleanup
aacddeb
minor cleanup
3bf81ad
Actually fixes offsets and read Byte Values for Maps and Arrays even …
7eaa6e9
change CI file to use JMH plugin to respect iteration and warmup valu…
32640cd
ok plugin didn't work apparently so reverting that and just reducing …
2d882c2
Merge branch 'dev' into lazy-directory
880aeb0
trying to update github ci to make jmh actually work correctly
8831922
lazy directory deserialization
e361cf0
Merge branch 'main' into dev
73eade6
remove extra comments
02866d5
remove extra comments
6278665
Merge branch 'refs/heads/main' into dev
09443eb
Add merge and project APIs; optimize/simplify ImprintBuffers with Tre…
0c7b237
Optimize serialization path and remove ImprintWriter code in favor of…
574323e
Allow for record creation path from builder to bypass extra TreeMapping
b2bebee
Calculate estimated size as fields are added instead of deferring it
f1df8d7
Use idiomatic Directory interface and optimize builder
7420b7f
add large object profiling and refactor tests
4d86447
add Thrift competitor and fix framework issues
a722e45
Add single-field access test
9d0f2c8
correct benchmark methodology for fairness
4b2664c
micro-optiomize and attempt to make ComparisonBenchmark tests a littl…
b4cf85d
final optimization and reorganization into better project structure
cce8994
final optimization and reorganization into better project structure
f06ad98
Merge branch 'main' into zero-copy
50c8a4b
track custom map
eb40310
delete extra operations file because I moved it
b8449c8
adding comments and TODOs
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -42,14 +42,16 @@ dependencies { | |
| // Suppress SLF4J warnings | ||
| jmhImplementation 'org.slf4j:slf4j-nop:1.7.36' | ||
|
|
||
| // Competitor libraries for benchmarking (JMH only) | ||
| // Other serialization libraries for benchmarking (JMH only) | ||
| jmhImplementation 'com.google.protobuf:protobuf-java:3.25.1' | ||
| jmhImplementation 'org.apache.avro:avro:1.11.3' | ||
| jmhImplementation 'com.fasterxml.jackson.core:jackson-databind:2.16.0' | ||
| jmhImplementation 'com.google.flatbuffers:flatbuffers-java:23.5.26' | ||
| jmhImplementation 'com.esotericsoftware:kryo:5.4.0' | ||
| jmhImplementation 'org.msgpack:msgpack-core:0.9.8' | ||
| jmhImplementation 'org.msgpack:jackson-dataformat-msgpack:0.9.8' | ||
| jmhImplementation 'org.apache.thrift:libthrift:0.19.0' | ||
| jmhImplementation 'javax.annotation:javax.annotation-api:1.3.2' | ||
| } | ||
|
|
||
| protobuf { | ||
|
|
@@ -137,20 +139,74 @@ tasks.register('generateFlatBuffers', Exec) { | |
| } | ||
| } | ||
|
|
||
| // Task to download the Thrift compiler | ||
| tasks.register('downloadThrift', Exec) { | ||
| description = 'Download Thrift compiler' | ||
| group = 'build setup' | ||
|
|
||
| def thriftVersion = "0.19.0" | ||
| def thriftExecutable = file("${buildDir}/thrift/thrift.exe") | ||
| def thriftUrl = "https://archive.apache.org/dist/thrift/${thriftVersion}/thrift-${thriftVersion}.exe" | ||
|
|
||
| outputs.file(thriftExecutable) | ||
|
|
||
| onlyIf { | ||
| !thriftExecutable.exists() && System.getProperty('os.name').toLowerCase().contains('windows') | ||
| } | ||
|
|
||
| doFirst { | ||
| println "Downloading Thrift compiler for Windows from $thriftUrl..." | ||
| thriftExecutable.parentFile.mkdirs() | ||
| } | ||
|
|
||
| commandLine 'curl', '-L', '-o', thriftExecutable.absolutePath, thriftUrl | ||
|
|
||
| doLast { | ||
| println "Thrift compiler downloaded to: ${thriftExecutable}" | ||
| } | ||
| } | ||
|
|
||
| // Task to generate Java code from Thrift IDL files for JMH benchmarks | ||
| tasks.register('generateJmhThrift', Exec) { | ||
| dependsOn tasks.downloadThrift | ||
| description = 'Generate Java classes from Thrift schema' | ||
| group = 'build' | ||
|
|
||
| def thriftExecutable = file("${buildDir}/thrift/thrift.exe") | ||
| def schemaFile = file('src/jmh/thrift/test_record.thrift') | ||
| def outputDir = file('build/generated-src/thrift/jmh/java') | ||
|
|
||
| // Only run if the thrift executable exists (i.e., on Windows) | ||
| onlyIf { thriftExecutable.exists() } | ||
|
|
||
| commandLine thriftExecutable.absolutePath, '-r', '--gen', 'java', '-o', outputDir.absolutePath, schemaFile.absolutePath | ||
|
|
||
| inputs.file(schemaFile) | ||
| outputs.dir(outputDir) | ||
|
|
||
| doFirst { | ||
| outputDir.mkdirs() | ||
| } | ||
| } | ||
|
|
||
|
|
||
| // Add generated FlatBuffers sources to JMH source set | ||
| sourceSets { | ||
| jmh { | ||
| java { | ||
| srcDir 'build/generated/source/flatbuffers/jmh/java' | ||
| srcDir 'build/generated-src/thrift/jmh/java' | ||
| srcDir 'build/generated/sbe/java' | ||
|
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Tried to add SBE but gave up. I can revisit it in a later issue though (and the XML based schema makes me regret ever not liking Avro's json based one). I'm actually curious how it would do on Merge and Project, especially compared to flatbuffers. I can't imagine it's that much better than flatbuffers but could be interesting. |
||
| } | ||
| proto { | ||
| srcDir 'src/jmh/proto' | ||
| } | ||
| } | ||
| } | ||
|
|
||
| // Make JMH compilation depend on FlatBuffers generation | ||
| // Make JMH compilation depend on generation tasks | ||
| compileJmhJava.dependsOn generateFlatBuffers | ||
| compileJmhJava.dependsOn generateJmhThrift | ||
|
|
||
| // Handle duplicate proto files | ||
| tasks.named('processJmhResources') { | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,15 +1,14 @@ | ||
| namespace com.imprint.benchmark; | ||
| namespace com.imprint.benchmark.flatbuffers; | ||
|
|
||
| table TestRecordFB { | ||
| id: int; | ||
| name: string; | ||
| price: double; | ||
| table TestRecord { | ||
| id: string; | ||
| timestamp: long; | ||
| flags: int; | ||
| active: bool; | ||
| category: string; | ||
| tags: [string]; | ||
| metadata_keys: [string]; | ||
| metadata_values: [string]; | ||
| extra_data: [string]; | ||
| value: double; | ||
| data: [ubyte]; | ||
| tags: [int]; | ||
| metadata: [string]; // Representing map as a flat list of key/value strings for simplicity | ||
| } | ||
|
|
||
| root_type TestRecordFB; | ||
| root_type TestRecord; |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thrift was surprisingly fast but like the other binary formats it required quite a bit of gradle task work and it's own compiler