Add throughput testing infrastructure and fix Python telemetry performance #2742
+2,589
−84
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Throughput Server (examples/throughput_server/): - Standalone TCP/IP server for measuring COSMOS command/telemetry throughput - Dual-port operation for INST (7778) and INST2 (7780) targets - CCSDS packet encoding/decoding with configurable streaming rates - Time-compensated streaming to maintain accurate rates up to 100kHz - Pre-allocated buffers for minimal allocation in hot paths - Raw TCP rate test scripts (Ruby/Python) achieving ~300-500k cmd/s DEMO Plugin Changes: - Add THROUGHPUT_STATUS telemetry packet with rate/count metrics - Add throughput commands: START_STREAM, STOP_STREAM, GET_STATS, RESET_STATS - Add throughput_test procedures for INST (Ruby) and INST2 (Python) - Add throughput screen for real-time monitoring - Add plugin variables to toggle between simulator and throughput server - Configure LengthProtocol for CCSDS packet framing when using throughput server Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add fire-and-forget mode to CommandTopic.send_command when timeout <= 0 to skip ACK waiting for high-throughput command scenarios - Add thread-safe packet caching in TargetModel with 10-second timeout to reduce Redis lookups for repeated packet access - Cache is automatically invalidated when set_packet is called - Add unit tests for packet caching in both Ruby and Python Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Root cause: jsonpath_ng.parse() was recompiling JSONPath expressions on every call, taking ~2.7ms per call. When identifying packets in unique_id_mode (required for INST2 due to mixed CCSDS/JSON packet types), this caused ~5.7ms overhead per packet, limiting throughput to ~320 Hz. Changes: - Add lru_cache to JSONPath parsing in JsonAccessor (103x speedup) - Add orjson as optional dependency for faster JSON parsing - Fix missing self.queued initialization in Python interface_microservice - Update throughput test scripts for both Ruby and Python Results: - Python telemetry: 320 Hz → 3,545 Hz (10x improvement) - Python now outperforms Ruby at high rates (3,545 Hz vs 2,627 Hz) - Zero packet loss maintained at all tested rates Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Revert the bytearray optimization in Python protocols to maintain backward compatibility for custom protocol implementations. The change from bytes to bytearray could break user code that: - Type checks self.data expecting bytes - Relies on immutability of self.data - Uses bytes-specific operations The JSONPath caching fix (the real 10x performance improvement) remains intact. Also adds tests for UPDATE_INTERVAL option in both Ruby and Python interface_microservice to verify the queued writes functionality. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2742 +/- ##
==========================================
- Coverage 79.19% 78.23% -0.96%
==========================================
Files 670 473 -197
Lines 54253 34508 -19745
Branches 734 734
==========================================
- Hits 42967 26999 -15968
+ Misses 11206 7429 -3777
Partials 80 80
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
jmthomas
commented
Jan 18, 2026
| @interface.options.each do |option_name, option_values| | ||
| if option_name.upcase == 'OPTIMIZE_THROUGHPUT' | ||
| # OPTIMIZE_THROUGHPUT was changed to UPDATE_INTERVAL to better represent the setting | ||
| if option_name.upcase == 'UPDATE_INTERVAL' or option_name.upcase == 'OPTIMIZE_THROUGHPUT' |
Member
Author
There was a problem hiding this comment.
This was just a bug ... missing keyword that we already changed in Python and in the docs
| while True: # Loop until we get some data | ||
| try: | ||
| data = self.read_socket.recv(4096, socket.MSG_DONTWAIT) | ||
| data = self.read_socket.recv(65535, socket.MSG_DONTWAIT) |
Member
Author
There was a problem hiding this comment.
Now sure how much of an optimization this is but it matches Ruby
Member
Author
|
Results calculated on my Macbook Pro M3 Max with 36GB RAM with Docker CPU limit 10 and memory limit 16GB RAM. Ruby results: Python results: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds comprehensive throughput testing infrastructure and fixes a critical Python telemetry performance bottleneck, improving Python throughput by 10x.
Key Changes
JsonAccessorimproves throughput from ~320 Hz to 3,545 HzPerformance Results
Python now outperforms Ruby at high telemetry rates while maintaining zero packet loss.
Root Cause Analysis
The Python performance issue was caused by
jsonpath_ng.parse()recompiling JSONPath expressions on every call (~2.7ms per call). When identifying packets inunique_id_mode, this caused ~5.7ms overhead per packet. Addinglru_cacheto cache parsed expressions reduced this to ~0.12µs (23,000x speedup per call).Files Changed
Performance Fixes:
openc3/python/openc3/accessors/json_accessor.py- Add JSONPath caching withlru_cacheopenc3/python/openc3/microservices/interface_microservice.py- Fix missingself.queuedinitializationopenc3/python/pyproject.toml- Add orjson as optional dependencyThroughput Testing Infrastructure:
examples/throughput_server/- New standalone throughput testing serveropenc3-cosmos-demo/targets/INST/procedures/throughput_test.rb- Ruby throughput testopenc3-cosmos-demo/targets/INST2/procedures/throughput_test.py- Python throughput testopenc3-cosmos-demo/targets/*/screens/throughput.txt- Throughput monitoring screensCommand/Telemetry Optimizations:
openc3/lib/openc3/topics/command_topic.rb- Fire-and-forget modeopenc3/python/openc3/topics/command_topic.py- Fire-and-forget modeopenc3/lib/openc3/models/target_model.rb- Packet cachingopenc3/python/openc3/models/target_model.py- Packet cachingTest Coverage:
openc3/spec/microservices/interface_microservice_spec.rb- UPDATE_INTERVAL testopenc3/python/test/microservices/test_interface_microservice.py- UPDATE_INTERVAL testsopenc3/spec/models/target_model_spec.rb- Packet caching testsopenc3/python/test/models/test_target_model.py- Packet caching testsTest plan
🤖 Generated with Claude Code