Add bitSplit codec for splitting numeric values by bit ranges #355
+2,279
−9
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Summary: Introduces a new core bitSplit transform that splits numeric input elements into multiple output streams based on configurable bit widths. This is a core transform, that will then be used to generate multiple nodes, all compatible with the same decoder transform. The encoder extracts consecutive bit ranges (LSB to MSB order) from each input element into separate typed streams. The decoder reconstructs the original values by combining the bit ranges. Supports partial coverage where sum(widths) < element_width, with top bits required to be zero. The specific nodes that can be created from bitSplit include, for example, Floating component separation (sign, exponent, mantissa), for all size and formats (FP32, FP16, BF16, etc.). But the main use case I want to use this transform for is to replace a bunch of weird corner cases where the `zigzag` transform is employed as a sort of "poor man's bitshift" to produce a certain effect _in combination with_ `transpose`. According to analysis I did recently on a bunch of samples, this is very common in compressor graphs created by ACE. All of this could be replaced by a single evocation of a `bitSplit` derivative node, no only being faster and a little bit more compact, but more importantly providing a clear decision, as opposed to a muddy one emulated through multiple abused stages. Test Plan: make gtests && ./gtests --gtest_filter="*BitSplit*" All 28 tests pass covering: - Full coverage splits (8/16/32/64-bit inputs) - Partial coverage cases - Single stream passthrough - Edge cases (empty input, single element, all zeros/ones) - Asymmetric and many-small-width splits
for capabilities shared between compression and decompression
to be pure C11 with no dependencies
unused variable when debug is disabled
and removed local config file
use ZL_memcpy, not `memcpy` directly
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Introduces a new core
bitSplittransform that splits a numeric stream into multiple output streams according to a configurable set of bit widths.As a core transform,
bitSplitcan be instantiated to produce multiple nodes that all share a common, compatible decoder.On the encode path, the transform extracts consecutive bit ranges from each element (from LSB to MSB) into separate streams. On the decode path, it reconstructs the original values by reassembling these ranges. Partial coverage is supported (
sum(widths) < element_width); in that case, the remaining high bits must be zero.Performance depends on the number of ranges and the total covered bit width, but is on the order of ~1 GB/s on my devserver for both encoding and decoding.
bitSplitalso enables specialized nodes, e.g. floating-point component separation (sign, exponent, mantissa) across formats and sizes (FP32, FP16, BF16, etc.). Internally, it supports specialized instances to improve performance for common scenarios.This patch ships a BF16-specialized instance. On my devserver it reaches ~30 GB/s on decode and ~40 GB/s on encode using plain C; such performance requires vectorization, and the compiler auto-vectorizes the hot loop with
AVX2.The initial motivation is to replace several corner-case graphs where
zigzagis effectively used as a “poor man’s bitshift” in combination withtranspose. Recent sample analysis shows this pattern is common in compressor graphs produced byACE. These multi-stage constructs can be replaced with a singlebitSplitnode, improving throughput, slightly reducing output size, and providing a clearer decision signal than the current indirect emulation.Test Plan:
All 28 tests pass covering: