Chronicle Wire test backlog, cross-module regression coverage, and ops/docs scaffold — with tuned JaCoCo gates

TESTS_TODO.md

@@ -0,0 +1,49 @@
+# Chronicle Wire Test Backlog
+
+Purpose: capture the cross-module scenarios identified during the peer-project review so we can
+add targeted regression tests inside Chronicle Wire (`src/test/java/net/openhft/chronicle/wire`).
+All entries below include the external context that triggered the gap analysis to keep intent clear.
+
+| ID | Focus Area | External Context | Proposed Tests | Notes / Owners |
+|----|------------|------------------|----------------|----------------|
+| W-T1 | `DocumentContext` rollback + metadata handling | Queue tailers (`Chronicle-Queue/src/main/java/net/openhft/chronicle/queue/impl/single/BinarySearch.java`) rely on `DocumentContext.rollbackOnClose()` when a read loop does not advance. | Add junit cases under `DocumentContextLifecycleTest` that write nested metadata documents, forcibly exit without consuming bytes, and assert the next read resumes at the correct index for all `WireType`s. | Exercise both `readingDocument(true)` and `readingDocument(false)` paths; ensures queue search stays safe. |
+| W-T2 | Comparator-driven wire search | Queue binary search compares arbitrary `Wire` snippets and expects `NotComparableException` handling. | Introduce a `WireComparatorTest` that registers custom comparators, throws `NotComparableException`, and verifies the caller can reset the source `Bytes` safely. | Validates Chronicle Wire statefulness before downstream queue search loops. |
+| W-T3 | Method-reader history interception | Ring buffer services (`Chronicle-Ring/src/main/java/software/chronicle/enterprise/ring/internal/EnterpriseRingBuffer.java`) toggle `MessageHistory` interceptors created via `methodWriterBuilder`. | Extend `MethodReaderBuilderTest` (or create `MethodReaderHistoryTest`) to wire up an interceptor, mutate `MessageHistory`, and assert the generated proxy writes the captured history into the supplied `Bytes`. | Covers `TextWire.useTextDocuments()` special case triggered in `MethodReaderQueueEntryReader`. |
+| W-T4 | `WireType`-agnostic replays | `WireTcpHandler` in Chronicle Network decides between `BINARY` and `TEXT` at runtime. | Add parametrised tests that serialise the same payload with each `WireType`, feed it into `WireType.fromGrid` detection helpers (or equivalent APIs), and assert decoding remains deterministic even when the first payload byte matches both families. | Prevents silent regressions in auto-detection heuristics. |
+| W-T5 | Size-prefixed blob inspection | Chronicle Network/Queue components call `Wires.fromSizePrefixedBlobs` for diagnostics. | Create a `WiresSizePrefixedTest` that feeds corrupted headers (negative lengths, truncated blobs) and confirms the helper throws predictable exceptions instead of looping. | Aligns with `WireTcpHandler.logYaml` usage. |
+| W-T6 | Deterministic `BinaryWire` hashing | Chronicle Queue relies on stable binary hashes for ledger signatures. Existing tests only cover scalar fields. | Introduce golden-file tests that encode nested documents (anchors, sequences, class aliases) in `BinaryWire`, hash the bytes, and assert the hash stays stable across JVMs. | Re-use `BinaryWireTest` infrastructure; helps ChronMap replication too. |
+| W-T7 | Long/time converter boundaries | Converters like `MilliTimestampLongConverter` are used by `NetworkStats` and Chronicle Services SLAs. | Expand `LongConversionTest` to include overflow, negative timestamps, and non-decimal input strings (base32/base64). Confirm converters reject malformed data with descriptive exceptions. | Ensures peer modules can rely on zero-copy parsing. |
+| W-T8 | `WireDumper` resilience | Chronicle Queue Zero and diagnostics dump wires even when the header is unfinished. | Add `WireDumperTest` cases feeding incomplete `BytesStore` instances (length header only, body missing) to guarantee we produce bounded output and no `IndexOutOfBoundsException`. | Mirrors `ChronicleQueueZero.waitTillHeaderHasBeenFullyWritten` behaviour. |
+| W-T9 | Encrypted document passthrough | Queue builder exposes `codingSuppliers`, but Chronicle Wire lacks a direct encryption round-trip test. | Create a lightweight AES-based `Bytes` transformer in `wire` tests that wraps a `Wire` with encode/decode lambdas, verifying size-prefixed blobs remain consistent when encrypting `WireOut` then immediately decrypting to `WireIn`. | Gives confidence that downstream encryption hooks keep the format intact. |
+| W-T10 | Method writer proxy recycling | Ring buffer and queue modules demand zero-GC proxies even when the same interface is rebuilt per document. | Enhance `GenerateMethodWriter2CoverageTest` to repeatedly create/destroy method writers under heap pressure, asserting no stale classloader leaks and that intercepted invocations keep order. | Protects Chronicle Logger and high-volume queue writers. |
+| W-T11 | `ElasticByteBufferTest` padding & direct memory | Batch 5 plan in `Chronicle-Bytes/TESTS_TODO.md` flagged missing coverage for padding toggles and allocator growth. | Add parameterised cases that allocate direct buffers, flip `wire.usePadding(true/false)`, and assert capacity growth plus position resets when documents exceed the initial reservation. | Mirrors the instructions captured in `NEW_TESTS.md`; keep assertions tied to SPB helpers for clarity. |
+| W-T12 | `MethodWriterBytesTest` payload stability | Need to prove method writers snapshot or copy `Bytes` arguments so producers can reuse buffers. | Write tests that reuse a single `Bytes` instance across multiple dispatches, mutating it immediately after `methodWriter` calls, and assert consumers still observe the original payload (or a controlled failure). Include a negative test where mutation mid-read triggers a deterministic exception. | Validates Chronicle Queue expectations about method-writer isolation. |
+| W-T13 | `QueryWireTest` URI edge cases | QueryWire currently lacks coverage for percent-encoded characters, embedded NULs, and truncated fragments; cross-format interoperability is also untested. | Extend the suite with percent-encoded inputs, zero bytes, and truncated fragments, asserting `readPosition`/`writePosition` behaviour. Add a cross-wire proof: serialise with `QueryWire`, read back via `TextWire` or `BinaryWire`, and compare field equality. | Based on the Batch 5 worklog in `NEW_TESTS.md`. |
+| W-T14 | `WireTextBugTest` direct-bytes regression | The historical WireText bug only reproduces with direct `Bytes` and mutable objects. | Replicate the regression using `Bytes.allocateDirect`, then attempt to mutate the decoded POJO to ensure the text wire returns immutable views (or fails fast). Include assertions that no writable alias remains after reading. | Ensures the fix stays covered on both heap and direct buffers. |
+| W-T15 | `WireMarshaller` alias resolution | Chronicle Services config files rely on `@TypeAlias`/`@EnumAlias` to stay backwards compatible when field names change. | Add tests that serialise/deserialise enums and POJOs with multiple aliases (case changes, deprecated names) across `BinaryWire`, `TextWire`, and `YAMLWire`, asserting the correct type is chosen and unknown aliases surface descriptive errors. | Prevents config migrations from silently deserialising to wrong types. |
+| W-T16 | Tagged field schema evolution | Queue/Wire replication now emits tagged field numbers to allow forward compatibility. | Create a `TaggedFieldWireCompatibilityTest` that writes documents containing unknown tags, replays them with older schemas, and asserts skipped tags do not corrupt subsequent reads; include reverse tests to guarantee new readers can consume old encodings. | Mirrors Chronicle Queue’s requirement for flawless schema upgrades. |
+| W-T17 | `WireIn.peekXXX` idempotence | Services’ HA replay logic peeks into `WireIn` to decide whether to roll back or consume. | Extend `WireInPeekTest` covering `peekType`, `peekEvent`, and `peekDocumentLength` ensuring repeated peeks leave positions unchanged for all `WireType`s, especially when encryption/transforms are enabled. | Guards against subtle pointer drift noted during HA investigations. |
+| W-T18 | BinaryLight delta compression parity | Chronicle Queue’s `WireType.BINARY_LIGHT` plus delta optimisations lack Wire-level tests verifying partial field updates. | Add parameterised tests that encode a baseline document, then send deltas updating only subsets of fields; assert `BinaryLightWire` reconstructs the combined state and that `toString()` matches `BinaryWire` equivalents. | Needed for replication streams that mix full + delta payloads. |
+
+Next steps:
+1. Prioritise W-T1 → W-T4 to unblock queue/network feature work.
+2. Track each item against a JIRA/GitHub issue so the Decision Log can cite the test coverage improvements once implemented.
+
+## Additional Ideas (Queue-derived backlog)
+| Batch | Scope | Goal | Notes |
+|------|-------|------|-------|
+| W-07 | SPB corruption matrix | Add Wire-level tests that feed truncated SPB frames, zero-length payloads, and mismatched length prefixes through `BinaryWire` and `TextWire`, asserting `DocumentContext.isPresent()`/`isData()` mirror queue expectations. | Mirrors the queue regressions around zero-length documents and partial frames; keeps Wire primitives aligned. |
+| W-08 | QueueOffsetSpec formatting helpers | Add Wire-facing tests (or docs) showing how `QueueOffsetSpec` strings are embedded in YAML/JSON configs, ensuring Wire’s parser handles the same tokens (epoch, roll time, none). | Keeps the format stable for tools that rely on Wire serialization (e.g., config files). |
+
+## Chronicle Bytes Test Batches
+
+The Bytes library shares many primitives with Wire, so we need forward-looking plans that keep allocator, reference-counting and encoding behaviour in lockstep with Queue, Map and Network modules.
+
+| ID | Focus Area | External Context | Proposed Tests | Notes / Owners |
+|----|------------|------------------|----------------|----------------|
+| B-T1 | Reference counting + `ReferenceOwner` interplay | Chronicle Queue, Chronicle Map and Chronicle FIX all rely on deterministic `BytesStore` release to avoid direct-memory leaks. | Add a `BytesReferenceCountingTest` (heap + direct) that mixes `reserve`, `release`, `releaseLast` and `ReferenceOwner` scopes across multiple threads, asserting the counter never underflows and that leaked owners are reported via `Jvm.warn()` hooks. | Covered by `BytesReferenceCountingTest` (heap/direct); extend later if explicit warn-hook assertions are required. |
+| B-T2 | Elastic capacity growth + padding toggles | Queue tailers now pass the same `Bytes` into Wire and Bytes-level SPB helpers; inconsistent `realCapacity` updates have caused stalls. | Introduce `ElasticBytesCapacityTest` parameterised by allocator (heap/direct), growth factor and padding flags. Write payloads that exceed the initial capacity, assert `realCapacity` increases monotonically, and verify `readPosition`/`writeLimit` are restored after slicing. | Addressed by `ElasticBytesCapacityTest`; add performance assertions if future regressions demand it. |
+| B-T3 | Memory-mapped page rollover | `MappedBytes` backs queue stores and must honour page boundary calculations when `writePosition` jumps. | Create a `MappedBytesRolloverTest` that maps a tiny file (<2 pages), writes data straddling the boundary, forces `MappedBytes#resize`, and asserts reads before/after the rollover stay consistent without double-mapping. Include negative tests for truncated files. | Helps prevent regressions seen in Chronicle Queue enterprise warm-up runs. |
+| B-T4 | Thread-safe cursor sharing | Low-latency services often share one `Bytes` between producer/consumer loops using `BytesStore` slicing. | Add `SharedBytesCursorTest` that spawns paired threads: producer updates `writePosition`, consumer polls `readRemaining`. Verify CAS-based helpers (`Bytes#writeLimit`, `Bytes#writePositionVolatile`) prevent torn reads and detect misuse via `IllegalStateException`. | Aligns with `Chronicle-Network` `WireTcpHandler` backpressure logic. |
+| B-T5 | Encoding + converter boundaries | Network telemetry uses `Bytes.appendUtf8`, `parseUtf8` and converter APIs (`DecimalLongConverter`). | Extend `Utf8AndConverterTest` to cover malformed UTF-8 sequences, surrogate pairs (reject), ISO-8859-1 only mode, and converter overflow/underflow for decimal/hex encoders. Assert exceptions carry the failing offset for rapid diagnostics. | Reproduces issues highlighted in Services SLA testing. |
+| B-T6 | Zero-copy blob inspection helpers | `Bytes` exposes `typedBuffer`, `toTemporaryDirectByteBuffer`, and checksum utilities used by diagnostics. | Create `BytesBlobInspectionTest` that allocates heap/direct buffers, calls the zero-copy helpers, mutates the underlying storage, and ensures the exposed `ByteBuffer`s reflect changes while respecting bounds. Include checksum verification for corrupted blobs. | Supports Chronicle Diagnostics’ crash dump tooling; owner TBD. |

pom.xml

@@ -4,27 +4,26 @@
     Copyright 2013-2025 chronicle.software; SPDX-License-Identifier: Apache-2.0
 
 -->
-<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
-         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
     <modelVersion>4.0.0</modelVersion>
 
     <parent>
         <groupId>net.openhft</groupId>
         <artifactId>java-parent-pom</artifactId>
-        <version>1.27ea2-SNAPSHOT</version>
+        <version>1.27ea1</version>
         <relativePath/>
     </parent>
 
     <artifactId>chronicle-wire</artifactId>
-    <version>2.27ea13-SNAPSHOT</version>
+    <version>2.27ea14-SNAPSHOT</version>
     <name>OpenHFT/Chronicle-Wire</name>
     <!-- Core library for Chronicle's binary wire serialization -->
     <description>Chronicle-Wire</description>
     <packaging>bundle</packaging>
     <properties>
         <!-- Define the target coverage thresholds as properties -->
-        <jacoco.line.coverage>0.7</jacoco.line.coverage>
-        <jacoco.branch.coverage>0.6</jacoco.branch.coverage>
+        <jacoco.line.coverage>0.72</jacoco.line.coverage>
+        <jacoco.branch.coverage>0.66</jacoco.branch.coverage>
     </properties>
 
     <!-- Use BOMs to keep dependency versions consistent across modules -->

src/main/docs/background/openhft-knowledge-pack.adoc

@@ -0,0 +1,56 @@
+= OpenHFT Knowledge Pack for Chronicle Wire
+Chronicle Software
+:toc:
+:sectnums:
+:lang: en-GB
+:source-highlighter: rouge
+
+== Purpose
+
+This background note captures institutional knowledge about the OpenHFT ecosystem that Chronicle Wire
+depends on. It informs performance tuning, dependency management, and support policies.
+
+== Dependency Baseline
+
+[cols="1,2,2", options="header"]
+|===
+|Library |Notes |Tracking
+
+|`net.openhft:chronicle-bytes`
+|Provides the low-level `Bytes` abstraction used by every wire type. Chronicle Wire tracks the latest
+release from the `ea` stream but keeps compatibility with the most recent GA.
+|Chronicle BOM plus `project-requirements.adoc`.
+
+|`net.openhft:chronicle-core`
+|Supplies utility classes (thread affinity, OS detection, and annotations) referenced by Wire.
+|Keep aligned with the version bundled in the BOM to avoid `NoSuchMethodError`.
+
+|`net.openhft:affinity`
+|Used by some benchmarks and optional modules to pin threads when verifying latency claims.
+|Only required for integration and load testing.
+|===
+
+== Supported JVMs
+
+* Primary: JDK 8u382+, 11.0.20+, 17.0.8+, 21.0.1+.
+* Early Access: JDK 22 (monitored via the `ea` branch; no guarantees yet).
+* HotSpot is the reference runtime; OpenJ9 receives best-effort support.
+
+== Performance Notes
+
+* Chronicle Wire prefers G1 or ZGC for production when running on Java 17+.
+* Allocate direct memory via `-XX:MaxDirectMemorySize` to match Chronicle Queue retention;
+  undersizing this flag leads to forced recycling of buffers under load.
+* Use `-Djvm.compile.threshold=1000` or JIT profiles when benchmarking `MethodWriter` to reach
+  steady-state quickly.
+
+== Observability
+
+* Enable `-Dchronicle.wire.dumpOnError=true` during incident triage to print problematic documents.
+* Chronicle Services deployments typically send structured error events to Chronicle Telemetry;
+  Wire emits markers that this pipeline consumes.
+
+== References
+
+* `system-architecture.adoc` ties these dependencies back to the component diagram.
+* Chronicle public docs: https://chronicle.software/knowledge-base/

src/main/docs/data-requirements.adoc

@@ -0,0 +1,61 @@
+= Chronicle Wire Data Requirements
+Chronicle Software
+:toc:
+:sectnums:
+:lang: en-GB
+:source-highlighter: rouge
+
+== Purpose
+
+This document captures the structural expectations for Chronicle Wire payloads.
+It complements `functional-requirements.adoc` by focusing on envelopes, metadata,
+and schema-evolution guidance shared by all `WireType` implementations.
+
+== Message Envelope
+
+[cols="1,3", options="header"]
+|===
+|ID |Requirement
+
+|FN-101
+|Every wire document starts with an implicit envelope managed by `DocumentContext`.
+The writer must emit the document length (for binary wires) or delimiter (for textual wires)
+so downstream consumers can skip, replay, or rewind without parsing the payload itself.
+
+|FN-102
+|Field names shall be emitted verbatim for textual wires and mapped to 32-bit field identifiers
+(`WireKey`) for binary wires. Field identifiers must stay stable across releases to preserve backwards compatibility.
+
+|FN-103
+|When class metadata is present (`@type` in YAML/JSON or numeric type IDs in BinaryWire),
+it must reference the fully-qualified Java class or an alias registered via `ClassAliasPool`.
+|===
+
+== Scalar and Collection Semantics
+
+* Numeric types are encoded using the narrowest width that preserves the caller's precision.
+  `long` timestamps must stay in epoch nanoseconds to align with down-stream `Pauser` and engine components.
+* Text values default to UTF-8 bytes on the wire while remaining ISO-8859-1 inside documentation.
+* Collections are written as nested documents where each entry uses either an implicit numeric key
+  (arrays) or explicit field names (maps). Ordered collections retain insertion order.
+
+== Schema Evolution Policy
+
+1. Prefer additive changes (new optional fields, new message types).
+2. Deprecations must keep the existing field identifier reserved even if unused.
+3. When renaming a field, register the former name via `FieldNumberLookup`
+   and keep a test in `WireSchemaEvolutionTest.java` or an equivalent suite.
+4. Use explicit defaults in readers to avoid `NullPointerException` when older payloads omit the new field.
+
+== Metadata and Retention
+
+* Every document that leaves the JVM must carry a monotonically increasing `messageId`
+  or sequence supplied by the caller; Chronicle Wire does not synthesise one automatically.
+* Timestamps inherit the caller timezone; the library never rewrites them.
+* Payloads stored on disk or transmitted over the network must keep their native endianness.
+  Mixing `WireType#binary` and `WireType#binary_little_endian` within the same queue is unsupported.
+
+== References
+
+* `wire-schema-evolution.adoc` describes migration examples that satisfy the requirements above.
+* `src/test/java/net/openhft/chronicle/wire/GenerateJsonSchemaMainTest.java` keeps schema tooling covered by CI.