CNDB-16174: CNDB-15469: Implement NVQ for vector graphs built by compaction

[michaelsembwever]

https://github.com/riptano/cndb/issues/16174

Port into main-5.0 commit 97eaa52

CNDB-16174: CNDB-15469: Implement NVQ for vector graphs built by compaction

Fixes: https://github.com/riptano/cndb/issues/15469
CNDB PR: https://github.com/riptano/cndb/pull/15813

This PR integrates the jvector NVQ feature into SAI vector indexes built via compaction. This feature is disabled by default (by `cassandra.sai.vector.enable_nvq`) to continue providing the best recall when storage savings is not an explicit concern. The jvector library describes NVQ with:

> Support for Non-uniform Vector Quantization (NVQ, pronounced as "new
vec"). This new technique quantizes the values in each vector with high accuracy by first applying a nonlinear transformation that is individually fit to each vector. These nonlinearities are designed to be lightweight and have a negligible impact on distance computation performance.

This feature is only available in SAI on disk version `EC` and later. It can be enabled by setting `cassandra.sai.vector.enable_nvq` to `true` and selecting `cassandra.sai.latest.version=ec` or greater.

When enabled, we can expect NVQ to reduce the storage footprint of the graph (stored in the `TERMS` file) because quantized vectors are stored inline instead of the full precision vectors. A possible result of storing these smaller vectors is fewer iops due to improved efficiency of a graph node fitting within a single 4 kb page.

We do not have any new metrics exposed to track this feature beyond disk utilization.

When troubleshooting, this log line will help determine what features an on disk graph is using:

        logger.debug("Opened graph for {} for sstable row id offset {} with {} features", source, segmentMetadata.segmentRowIdOffset, features);

NVQ will be in the list if it is in use.

tl;dr: NVQ works for earlier versions of CC because the on disk format hasn't changed and jvector knows how to read it. If you enable NVQ on CC without this PR and with `ann_use_synthetic_score = true`, you might see out of order results.

One side effect of NVQ is that the NVQ vector similarity score is slightly different than the full precision score. This is primarily a problem when the synthetic score is in use
(`cassandra.sai.ann_use_synthetic_score`) because the synthetic score was based on the score from the index. Now that this score does not necessarily equal the FP sim score, we must compute the FP sim score before sending the synthetic score to the coordinator. Otherwise, we will end up with out of order vectors.

Because older versions of CC do not correct for this, it is possible to send the wrong score to the coordinator. However, because this feature is disabled by default, there is not really a risk of sending the wrong score.

[github-actions]

Checklist before you submit for review

• This PR adheres to the Definition of Done
• Make sure there is a PR in the CNDB project updating the Converged Cassandra version
• Use NoSpamLogger for log lines that may appear frequently in the logs
• Verify test results on Butler
• Test coverage for new/modified code is > 80%
• Proper code formatting
• Proper title for each commit staring with the project-issue number, like CNDB-1234
• Each commit has a meaningful description
• Each commit is not very long and contains related changes
• Renames, moves and reformatting are in distinct commits
• All new files should contain the DataStax copyright header instead of the Apache License one

[michaelsembwever]

Included https://github.com/riptano/cndb/issues/16174

[djatnieks]

There are failures in VectorCompaction100dTest and VectorSiftSmallTest that may be relevant to check out further.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
90.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[michaelsembwever]

There are failures in VectorCompaction100dTest and VectorSiftSmallTest that may be relevant to check out further.

they work locally for me. in jenkins they are timing out…

[michaelsembwever]

There are failures in VectorCompaction100dTest and VectorSiftSmallTest that may be relevant to check out further.
they work locally for me. in jenkins they are timing out…

they remain flaky with timeouts in jenkins.
what should we do about this @djatnieks ? merge the PR and raise them as a separate issue to address later ?

I've looked into the logs and don't see anything that stands out– it's busy the whole time (though I wonder if it's thrashing repeatedly on something it shouldn't be…)

log from jenkins: TEST-org.apache.cassandra.index.sai.cql.VectorSiftSmallTest.log

[cassci-bot]

❌ Build ds-cassandra-pr-gate/PR-2160 rejected by Butler

---

5 regressions found
See build details here

---

Found 5 new test failures

Test	Explanation	Runs	Upstream
o.a.c.cql3.validation.operations.AggregationQueriesTest.testAggregationQueryShouldNotTimeoutWhenItExceedesReadTimeout (compression)	NEW	🔴🔴	2 / 26
o.a.c.index.SecondaryIndexManagerTest.testIndexRebuildWhenAddingSStableViaRemoteReload (compression)	NEW	🔴🔴	20 / 26
o.a.c.index.sai.cql.VectorCompaction100dTest.testOneToManyCompactionTooManyHoles[dc true] ()	NEW	🔴⚪	0 / 26
o.a.c.index.sai.cql.VectorSiftSmallTest.testMultiSegmentBuild[db false] ()	NEW	🔴🔴	0 / 26
o.a.c.metrics.TrieMemtableMetricsTest.testContentionMetrics (compression)	NEW	🔵🔴	9 / 26

---

Found 5 known test failures

[djatnieks]

what should we do about this @djatnieks ? merge the PR and raise them as a separate issue to address later ?

Yes, I think that is a good plan since they pass locally and let's us keep moving forward. Maybe someone from the query team could take a look too if we have a ticket.