explore additional dynamic kalpha settings to improve filtered ann accuracy

[lyang24]

some ideas:

1. Instead of global filter_ratio × 0.7, compute kAlpha per-candidate based on local neighborhood passrate (source https://arxiv.org/abs/2506.23397)

what does maintainers think about this issue?

[liliu-z]

@lyang24 Thanks for this question.

This paper presents a variant of ACORN that introduces weights with a slight increase in computation to improve recall. However, it fundamentally still belongs to the ACORN family of approaches. The limitation of ACORN is that it has relatively weak connectivity for highly-filtered isolated nodes, as it only selects valid points and relies on two-hop searches to improve connectivity.

Knowhere's Alpha strategy has stronger penetration capability for highly-filtered isolated nodes because we continuously navigate through the invalid graph. The advantage is achieving better recall under high filtering conditions, but at the cost of more performance overhead.

Our Cloud solution Cardinal applies another Alpha variant to improve robustness, but the overall philosophy is to prioritize recall.

For ACORN-family algorithms, if the community is interested, we very much welcome any contributions—as long as they are isolated from existing strategies using a flag.

[lyang24]

The limitation of ACORN is that it has relatively weak connectivity for highly-filtered isolated nodes, as it only selects valid points and relies on two-hop searches to improve connectivity.

indeed this has been pointed out by many peer papers, in compass https://arxiv.org/pdf/2510.27141

Under low passrate (Figure 10), both NaviX and SeRF
have difficulty producing reasonable recall due to the graph
disconnectivity and aggressive graph compression strategy 

usually i have seen the compensation strategy for highly-filtered isolated nodes is to brute force on flat index. I think with knowhere's alpha strategy on very high filtered senario in a large graph can potentially have a very large ef.

there are many advances in filtered ann reasearch - like the compass paper combines b+ trees and hnsw doing Dijkstra-like multi-source best-first search over (vector distance + predicate space).

will do more research on this area and come up with a studied proposal - if everything goes well i can put the impl under a feature/ experiment flag and run some benches.

[liliu-z]

The limitation of ACORN is that it has relatively weak connectivity for highly-filtered isolated nodes, as it only selects valid points and relies on two-hop searches to improve connectivity.

indeed this has been pointed out by many peer papers, in compass https://arxiv.org/pdf/2510.27141

Under low passrate (Figure 10), both NaviX and SeRF
have difficulty producing reasonable recall due to the graph
disconnectivity and aggressive graph compression strategy 

usually i have seen the compensation strategy for highly-filtered isolated nodes is to brute force on flat index. I think with knowhere's alpha strategy on very high filtered senario in a large graph can potentially have a very large ef.

there are many advances in filtered ann reasearch - like the compass paper combines b+ trees and hnsw doing Dijkstra-like multi-source best-first search over (vector distance + predicate space).

will do more research on this area and come up with a studied proposal - if everything goes well i can put the impl under a feature/ experiment flag and run some benches.

We are optimistic about the engineering feasibility of executing different strategies based on varying filter selectivity levels. If we want to benchmark different strategies, I think we can try to provide corresponding tools for that purpose.

cc @foxspy