Skip to content

[BUG] evaluation: BM25/Embedding index filenames mismatch when running with --from-conv/--to-conv, causing empty retrieval #127

@Donghua-Cai

Description

@Donghua-Cai

I encountered a problem when reproducing the evaluation experiments on the longmemeval dataset.

When using sliced runs (e.g. --from-conv 234 --to-conv 264) can make search fail with:

BM25 index not found: bm25_index_conv_234.pkl

Root cause: index files may be built with local sequential ids (0..N-1) while search looks up global conversation ids (234..263).

Result: search_results are empty, and evaluation scores drop for the wrong reason.

Repro:

uv run python -m evaluation.cli
--dataset longmemeval
--system evermemos
--from-conv 234 --to-conv 264
--run-name test
Expected: sliced runs should generate/load indexes with consistent conversation ids so search can find bm25_index_conv_<conv_id>.pkl.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions