Problem: We have datasets/splits with often only ca. 50 Queries.
How can we adjust our setup to run evaluations on more queries, so that retrieval efficiency numbers get more reliable.
In the current abstraction, we could:
- just duplicate queries, via a flag or environment variable to repeat queries
Another variant:
- Use LLMs to generate user query variants
- get the query length into evaluations as well