Skip to content

Allow evaluations on more queries #21

@mam10eks

Description

@mam10eks

Problem: We have datasets/splits with often only ca. 50 Queries.

How can we adjust our setup to run evaluations on more queries, so that retrieval efficiency numbers get more reliable.

In the current abstraction, we could:

  • just duplicate queries, via a flag or environment variable to repeat queries

Another variant:

  • Use LLMs to generate user query variants
    • get the query length into evaluations as well

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions