Request: Script to generate example_output.json with category, answer, and vlm_answer fields

[GuangtaoLyu]

Hi maintainers,

Thank you for sharing this dataset and the evaluation framework!

I'm currently generating predictions for this dataset and noticed that the expected output format follows example/example_output.json. Each QA pair in the output should contain the following fields:

• category
• answer
• vlm_answer

Could you please provide a simple script or code snippet demonstrating how to process the dataset and export the results into this specific JSON structure? A minimal working example would be greatly appreciated to ensure full compatibility with your evaluation pipeline.

Thanks in advance for your time and support!

[Charrrrrlie]

Hi @GuangtaoLyu!

Thank you for your attention to our work.

In our CourSI-Bench, each item in the JSON file includes both its category and corresponding answer. You only need to match your generated prediction (the vlm_answer) with the provided ground-truth entries.

i.e., you can get the category and answer from the benchmark JSON and match them with your vlm_answer.