Incorrect ground truth in benchmark at indices 99, 100, 279, 280

[giftedsoda]

Hi, I identified some labeling errors in the benchmark dataset. These incorrect answers affect the evaluation accuracy. Specifically, the ground truth for indices 99, 100, 279, and 280 appears to be wrong.Since this is a benchmark, these errors might lead to inaccurate model scoring.

[Luodian]

MMVP Ground Truth Verification

This verification was performed using lmms-lab's eval agents for massive-scale dataset ground-truth auditing. We are exploring automated approaches to catch annotation errors across vision-language benchmarks.
GitHub: https://github.com/EvolvingLMMs-Lab/lmms-eval

Ref: MMVP Issue #30 reported indices 99, 100, 279, 280 as potential errors.

---

Double Confirmed Errors

Index 99 - Elephant Tusks

Index	Image	Question	GT	Visual Reality
98		Does the elephant have long or short tusks?	A (Long)	Long
99		Does the elephant have long or short tusks?	B (Short)	Long

Verdict: Index 99 clearly shows long tusks (extending past jaw, pronounced curvature). GT is wrong.

---

Index 279 - Person Posture

Index	Image	Question	GT	Visual Reality
278		Is the elderly person standing or sitting?	A (Standing)	Standing
279		Is the elderly person standing or sitting?	B (Sitting)	Standing

Verdict: Index 279 shows person standing (piggyback ride, upright, no chair). GT is wrong.

---

Plausibly Not Errors

The original issue also flagged indices 100 and 280. After visual verification, these appear correct:

Index 100 - Volleyball Action

Index	Image	Question	GT	Visual Reality
100		What is the woman more likely to do?	A (Bump)	Bump stance
101		What is the woman more likely to do?	B (Set)	Set stance

Verdict: Index 100 shows classic forearm pass (bump) position - arms extended, low stance. GT is correct.

---

Index 280 - Drinks on Table

Index	Image	Question	GT	Visual Reality
280		Are there drinks on the table?	A (Yes)	Drinks visible
281		Are there drinks on the table?	B (No)	No drinks

Verdict: Index 280 shows glasses of juice on table. GT is correct.

[giftedsoda]

Thanks for the verification! Yes, those are exactly the error cases I found. I appreciate you taking the time to check them. This is a really excellent project, keep up the great work! @Luodian