[Bug] TextVQA，输出predcition正确，但是result精度为0

[MansonX]

操作系统及版本

ubuntu 22.04

安装工具的python环境

在anaconda/miniconda创建的python虚拟环境

python版本

3.10

AISBench工具版本

3.1.20260306

AISBench执行命令

ais_bench --models vllm_api_general_chat --datasets textvqa_gen_base64 --mode all --debug

模型配置文件或自定义配置文件内容

cli_args=dict(
cfg_time_str='20260312_122302',
config=None,
config_dir='configs',
custom_dataset_data_type=None,
custom_dataset_infer_method=None,
custom_dataset_meta_path=None,
custom_dataset_path=None,
datasets=[
'textvqa_gen_base64',
],
debug=True,
dir_time_str='20260312_122302',
dry_run=False,
dump_eval_details=False,
dump_extract_rate=False,
max_num_workers=1,
max_workers_per_gpu=1,
merge_ds=False,
mode='all',
models=[
'vllm_api_general_chat',
],
num_prompts=None,
num_warmups=1,
pressure=False,
pressure_time=15,
reuse=None,
search=False,
summarizer=None,
work_dir=None)
datasets=[
dict(abbr='textvqa',
eval_cfg=dict(
evaluator=dict(
type='ais_bench.benchmark.datasets.TEXTEvaluator')),
image_type='image_base64',
infer_cfg=dict(
inferencer=dict(
type='ais_bench.benchmark.openicl.icl_inferencer.GenInferencer'),
prompt_template=dict(
template=dict(
round=[
dict(prompt_mm=dict(
image=dict(
image_url=dict(
url='data:image/jpeg;base64,{image}'),
type='image_url'),
text=dict(
text='{question} Answer the question using a single word or phrase.',
type='text')),
role='HUMAN'),
]),
type='ais_bench.benchmark.openicl.icl_prompt_template.icl_prompt_template_mm.MMPromptTemplate'),
retriever=dict(
prompt_template=dict(
template=dict(
round=[
dict(prompt_mm=dict(
image=dict(
image_url=dict(
url='data:image/jpeg;base64,{image}'),
type='image_url'),
text=dict(
text='{question} Answer the question using a single word or phrase.',
type='text')),
role='HUMAN'),
]),
type='ais_bench.benchmark.openicl.icl_prompt_template.icl_prompt_template_mm.MMPromptTemplate'),
type='ais_bench.benchmark.openicl.icl_retriever.ZeroRetriever')),
path='ais_bench/datasets/textvqa/textvqa_val.jsonl',
reader_cfg=dict(
input_columns=[
'question',
'image',
],
output_column='answer'),
type='ais_bench.benchmark.datasets.TEXTVQADataset'),
]
eval=dict(
partitioner=dict(
out_dir='outputs/default/20260312_122302/results/',
type='ais_bench.benchmark.partitioners.naive.NaivePartitioner'),
runner=dict(
debug=True,
max_num_workers=1,
max_workers_per_gpu=1,
task=dict(
type='ais_bench.benchmark.tasks.openicl_eval.OpenICLEvalTask'),
type='ais_bench.benchmark.runners.local.LocalRunner'))
infer=dict(
partitioner=dict(
out_dir='outputs/default/20260312_122302/predictions/',
type='ais_bench.benchmark.partitioners.naive.NaivePartitioner'),
runner=dict(
debug=True,
max_num_workers=1,
max_workers_per_gpu=1,
task=dict(
type='ais_bench.benchmark.tasks.openicl_api_infer.OpenICLApiInferTask'),
type='ais_bench.benchmark.runners.local.LocalRunner'))
models=[
dict(abbr='vllm-api-general-chat',
api_key='',
attr='service',
batch_size=64,
generation_kwargs=dict(
chat_template_kwargs=dict(
enable_thinking=False),
ignore_eos=False,
temperature=0.0),
host_ip='141.61.81.13',
host_port=7070,
max_out_len=1024,
model='qwen35',
path='/mnt/share/w00957216/qwen35',
pred_postprocessor=dict(
type='ais_bench.benchmark.utils.postprocess.model_postprocessors.extract_non_reasoning_content'),
request_rate=0,
retry=2,
stream=False,
trust_remote_code=False,
type='ais_bench.benchmark.models.VLLMCustomAPIChat',
url='',
use_timestamp=False),
]

预期行为

prediction结果符合golden的情况下summary精度大于0

实际行为

部分prediction：
{"data_abbr": "textvqa", "id": 4949, "success": true, "uuid": "d3cf79ab769e4cc1a59733edcb6ef585", "origin_prompt": [{"role": "HUMAN", "prompt": [{"image_url": {"url": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEASABIAAD/4gxYSUNDX1BST0ZJTEUAAQEAAAxITGlubwIQAABtbnRyUkdCIFhZWiAHzgACAAkABgAxAABhY3NwTVNGVAAAAABJRUMgc1JHQgAAAAAAAAAAAAAAAAAA9tYAAQAAAADTLUhQICAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABFjcHJ0A ..."}, "type": "image_url"}, {"text": "which year did this happen? Answer the question using a single word or phrase.", "type": "text"}]}], "prediction": "2011<|im_end|>\n<|endoftext|>", "gold": [{"answer": "2011", "answer_confidence": "yes", "answer_id": 0}, {"answer": "2011", "answer_confidence": "yes", "answer_id": 1}, {"answer": "2011", "answer_confidence": "yes", "answer_id": 2}, {"answer": "2011", "answer_confidence": "yes", "answer_id": 3}, {"answer": "2011", "answer_confidence": "yes", "answer_id": 4}, {"answer": "2011", "answer_confidence": "yes", "answer_id": 5}, {"answer": "2011", "answer_confidence": "yes", "answer_id": 6}, {"answer": "2011", "answer_confidence": "yes", "answer_id": 7}, {"answer": "2011", "answer_confidence": "yes", "answer_id": 8}, {"answer": "2011", "answer_confidence": "yes", "answer_id": 9}]}
{"data_abbr": "textvqa", "id": 4945, "success": true, "uuid": "68fffed926db4ae0a2d08626eda51fff", "origin_prompt": [{"role": "HUMAN", "prompt": [{"image_url": {"url": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEASABIAAD/4gxYSUNDX1BST0ZJTEUAAQEAAAxITGlubwIQAABtbnRyUkdCIFhZWiAHzgACAAkABgAxAABhY3NwTVNGVAAAAABJRUMgc1JHQgAAAAAAAAAAAAAAAAAA9tYAAQAAAADTLUhQICAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABFjcHJ0A ..."}, "type": "image_url"}, {"text": "what is the title of the thin red book? Answer the question using a single word or phrase.", "type": "text"}]}], "prediction": " el arte de cocinar<|im_end|>\n<|endoftext|>", "gold": [{"answer": "la sombra del aguilla", "answer_confidence": "yes", "answer_id": 0}, {"answer": "la sombra del aguila", "answer_confidence": "yes", "answer_id": 1}, {"answer": "la sombra del aquila", "answer_confidence": "yes", "answer_id": 2}, {"answer": "la sombra del aguila", "answer_confidence": "yes", "answer_id": 3}, {"answer": "la sombra de agulla", "answer_confidence": "yes", "answer_id": 4}, {"answer": "la sombre del aguilla", "answer_confidence": "yes", "answer_id": 5}, {"answer": "la sombra del aguilla", "answer_confidence": "yes", "answer_id": 6}, {"answer": "la sombra del aguila", "answer_confidence": "yes", "answer_id": 7}, {"answer": "la sombra del aquila", "answer_confidence": "yes", "answer_id": 8}, {"answer": "la sombra del águila", "answer_confidence": "yes", "answer_id": 9}]}
{"data_abbr": "textvqa", "id": 4975, "success": true, "uuid": "eaf538a406d34cec8320f7e51df28a34", "origin_prompt": [{"role": "HUMAN", "prompt": [{"image_url": {"url": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEA8ADwAAD/4gxYSUNDX1BST0ZJTEUAAQEAAAxITGlubwIQAABtbnRyUkdCIFhZWiAHzgACAAkABgAxAABhY3NwTVNGVAAAAABJRUMgc1JHQgAAAAAAAAAAAAAAAAAA9tYAAQAAAADTLUhQICAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABFjcHJ0A ..."}, "type": "image_url"}, {"text": "what is the name of this wine? Answer the question using a single word or phrase.", "type": "text"}]}], "prediction": " rubino del casale<|im_end|>\n<|endoftext|>", "gold": [{"answer": "rubino del casale", "answer_confidence": "yes", "answer_id": 0}, {"answer": "rubino del casale ", "answer_confidence": "yes", "answer_id": 1}, {"answer": "rubino del casale ", "answer_confidence": "yes", "answer_id": 2}, {"answer": "rubino", "answer_confidence": "yes", "answer_id": 3}, {"answer": "rulino del casale", "answer_confidence": "yes", "answer_id": 4}, {"answer": "vino da tavola rosso", "answer_confidence": "yes", "answer_id": 5}, {"answer": "rubino del casale", "answer_confidence": "yes", "answer_id": 6}, {"answer": "rubinode casale", "answer_confidence": "yes", "answer_id": 7}, {"answer": "rubino del casale", "answer_confidence": "yes", "answer_id": 8}, {"answer": "rubino del casale", "answer_confidence": "yes", "answer_id": 9}]}
{"data_abbr": "textvqa", "id": 4890, "success": true, "uuid": "d2713693c1fa40cdb87198565851762a", "origin_prompt": [{"role": "HUMAN", "prompt": [{"image_url": {"url": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEA8ADwAAD/4gJASUNDX1BST0ZJTEUAAQEAAAIwQURCRQIQAABtbnRyUkdCIFhZWiAHzwAGAAMAAAAAAABhY3NwQVBQTAAAAABub25lAAAAAAAAAAAAAAAAAAAAAAAA9tYAAQAAAADTLUFEQkUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAApjcHJ0A ..."}, "type": "image_url"}, {"text": "what is the address for the business on the ad? Answer the question using a single word or phrase.", "type": "text"}]}], "prediction": "4001-4053 ravenswood ave.<|im_end|>\n<|endoftext|>", "gold": [{"answer": "manz", "answer_confidence": "yes", "answer_id": 0}, {"answer": "4001-4053 ravenswood ave. chicago, ill", "answer_confidence": "yes", "answer_id": 1}, {"answer": "4001-4053 ravenswood ave. chicago, ill.", "answer_confidence": "yes", "answer_id": 2}, {"answer": "4001-4053 ravenswood ave. chicago, ill.", "answer_confidence": "yes", "answer_id": 3}, {"answer": "4001-4053 ravenswood ave", "answer_confidence": "yes", "answer_id": 4}, {"answer": "4001-4053 ravenswood ave chicago ill", "answer_confidence": "yes", "answer_id": 5}, {"answer": "4001-4053 ravenswood ave. chicago, ill.", "answer_confidence": "yes", "answer_id": 6}, {"answer": "4001-4053 ravenswood ave chicaago, ill", "answer_confidence": "yes", "answer_id": 7}, {"answer": "4001-4053 ravenswood ave. chicago, ill", "answer_confidence": "yes", "answer_id": 8}, {"answer": "4001-4053 ravenswood ave. chicago,ill.", "answer_confidence": "yes", "answer_id": 9}]}

summary:
tabulate format
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dataset version metric mode vllm-api-general-chat

---

textvqa e9d882 accuracy gen 0.00
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$

-------------------------------------------------------------------------------------------------------------------------------- THIS IS A DIVIDER --------------------------------------------------------------------------------------------------------------------------------

csv format
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dataset,version,metric,mode,vllm-api-general-chat
textvqa,e9d882,accuracy,gen,0.00
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$

markdown format
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

dataset	version	metric	mode	vllm-api-general-chat
textvqa	e9d882	accuracy	gen	0.00

$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
-------------------------------------------------------------------------------------------------------------------------------- THIS IS A DIVIDER --------------------------------------------------------------------------------------------------------------------------------

raw format
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Model: vllm-api-general-chat
textvqa: {'accuracy': 0.0}
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$

前置检查

• 我已读懂主页文档的快速入门，无法解决问题
• 我已检索过FAQ，无重复问题
• 我已搜索过现有Issue，无重复问题
• 我已更新到最新版本，问题仍存在

[github-actions]

🌟欢迎给AISBench benchmark测评工具提issue，您可以先尝试以下途径寻求issue的解决方法：

1. 【强烈推荐❤️‍🔥】确保issue描述完整后，可以试着将issue交给回答，deepwiki包含了和工具相关的所有知识库
2. 在 工具历史issue 中通过问题的关键日志片段或关键词检索相似issue
3. 在 官方文档 的搜索栏中通过问题的关键日志片段或关键词检索
4. 寻求仓库维护者的帮助（issue评论区中 @SJTUyh）

[github-actions]

🤖 基于AI机器人的issue内容完整性检查结果:

问题内容检查结果

检查状态

PASS

详细检查

• 操作系统及版本: COMPLETE
• 安装工具的python环境: COMPLETE
• python版本: COMPLETE
• AISBench工具版本: COMPLETE
• AISBench执行命令: COMPLETE
• 模型配置文件或自定义配置文件内容: COMPLETE
• 实际行为: COMPLETE

改进建议

所有内容已完备！

补充说明

该Issue内容非常详细和完整，包含了所有必需的信息，并且提供了详细的模型配置和实际运行结果，非常有利于问题的诊断。

👉 如果想重新检查，在评论区@issue_checker即可。

【强烈推荐❤️‍🔥】确保issue描述完整后，可以试着将issue交给回答，deepwiki包含了和工具相关的所有知识库

[GaoHuaZhang]

根本原因是模型输出的结尾 token <|endoftext|> 没有在评估前的“规范化”里被去掉，导致规范化后的预测字符串和标准答案永远对不上，所以 accuracy 一直是 0。

1. 评估时对 prediction 的规范化（TEXTEvaluator）
   在 ais_bench/benchmark/datasets/textvqa.py 里，TEXTEvaluator.score() 会对每条 prediction 做如下处理（约 313–319 行）：
   replace('\n', ' ')、replace('\t', ' ')、strip()
   remove_special_characters(res_data)
   process_punctuation(res_data)
   process_digit_article(res_data)
   然后才用处理后的 res_data 和 gold 的 answer 做相等比较（326 行）。
2. 实际只去掉了 <|im_end|>，没有去掉 <|endoftext|>
   remove_special_characters 使用的列表在 第 187 行 定义为：
   textvqa.py Lines 187-187
   self.special_tokens = ['☞', '☟', '☜', '', '<|im_end|>']
   也就是说，只会把 <|im_end|> 等这几个 token 删掉，没有包含 <|endoftext|>。
   而你的 prediction 是这种形式：
   "2011<|im_end|>\n<|endoftext|>"
   " rubino del casale<|im_end|>\n<|endoftext|>"
   处理过程是：
   换行变空格："2011<|im_end|> <|endoftext|>"
   remove_special_characters 只删掉 <|im_end|>，得到："2011 <|endoftext|>"
   process_punctuation 等不会处理 <|endoftext|> 这种“词”
   最终参与比较的是 "2011 <|endoftext|>"，而 gold 是 "2011"
   字符串不相等 → 该条判错 → 大量样本都如此 → summary 里 accuracy 为 0.00
   所以：prediction 在语义上已经和 gold 一致（如 "2011"、"rubino del casale"），但因为末尾多了未被移除的 <|endoftext|>，在严格字符串比较下全部被判错。
   这通常是由于模型经过强化学习引入的回答格式化不一致，临时适配可以修改 self.special_tokens = ['☞', '☟', '☜', '', '<|im_end|>']中补充你回答中包含需要过滤特殊字符，或者参考ais_bench/benchmark/configs/datasets/textvqa/glm4v_textvqa_gen_base64.py 实现自定义evaluator进行精度计算