does the dataset adapt to evaluating RAG?

[JunjieChiang]

does the dataset adapt to evaluating RAG? From the paper, it is for evaluating VLM's VQA performance.

[zhaowei-wang-nlp]

Hi Junjie,

Certainly! You can build RAG systems to tackle tasks in our MMLongBench. You may evaluate whether your RAG can accurately locate the answer sentence (needle) within a long input (haystack) and whether it can provide the correct answer based on the retrieved parts.

If the provided context is not long enough for your needs, you can expand it using data from the original datasets to create longer inputs. All details regarding context construction are provided in Appendix A of our paper. For example, in the Visual RAG task, we use passages from Wikipedia to construct the long context. If you wish to build even longer contexts, you can simply add more Wikipedia passages. Our Wikipedia preprocessing follows the methodology described in the KILT paper (https://github.com/facebookresearch/KILT).

Meanwhile, we are also planning to release 256K and 512K versions of our benchmark.

It’s great to see our benchmark enabling more and more research!