Large language models (LLMs) have demonstrated remarkable performance across a range of natural language processing (NLP) tasks. However, they encounter significant challenges in automated reasoning, especially in multi-step reasoning scenarios. In order to solve complex reasoning problems, LLMs need to perform faithful multi-step reasoning based on a given set of facts and rules. A lot of work has focused on guiding LLMs to think logically by generating reasoning paths, but ignores the relationship among available facts. In this paper, we introduce MindMap, a straightforward yet powerful approach for constructing evidence chains to support reasoning in LLMs. An evidence chain refers to a set of facts that are associated with the same subject. In this way, we can organize related facts together to avoid missing relevant information. MindMap can seamlessly integrate with existing reasoning frameworks, such as Chain-of-Thought (CoT) and Selection-Inference (SI), by enabling the model to generate and select relevant evidence chains from independent facts. The experimental results on the bAbI and ProofWriterOWA datasets demonstrate the effectiveness of MindMap. Our approach can significantly enhance the performance of CoT and SI, particularly in multi-step reasoning tasks.
The main framework of the proposed MindMap approach.
The figure above the head illustrates the MindMap's workflow for answering a question from the bAbI dataset, comprising three modules: evidence chain construction, summarization, and utilization for reasoning.
Evidence chain construction:To construct these chains, we deploy the Evidence Chain Construction module. This involves creating subjects using entity extraction and dependency parsing from the Stanford CoreNLP toolkit. We then form an evidence chain for each subject by collating facts that include the specific subject.
Evidence chain summarization: evidence chains vary in length. To achieve brevity, we introduce an evidence chain summarization module. It guides a Large Language Model to generate summaries focusing on key entities in each chain.
Evidence chain utilization for reasoning: During inference, we substitute individual facts with these summarized evidence chains for reasoning. This modification doesn't alter the underlying reasoning framework, demonstrating MindMap's compatibility with various reasoning approaches.
In this paper, experiments using the vicuna-7b model are used.
model weight：vicuna-7b
git clone https://github.com/Yangyu-Wu/MindMap
cd MindMap
pip install -r requirements.txt
We use the entity extraction and dependency parsing modules in the Stanford CoreNLP toolkit stanza to extract the entities which are subjects in the facts.
python consturct_EC.py
We use instructions to guide a LLM to generate a summary covering main entities in each evidence chain. The relevant prompt is in /prompt/consturction
python summarize_EC.py
Please set the model storage location and related parameters in main.py
python main.py
python evaluate.py
More technical details can be found in our paper.
If you find MindMap useful or relevant to your project and research, please kindly cite our paper:
@inproceedings{wu2024mindmap,
title={MindMap: Constructing Evidence Chains for Multi-Step Reasoning in Large Language Models},
author={Wu, Yangyu and Han, Xu and Song, Wei and Cheng, Miaomiao and Li, Fei},
booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
volume={38},
number={17},
pages={19270--19278},
year={2024}
}