Running with local huggingface models

[namin]

Hi again,

The paper says experiments were done with llama, but I don't see any part of this codebase referring to llama (except for an output formatting directive). I did a small attempt at converting the lm_tools.py to run on a local llama, but the results are not good. It could be that I am doing something wrong. Are there plans to release the llama experiments as well as the OpenAI ones?

Here is my attempt:
metareflection@4919ee9

Thank you.

[namin]

Note that it seems OpenAIModel of synchromesh doesn't take a cache_path directory but one is given in the code in lm_tools.

But it doesn't look like synchromesh supports open models? In which case, I am not sure how to translate PeanoLMReasoner to use llama. Is that something you've done?

[namin]

It looks like synchromesh has these features now, but the submodule was not updated.

[namin]

My further attempt can be tracked here:
https://github.com/metareflection/certified-reasoning/compare/08037a7...metareflection:certified-reasoning:open?expand=1

[gpoesia]

Hi @namin!

You're right, the current main branch of Synchromesh doesn't have the HuggingFace integration on it. We used the version that is in this branch, which should let you run any Hugging Face model, including LLaMA.

I have some uncommitted work that organizes this messy dependency structure (make synchromesh into a proper PyPI package, merge the huggingface adapter into main, and just have that as a library dependency here instead of a submodule), which shouldn't take long for me to finish, but I dropped the ball in the last months. But let me do that this week and update you here! (it seems like your implementation should also almost work, you'd just need to use the logit_bias parameter in your completion function, when it is not None). Note that we used LLaMA 13B in the paper - 7B was notably worse with just prompting. That said, I didn't rerun with LLaMA 2, so perhaps 7B might be enough.

[namin]

That's awesome! I'll await your update. Thank you!

[gpoesia]

Hi @namin!

I've finally gotten around to doing this and documenting as well (it had been lurking in the repos for a bit...). synchromesh now supports LLaMA (or other hugging face models) in the main branch, and you can use that here too if you like (lm_tool.py now has HuggingFaceLMReasoner and PeanoHuggingFaceLMReasoner, ). I ran with LLaMA 2 13B with 8-bit quantization, and the results don't seem to have changed much from the paper, perhaps slightly better.

The other thing is I added a flag to synchromesh to print some simple profiling info (how much time is spent in LLM inference vs in total, so you can measure the impact of CSD). For these experiments, the overhead of CSD itself is relatively small, ranging from 5-10% of the running time:

Evaluating peano-/scr/poesia/13B on ./prontoqa/5hop_random_seed19.json

Loaded prompt from prompts/peano_prontoqa_short_prompt

185.75s in inference, 153.25s on LLM inference (82.5%)
(./prontoqa/5hop_random_seed19.json, example22, peano-/scr/poesia/13B) success? True (186.40s)

175.84s in inference, 164.34s on LLM inference (93.5%)
(./prontoqa/5hop_random_seed19.json, example23, peano-/scr/poesia/13B) success? False (176.50s)
179.97s in inference, 164.34s on LLM inference (91.3%)

(./prontoqa/5hop_random_seed19.json, example24, peano-/scr/poesia/13B) success? False (180.63s)
178.02s in inference, 164.19s on LLM inference (92.2%)

(./prontoqa/5hop_random_seed19.json, example25, peano-/scr/poesia/13B) success? False (178.54s)
139.92s in inference, 124.79s on LLM inference (89.2%)

(./prontoqa/5hop_random_seed19.json, example26, peano-/scr/poesia/13B) success? True (140.67s)
106.99s in inference, 104.12s on LLM inference (97.3%)

(./prontoqa/5hop_random_seed19.json, example27, peano-/scr/poesia/13B) success? True (107.51s)
148.02s in inference, 140.91s on LLM inference (95.2%)

(./prontoqa/5hop_random_seed19.json, example28, peano-/scr/poesia/13B) success? True (148.78s)
171.90s in inference, 164.23s on LLM inference (95.5%)

The major difference for the certified reasoning experiments is that we do generate a lot more tokens when using formalization (since the LLM first formalizes the whole context, before reasoning). So that does have a large impact on running time. For these larger 5-hop examples, we're running CSD on ~800 tokens of output in an average of around 8s total spent on CSD, so around 100 tokens/s.

Let me know if I can help with anything or if you have questions! I'll leave the issue open for now just in case.