Evaluation pipeline question

[mbehrooz]

Hi all, SOTOPIA n00b here :)

I need to run a simulation to compare a prompting technique to regular prompting of the same model. I have small local models and no GPT 4. I intend to use Llama3.1 (8b, 70b as agents, judge)

Currently, I have VLLM running locally with 8b model, and SOTOPIA does connect to that in the minimal example (agent1, 2, and env pointing to the same thing for now).

What I want to do from here is to change the system parameter of one of the LLMs to implement custom prompting for agent 1.

And then I don't have an idea how to "run the actual thing" from there on. Do I sample agents and interactions (from db dump) and run them in a loop? Is one of the example evaluation codes more suitable?

[mbehrooz]

@ProKil @Jasonqi146 @lwaekfjlk Sorry to tag you all - but if anyone could answer I'd be very grateful, thank you.

[Jasonqi146]

To clarify, do you mean that you need two custom LLMs with different system prompts? What is the current method you used for connecting to custom models?

[mbehrooz]

Thanks @Jasonqi146! I currently use VLLM (with "custom/" path in model_dict, to serve a local Llama model, and was able to run the basic example in the notebook!

What I need now is:

• run the full evaluation experiment where I can compare LLMs (say llama 8B as agents and 70B as a judge). I'm not sure which file in the examples is suited for that, or if there is one.
• for example, I'd want to change the system parameter of one of the LLMs to see how differently it'd perform. I'm assuming I have to sample from the database.

If I can also get the contexts and chat transcripts of the interactions in a file/db, that'd be wonderful. I assume that's what's given to the judge LLM.

Sorry for what's likely basic questions -- FWIW, I love SOTOPIA :)