Replicating Webshop Results

[ai-nikolai]

@archiki On another note - when running your code with the ReAct prompts (out of the box). We receive a score of 0.070 0.0 0.0 when running on the first 30 samples. Could you confirm what your results are?

[ai-nikolai]

So the results also running 30 datapoints using two different version of gpt3.5 turbo yields:

1. gpt-3.5-turbo-1106:
   0.0 0.0 0.0

2. gpt-3.5-turbo (i.e. gpt-3.5-turbo-0125)
   0.070 0.0 0.0

Could you please clarify how you are getting the 30% Sucess Rate and 60+ Score on Webshop?

[ai-nikolai]

ysymyth/ReAct#34

[ai-nikolai]

noahshinn/reflexion#49

[ai-nikolai]

Problem:
Results on Webshop are very bad these days. Could you please help!

Ask:
Could you run Webshop and ReAct agent on a few test samples (e.g. 30) with some models to compare results: gpt-3.5-turbo-1106, gpt-3.5-turbo-0125, gpt-4o-mini.

---

Results:

Webshop results are terrible today using existing code-repos:

Agent	Model	Code Repo	No. Test Samples	Success Rate
ReAct	gpt-3.5-turbo-instruct	Reflexion	30	13%
ReAct	gpt-4o-mini	Reflexion	30	0.0%
ReAct	gpt-3.5-turbo-1106	ADaPT	30	0.0%
ReAct	gpt-3.5-turbo-0125	ADaPT	30	0.0%
ReAct	gpt-3.5-turbo-instruct	ReAct	30	13%
ReAct	gpt-3.5-turbo-instruct	Ours	30	13%
ReAct	gpt-3.5-turbo-1106	Ours	30	0.0%
ReAct	gpt-3.5-turbo-1106	ReAct	30	0.0%
ReAct	gpt-3.5-turbo-1106	StateAct	30	0.0%

Webshop results using the same models 6 months ago:

Agent	Model	Code Repo	No. Test Samples	Success Rate
ReAct	gpt-3.5-turbo-1106	StateAct	500	18.2%
ReAct	gpt-3.5-turbo-0125	StateAct	500	14.6%
ReAct (2 years ago)	gpt-3.5-turbo (not reported Version)	ADaPT, ReAct & Reflexion	100/500	30%