About the evaluation of Web action

[Pefect96]

Great Job! However, I could not find the evaluation script of the web action.

I hope to receive assistance from the author. Thank you very much.

[Gabesarch]

Hi @Pefect96, we evaluate the model on VisualWebArena, following the original repo instructions (https://github.com/web-arena-x/visualwebarena/tree/main). We use set of marks inputs on the images without textual set of marks.

[Pefect96]

Hi @Pefect96, we evaluate the model on VisualWebArena, following the original repo instructions (https://github.com/web-arena-x/visualwebarena/tree/main). We use set of marks inputs on the images without textual set of marks.

OK, Thanks for your help.

Additionally, in Appendix Table A2, the GRPO training hyperparameters include a warmup ratio of 0.05 and a batch size of 32. However, the provided script run_vigorl.sh does not use a warmup schedule, and the batch size is set to 64. Could you clarify which configuration was used to produce the results reported in the paper?

When i training the web grounding model by employing the provided ViGoRL-MCTS-SFT-3b-Web-Grounding model, the performance of the trained model cannot match that described in the paper. For example, the perfoemance of ScreenSpot Scree-V2 and ScreenSpot-Pro are 23.66% and 81.68 (31.1 and 86.5 in the paper ). Could you offer some assistance or review the script you provided? Is there something wrong with me?

[Gabesarch]

Hi @Pefect96,

I am not able to replicate your low numbers. I just trained (https://github.com/Gabesarch/grounded-rl/blob/main/src/trainer/rl/examples/run_vigorl.sh) with:

ROLLOUT_BATCH_SIZE=128
GLOBAL_BATCH_SIZE=64
WARMUP_RATIO=0.0

After 500k steps, I got:
screenspot_pro: count=1581, accuracy=0.3251
screenspot_v2: count=1272, accuracy=0.8640

While the batch size I used is higher, I would not expect this to cause such a decrease in numbers.

Please make sure your evaluation script uses the correct setting for max_pixels as described here: #16 (comment)