Has anyone actually reproduced the results of this repo?

[EvanWeiner]

This is an open question to the community (but authors, please chime in as well).

I have seen a bunch of questions related to reproducibility. I want to openly ask if anyone has independently reproduced the results, in full or partially, of this repo.

Before people spend a lot of time and effort working on this, it would be great to know of at least one independent verification!

[IsaacGHX]

Thanks for raising this. Reproducibility is fundamental to the community, and we really appreciate you pushing for it.

We stand by our reported results. At the same time, we understand that agentic systems are complex. Factors like environment configuration, specific tool settings, API constraints, or skipping preliminary environment checks can easily prevent the system from performing at its best.

To make the reproduction process smoother and help everyone verify the results, we have significantly updated our resources:

1. Clearer Protocol: A step-by-step evaluation guide to ensure everything is aligned.
2. Environment Verification: We added new testing scripts. We highly recommend running these first to isolate and verify that your tools and engines are ready before running the full benchmarks.
3. Proof of Results: To provide full transparency, we have hosted our evaluation logs on Google Drive for direct inspection.

Our goal is to make AgentFlow as robust as possible. If you follow the new guide and still notice discrepancies, please share your logs with us. We are eager to look into it and improve the codebase or documentation further based on your feedback.

Best,

The AgentFlow Team

[yanyiwei]

Hi, I am able to get the training working. The validation accuracy starts to increase after training about 30 steps.

I trained Qwen2.5-7B-Instruct using the same config in config.yaml listed in the repo.