Assisted or un-assisted

[saisurbehera]

Are the results for devin assisted or unassisted? I am confused by the wording

[moresearch]

As an agent, Devin does not receive any list of files and instead navigates files on its own, which is more comparable to the “unassisted” LLM.

Because neither unassisted nor assisted is strictly comparable to the agent setting, where Devin is given the entire repo and can navigate the files freely, we choose the stronger numbers for the baseline comparison. We believe that running an agent end to end is the more natural setting for SWE-bench, as it’s more similar to real-world software development. We anticipate more agent results in this setting going forward.

source: https://www.cognition-labs.com/post/swe-bench-technical-report

[moresearch]

For more information regarding the wording, please review the original paper from Princeton https://www.swebench.com/