Missing file: scripts/evaluator_trial.py not found in the repository

[ckq328]

Hi DoVer authors and maintainers @MBJinX,

Thank you for releasing this excellent work and the source code!

I am currently trying to reproduce the experiments following the instructions in the README. However, I encountered a missing file issue. When trying to evaluate the baseline using the provided command:

python scripts/evaluator_trial.py --scenario 1

It returns a "file not found" error. It appears that evaluator_trial.py (and potentially the entire scripts/ directory) might have been accidentally omitted when the code was pushed to the repository.

The script should execute the baseline multi-agent system for scenario 1 and generate the necessary logs/traces for DoVer to process.

Could you please upload the missing evaluation scripts, or point me to the correct entry point if the file has been renamed?

Thank you for your time and help!

[MBJinX]

Hi @ckq328 ,

Thank you for your interest in DoVer and for pointing this out!

I have completed my internship and no longer have direct access to the latest internal version of the repository to push the missing files. However, the logic of evaluator_trial.py is quite straightforward. Its primary goal is to extract the final answer from the MagenticOne (M1) system's logs and match it against the Ground Truth.

You can recreate this script using the following logic. The script performs these key steps:

1. Load Ground Truth: Reads the expected answer from Agents_Failure_Attribution/Who&When/Hand-Crafted/{scenario_id}.json.
2. Extract System Output: Parses the execution history (history.json) to find the last TextMessage sent by the MagenticOneOrchestrator.
3. Matching Logic:

• Normalization: Lowercasing, removing punctuation, and handling number words (e.g., matching "3" with "three").
• Substring Match: Checks if the ground truth exists within the final answer.
• Jaccard Similarity: As a fallback, it calculates token-based Jaccard similarity (threshold $\ge 0.8$).

Best regards,
Ming

[ckq328]

Hi @MBJinX ,

Thank you so much for the detailed explanation! Your logic for evaluator_trial.py is very clear.

However, as I continued to the next step, I encountered another missing file error when running log_decomposer.py. It seems that failure_proposer.py (and potentially other scripts in the scripts/ directory) was also omitted from the repository.

Additionally, if you happen to have a local backup of the complete scripts/ folder, would it be possible to share it directly via email or a gist? That would be incredibly helpful to ensure no other dependency files are missing for the subsequent steps.

Thank you again for your time and guidance!

[MBJinX]

Hi @ckq328 ,

Maybe you can contact me with email: mam2022@ion.ac.cn.

Hope I can help you with this missing files.

Ming