Reproducing SQuAD Single-Passage SEAL: Seeds and Passage Selection

[Kelseyaha]

Hi, thanks for the great work!

Regarding the SQuAD knowledge-incorporation (single-passage) experiments: I checked the provided data and found that the two ReSTEM rounds use different batches of 50 passages.

To fully reproduce the results, could you clarify: How were the 50 passages for Round 1 and Round 2 selected from SQuAD-train? What the exact random seeds and start index were used?

Thanks!

[Kelseyaha]

I have one more question regarding the data used in the SQuAD single-passage setting. I noticed that, in the released data, the 50 passages used for Round 1 in the main reported experiments are not the same 50 passages used in the alternate_prompts directory. The pipeline appears to first extract 100 passages, and the model is trained on a subset of 50—but this subset does not match the 50 passages in alternate_prompts. In addition, using the seeds provided in the repo, I am unable to reproduce either of these selections exactly.

Could you clarify why the 50 passages used in the main reported Round 1 results differ from the 50 passages used in alternate_prompts? This would greatly help with fully reproducing the experiments. Thank you again!