About reproducibility #7
-
|
Hi, thank a lot for your paper. I'm interested in reproducing your benchmark, what hardware did you use for it? I would ask in the discussions, but they don't exist. Ideally, if you added it to the README file. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
|
Hi! Thanks for the suggestion regarding discussions, I made them available. Regarding reproducibility – for the bulk of the experiments we've used A100 GPUs (the only exception I can think of is CatBoost which is significantly better on CPUs – so we ran it on cpu). I would not expect byte-level reproducibility if you use a slightly different GPU or newer environment – we did this in https://github.com/yandex-research/tabm for example (updated and slighlty different env). Nevertheless, results should still be informative. So there is no general remark regarding hardware. Side note: you can actually find the hardware used for each run in the |
Beta Was this translation helpful? Give feedback.
Hi! Thanks for the suggestion regarding discussions, I made them available.
Regarding reproducibility – for the bulk of the experiments we've used A100 GPUs (the only exception I can think of is CatBoost which is significantly better on CPUs – so we ran it on cpu). I would not expect byte-level reproducibility if you use a slightly different GPU or newer environment – we did this in https://github.com/yandex-research/tabm for example (updated and slighlty different env). Nevertheless, results should still be informative. So there is no general remark regarding hardware.
Side note: you can actually find the hardware used for each run in the
report.jsonfiles like that (link)