It should be possible to give replay patterns in case a benchmarks failed, together with a post mortem of the failed benchmark execution (eg files with stdin and stdout). It's otherwise a source of stress when closing in on deadlines with large evaluation in the pipe. In other words, one failed benchmark shouldn't compromise an entire campaign.