very long inference time with Mistral-7b-sft-beta

[skepsun]

Thanks for your great work. I want to apply RAIN to Mistral ([https://huggingface.co/HuggingFaceH4/mistral-7b-sft-beta]) on HH test set. I only changed scripts of loading dataset in main.py (I want inference on entire test set, thus I just set ds1 = ds):

ds = load_dataset("Anthropic/hh-rlhf", split="test")
ds = ds.shuffle(seed=42)
ds1 = ds
# ds1 = ds.select(range(args.start, args.end))

Then I run this command:

python allocation.py --modelname path_to_mistral --nump 8 --outdir mistral --s 0 --e 1

The task lasts for 17 hours, but in res_x.json(x=0,1,2,3,4,5,6,7) there is still only the same first question with answers. I think --s and --e will not work, no matter how I set them. But multiple processes seem to inference on the same data? And the inference is very very very slow ...