I tried the data mixture: r2r+rxr+scanqa+human(all of these I sampled 2% for them) to finetune the model navila-llama3-8b-8f (already sft-trained) with low learning rate about 5e-5, getting worse success rate while evaluating. Then I tried to delete the r2r dataset getting similar results.
Since I want to construct my own dataset to finetune this model for special scenes, I wonder how to mix my dataset and which special samples I should augment. I'd strongly appreciate it if you could give some advice on learning rate or other parameters!
I tried the data mixture: r2r+rxr+scanqa+human(all of these I sampled 2% for them) to finetune the model navila-llama3-8b-8f (already sft-trained) with low learning rate about 5e-5, getting worse success rate while evaluating. Then I tried to delete the r2r dataset getting similar results.
Since I want to construct my own dataset to finetune this model for special scenes, I wonder how to mix my dataset and which special samples I should augment. I'd strongly appreciate it if you could give some advice on learning rate or other parameters!