Training on Bridge V2 Offline Dataset

[derektan95]

Hi @ColinQiyangLi ,

Thanks for this amazing work! I am currently trying to train your QC flow-matching policy with the Bridge V2 offline dataset, a dataset that's often used to train generalist policies for evaluation in e.g. SIMPLER environment. My understanding from your paper is that it is possible especially for the offline training stage.

May I have a rough sensing on how I could adapt your code to train on Bridge V2? Any specific guidance on code / hyperparameters in this repository that I should pay more attention? Thanks!

Warm Regards,
Derek

[verityw]

If you are interested in massively multi-task training, I would suggest looking at OpenVLA. For single- / few-task (which I would recommend for limited compute), maybe see this. The former's implementation of dataloading w/. TFDS and dlimp should be helpful in either case.

Note however that chunking with Bridge policies may be somewhat challenging; the data is collected at a fairly low frequency (with most policies for it being run at like 4-6 Hz), and as such, the action magnitude is quite large. Thus, it's unclear how much things like temporal consistency bought by Q-chunking would help with it.

[derektan95]

Hi @verityw ,

Thanks for your comments. I should be able to get the dataloader up with your references.

On your second point, I realized there are a few notable works that uses action chunking in SIMPLER environment (e.g. open-pi-zero), so in theory it should be possible to use the Q-chunking framework to train a action chunking policy in the SIMPLER environment?

[ColinQiyangLi]

Hi @derektan95, thanks for your kind words and interest in our work!

For getting started on a new dataset, I would recommend start with the best-of-n version (config["actor_type"] = "best-of-n") with a low n (e.g., n=1 is BC). What might be helpful is to have a simple task where you know action chunking BC should work reasonably well (better than not using action chunking). And then tune the best-of-n parameter (or try the fql version, config["actor_type"] = "distill-ddpg" to tune the alpha parameter).

Hope this is helpful and let me know if you have any other questions!

[derektan95]

Hi @ColinQiyangLi ,

Thanks for your response, I will definitely give this a shot. Just to check, if I want to train a strong 1-step TD baseline (to be trained with bridge dataset), would you recommend QC-FQL or simply FQL?

Another side question - have you tried training QC with image inputs and language inputs? Perhaps something similar to V-GPS. Would love to get some guidance on how to tune your config to make this work, as I'm unsure if introducing higher dimensional inputs into this RL framework would introduce training instability.

Thanks again!