Question Regarding TA-Tok Training Code

[nihaotian1]

Dear authors,

Thank you for your great work! From my understanding, the training pipeline involves first pre-training the TA-Tok tokenizer and de-tokenizer, followed by jointly training the full MLLM. I've found the code for the MLLM training using the released TA-Tok checkpoint. Would it be possible to also release the code for pre-training the TA-Tok components themselves?

Btw, I'm also wondering how many GPU hours it takes to train TA-Tok themselves?

[csuhan]

Yes we will release the tokenizer training code soon.

We train TA-Tok on 128GPU for around 2 days. But it does not mean that you need such long GPU hours and so many data. We do it just for better convergence.