How to finetune audio duplex ability?

[edward15241]

This work and demo is very impressive. 😎 👍 👍

I'm interest in fine-tuning the model, particularly for improving speech duplex interaction (simultaneous listening and speaking) and interruption handling. However, it would be important to avoid catastrophic forgetting that might degrade these capabilities.

I would appreciate some clarification about the training design:

1. Interruption training
   How is the "barge-in" trained in this model? Is it implemented in a way similar to Moshi-style streaming speech interaction, or FLM-Audio style duplex conversational modeling?

2. Duplex interaction (listen while speaking)
   How is the model trained to listen while speaking? Does the training data contain overlapping speech segments or a special interaction format that enables duplex behavior and monologue generation?

3. About Some finetuning details
   If we want to fine-tune data to keep duplex capabilities, How to input data to model? ( I assume the training data might follow a format similar to the Hugging Face chat template. However, I'm not sure how barge-in events or interruption labels are encoded in the dataset)

Thank you very much 😍

[tc-mb]

Hi @edward15241, thank you for your interest in MiniCPM-o and these great questions!

We're currently working on preparing the finetuning code and documentation, including details around duplex and speech interaction capabilities. We'll share more once it's ready.

We appreciate your patience and stay tuned!

[bokesyo]

Hi @edward15241 , thank you a lot for your feedback!
I got your question, and I will write a detailed documentation about your question about model training and data schema in the next two days. And by the end of this month we will open-source the duplex mode fine-tuning codebase. 🤗