[data, model] feat: support Qwen3-VL textual token-based time encoding

[GarrickBot]

Overview

This PR integrates the specific data processing and modeling logic required for Qwen3-VL, specifically focusing on its unique textual token-based time encoding strategy.

Unlike previous model versions, Qwen3-VL requires explicit time tags (e.g., <t 1.5 seconds>) inserted into the context before visual tokens. This PR implements the official timestamp calculation logic, adds a dedicated data processor for Qwen3-VL video samples, and optimizes the underlying video loading utilities for better memory efficiency during training.

Design & Code Changes

The changes are primarily concentrated in the data processing pipeline and chat template adapter:

• multimodal_chat_template.py:

  • New Class: Added Qwen3VLChatTemplate inheriting from Qwen2VLTemplate.
  • Timestamp Logic: Implemented _calculate_timestamps to replicate the official strategy: padding frame indices, converting to seconds, and averaging based on temporal_patch_size=2.
  • Message Encoding: Updated encode_messages to dynamically inject <t {timestamp} seconds> tags and <|vision_start|>/<|vision_end|> tokens into the input stream.

• vlm_data_process.py:

  • New Processor: Added process_sample_qwen3_vl to handle the specific requirements of Qwen3-VL video inputs.
  • Data Adapter: Added logic to automatically construct the conversations field from question and answer keys if the standard conversation format is missing.
  • Metadata Fix: Updated video metadata handling to correctly generate and assign frames_indices.

• video_utils.py:

  • Workflow Optimization: Refactored load_video_from_bytes_list to directly support pre-stored frame byte lists as input. This bypasses the need for online frame extraction (decoding), significantly reducing memory overhead.
  • Metadata Adaptation: Updated the function to correctly handle input VideoMetadata and ensure the output includes the modified VideoMetadata (reflecting changes in FPS or frame counts) for downstream processing.

Data Format

The new data processor (process_sample_qwen3_vl_video_r1) expects input samples to follow this schema:

sample = {
    # 1. 'conversations' is required (or auto-constructed from question/answer)
    "conversations": [
        {
            "role": "user", 
            "content": [
                {"type": "video"}, 
                {"type": "text", "text": "Describe this video."}
            ]
        },
        {"role": "assistant", "content": "This is a video of..."}
    ],
    
    # 2. 'fps' must be a list containing the float FPS value
    "fps": [2.0],
    
    # 3. 'videos' must be a list of raw byte frames
    "videos": [
        [b'\xff\xd8...', b'\xff\xd8...', ...] # List[bytes]
    ]
}

[CLAassistant]

All committers have signed the CLA.

[gemini-code-assist]

Code Review

This pull request introduces support for Qwen3-VL's textual token-based time encoding. The changes are well-structured, adding a new chat template, a data processor, and optimizing video loading. However, I've identified a critical issue with a duplicate function definition that will cause the new logic to be ignored, and a couple of high-severity issues related to potential crashes and unexpected side effects. Please review the detailed comments.

[Coach257]

Can we merge this function into fetch_videos? It seems return metadata will be necessary, so fetch_videos returns videos, video_metadata, audios, audio_metadata

[GarrickBot]

I have updated the logic to support returning the full set of 4 values (video, video_metadata, audio, audio_metadata). However, I decided to keep fetch_videos_metadata separate from fetch_videos.
This design ensures isolation: fetch_videos_metadata is for the new pipeline requiring full metadata, while fetch_videos falls back to the original behavior (returning 2 values) to ensure backward compatibility for other tasks.

[Coach257]

As the returns[1] is metadata, the second return should be a dict.

[GarrickBot]

The return value was actually already a List[Dict], but the previous variable name final_fps_inputs was indeed misleading (it sounded like a list of floats).
I have renamed the variables in the latest commit to explicitly reflect that they are lists of metadata dictionaries, which should clear up the confusion.

[Coach257]

@piyifan123 Please check if there's any conflict with the origin process_sample_qwen3_vl. If no, use this new function?