Support Qwen3.5 mm

[tastelikefeet]

PR type

• Bug Fix
• New Feature
• Document Updates
• More Models or Datasets Support

PR information

Write the detail information belongs to this PR.

Experiment results

Paste your experiment result here(if needed).

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request integrates the Qwen3.5 multi-modal model into the Twinkle framework, expanding its capabilities to handle complex multi-modal tasks. The changes involve a complete overhaul of how multi-modal inputs are processed and fed into the model, ensuring seamless operation and efficient training. A new example is provided to guide users on leveraging these new features for multi-modal model training.

Highlights

• Qwen3.5 Multi-modal Model Support: Introduced comprehensive support for the Qwen3.5 multi-modal model, enabling its use within the Twinkle framework for training and evaluation.
• Multi-modal Training Example: Added a new example (cookbook/mm/fsdp2.py) demonstrating multi-modal training with Qwen3.5-4B, utilizing FSDP (Fully Sharded Data Parallel) and LoRA (Low-Rank Adaptation).
• Refactored Multi-modal Data Processing: Significantly refactored the internal handling of multi-modal data, including image/video preprocessing, embedding merging, and the integration of these processes into the model's forward pass via new global and per-optimizer group hooks.
• Enhanced Multimedia File Loading: Implemented robust utilities for loading multimedia files (images, videos) from various sources, including local paths, URLs, and base64 encoded strings, improving flexibility and reliability.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• README.md
  • Removed outdated quick start guide link.
• README_ZH.md
  • Removed outdated quick start guide link (Chinese version).
• cookbook/mm/fsdp2.py
  • Added new example for multi-modal training with Qwen3.5-4B using FSDP and LoRA.
• src/twinkle/hub/hub.py
  • Modified 'load_dataset' to accept and pass through additional keyword arguments.
• src/twinkle/model/multi_lora.py
  • Updated '_patch_peft' to correctly handle 'PeftModel' activation and assignment.
• src/twinkle/model/transformers/multi_lora_transformers.py
  • Added 'torch' import.
  • Implemented 'register_global_mm_forward_hook' to apply template's 'pre_forward_hook' globally for multi-modal models.
• src/twinkle/model/transformers/transformers.py
  • Added '_handler' attribute to 'OptimizerGroup'.
  • Implemented 'register_mm_forward_hook' and 'unregister_mm_forward_hook' to manage model forward hooks for multi-modal processing.
• src/twinkle/processor/base.py
  • Enhanced 'to_tensor' to handle VLM concatenation fields.
  • Relaxed assertion in '_get_packed_seq_params' to allow batch processing.
  • Expanded '_keys' in 'to_transformers_dict' to include VLM-specific fields ('pixel_values', 'image_grid_thw').
  • Added logic to handle squeezed 'image_thw' in '_collate_macro_batch'.
• src/twinkle/template/base.py
  • Imported 'inspect', 'copy', 'load_image', and 'to_device'.
  • Added 'config' attribute to 'Template' for model configuration.
  • Improved 'preprocess_image' to handle dictionary inputs for image paths/bytes.
  • Refactored '_build_mm_messages' to simplify media handling and support audio placeholders.
  • Removed 'post_encode', '_get_text_embeddings', '_get_vision_embeddings', '_merge_vision_embeddings' methods.
  • Introduced new 'pre_forward_hook' and '_post_encode' for more flexible input processing.
• src/twinkle/template/qwen3_vl.py
  • Imported 'requires' and 'get_inputs_embeds_hf'.
  • Updated 'preprocess_image' and 'preprocess_video' to use 'requires' and 'super().preprocess_image' for robust media loading.
  • Replaced 'post_encode' with '_post_encode' and integrated with the new global hook mechanism for Qwen3-VL's embedding merging.
• src/twinkle/template/utils.py
  • Imported 'TypeVar' and 'to_device'.
  • Fixed 'encode_func' calls to remove trailing commas.
  • Refactored '_transfer_single_message' and 'transfer_to_standard_message' to support audio placeholders and a more generic media handling.
  • Added 'get_inputs_embeds_hf' function for handling multi-modal embeddings for Hugging Face models.
• src/twinkle/utils/init.py
  • Exported 'load_image' and 'load_mm_file' from 'vision_tools'.
• src/twinkle/utils/vision_tools.py
  • Added new file containing 'load_mm_file' and 'load_image' utilities for loading multimedia files from various sources.

Activity

• The pull request was created by tastelikefeet to add support for the Qwen3.5 multi-modal model.
• The author categorized this as a 'New Feature'.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces support for Qwen3.5 multi-modal models, involving significant refactoring in the template and processor components for generic multi-modal input handling, and includes a new example script for Qwen3.5 mm. A critical security vulnerability has been identified in the newly added src/twinkle/utils/vision_tools.py file: the load_mm_file function is susceptible to Server-Side Request Forgery (SSRF) and Path Traversal. This stems from fetching remote URLs and reading local files based on unvalidated user-supplied paths in multimodal messages, requiring strict validation and sanitization. Furthermore, the code review identified an incorrect import and a potential AttributeError that must be addressed, along with suggestions for enhancing the new example script and a utility function for better code quality.

I am having trouble creating individual review comments. Click here to see my feedback.

src/twinkle/model/multi_lora.py (374)

This line _module.active_adapters = _peft_model.active_adapters will likely raise an AttributeError. In this else block, _module is not an instance of PeftModel and therefore does not have the active_adapters attribute. This assignment seems incorrect and will cause a runtime error. The original module is a plain torch.nn.Module or PreTrainedModel which is being wrapped by get_peft_model.

src/twinkle/model/transformers/multi_lora_transformers.py (4)

The import from sympy.printing.pytorch import torch is incorrect. It seems you intended to import the torch library, but you are importing it from sympy. This will lead to errors as it does not provide the PyTorch library. The correct import is import torch.

import torch

src/twinkle/utils/vision_tools.py (21-29)

The load_mm_file function performs an unvalidated HTTP GET request using the path parameter if it starts with "http". Since this function is used to process images and other multimodal files from user-provided messages (e.g., in Template.preprocess_image), an attacker can provide a malicious URL to perform Server-Side Request Forgery (SSRF). This could allow the attacker to scan internal networks, access internal services, or retrieve sensitive metadata from cloud providers.

To remediate this, implement a strict allow-list of domains for remote file loading, or validate that the URL points to a trusted source. Alternatively, provide a configuration option to disable remote file loading if it's not required.

src/twinkle/utils/vision_tools.py (32-34)

The load_mm_file function allows reading arbitrary files from the local filesystem if the path parameter points to an existing file. Since this function is used to process multimodal data from user-supplied messages, an attacker can provide a local file path (e.g., ../../etc/passwd) to read sensitive files. Although the content is subsequently passed to Image.open, the file is already read into memory, and an attacker might be able to infer information about the file or exploit vulnerabilities in the image processing library.

To remediate this, restrict file access to a specific directory (e.g., a temporary upload directory) and validate that the path does not contain directory traversal sequences (e.g., ..). Alternatively, provide a configuration option to disable local file loading.

cookbook/mm/fsdp2.py (31)

The function name eval shadows a Python built-in function. This can lead to confusion and potential bugs. It's a good practice to avoid using names of built-in functions. Please rename it to evaluate and update the call on line 87 accordingly.

def evaluate(model):

cookbook/mm/fsdp2.py (57)

According to PEP 8, imports should generally be at the top of the file. Moving this import to the top level improves readability and helps static analysis tools. It also avoids the overhead of repeated import checks if the function is called multiple times. Please move this import to the top of the file.

cookbook/mm/fsdp2.py (76)

Using float('inf') is more idiomatic for initializing a variable that will hold a minimum value, making the code's intent clearer than using a magic number like 99.0.

    loss_metric = float('inf')

cookbook/mm/fsdp2.py (87)

This should be updated to call the renamed evaluate function to avoid shadowing the built-in eval.

            metrics = evaluate(model)

src/twinkle/utils/vision_tools.py (38)

Using assert for input validation is not robust. Asserts can be disabled with the -O flag in Python, which would remove this check in production environments. It's better to use an if check and raise a ValueError with a clear error message if the data URI format is invalid. This also allows for a more descriptive error message.

                if not match_:
                    raise ValueError("Invalid base64 data URI format.")