Convergence issues when using C-Radio (v2/v3/v4) as Vision Encoder in LLaVA-style pretraining

[haoyi199815]

Hello,
I am attempting to integrate the C-Radio series (v2, v3, v4) as the vision encoder for a MLLM. I am following a standard LLaVA architecture (Vision Encoder + MLP Projector + LLM).

The Setup

• Architecture: ViT (C-Radio) + 2-layer MLP Projector + LLM.
• Training Stage: Pretraining.
• Strategy: Freeze Vision Encoder, Freeze LLM, only train the Projector.
• Data: Standard LLaVA 558K align image-text pairs

The Issue
I have observed that when using c-radiov2, c-radiov3, or c-radiov4, the model fails to converge during this pretraining stage. The loss remains high.
To verify my training pipeline, I tested the exact same setup with other vision encoders:

• SigLIP/SigLIP2: Converges normally.
• DINOv2/DINOv3: Converges normally.
  This suggests the pipeline and hyperparameters are generally correct, but there is a specific incompatibility or configuration mismatch with the C-Radio features.

Are there any known issues or specific requirements when using C-Radio features for MLLM alignment? Any insights or suggestions would be greatly appreciated. Thanks!

[gheinrich]

Hello, we tested all these C-RADIO versions in LLaVA-style MLLM such as Nemotron Nano VL, they should work fine, and they usually work with the same hyperparameters as SigLIP2. We usually use 2x2 pixel unshuffling to further reduce the number of vision tokens.
In your setup, are you pre-normalizing RGB pixel values? By default, C-RADIO expects inputs in the [0, 1] range (https://github.com/NVlabs/RADIO?tab=readme-ov-file#preprocessing).

[haoyi199815]

import torchvision.transforms as T

norm_mean_ = [0.48145466, 0.4578275, 0.40821073]
norm_std_ = [0.26862954, 0.26130258, 0.27577711]

image = Image.open(str(image_path)).convert('RGB')
    
image = T.ToTensor()(image)
if isinstance(input_size, tuple):
    resized_img = T.Resize(input_size, interpolation=T.InterpolationMode.BICUBIC)(image)
elif isinstance(input_size, int):
    resized_img = T.Resize((input_size, input_size), interpolation=T.InterpolationMode.BICUBIC)(image)
else:
    raise ValueError("Wrong Image Processor Crop Size")
pixel_values = resized_img.unsqueeze(0)
norm_mean = torch.Tensor(norm_mean_).view(1, 3, 1, 1)
norm_std = torch.Tensor(norm_std_).view(1, 3, 1, 1)
pixel_values = (pixel_values - norm_mean) / norm_std

Yes, I process the pixel values as this, convert image to RGB and normalize it into [0, 1] range

[mranzinger]

Your normalization is actually converting the inputs to mean-0 std-1 (using imagenet statistics). For radio, you should either hand the model the image after pixel_values = resized_img.unsqueeze(0) (e.g. don't apply norm_mean/std), or you should call model.make_preprocessor_external() when you create it, and then you can hand it the inputs after pixel_values = (pixel_values - norm_mean) / norm_std.

[haoyi199815]

In fact, that’s exactly what I did. I call model.make_preprocessor_external(), and pass the pixel_values = (pixel_values - norm_mean) / norm_std to the radio model.

[mranzinger]

What resolution are you using? Are you tiling?