Guidance Request: Ideal Way to Adapt MedGemma for Multi-View Medical Images (5-View Case)

[blackpearl006]

Context

We are using MedGemma 1.5 on a medical imaging task where each sample consists of 5 correlated images from the same patient (breast thermography: frontal, left/right oblique, left/right lateral).

Current MedGemma documentation and examples focus on single-image inputs, so we are seeking confirmation of the ideal and recommended adaptation strategy for this multi-view setting.

Our Understanding of the Ideal Approach

For a fixed multi-view medical imaging problem (5 views per case, ~3,000 cases), the most appropriate approach appears to be:

Late Fusion (Feature-Level Fusion)

• Encode each view independently using the MedGemma (or MedSigLIP) image encoder with shared weights
• Fuse per-view embeddings using concatenation, attention, or a small transformer
• Train a lightweight task-specific head on top of the fused representation

This preserves per-view semantics, scales well, and aligns with standard practice in multi-view medical imaging literature.

Alternatives (Less Ideal)

• Image montage (early fusion): simple but loses per-view structure and resolution
• Multi-image prompt-only fusion: possible for exploration, but unclear whether the vision encoder is designed to jointly reason over multiple images in a single request

Questions

• Is feature-level late fusion the recommended pattern for multi-view medical imaging with MedGemma?
• Can the MedGemma image encoder be reliably used as a frozen feature extractor for this setup?
• Are there reference examples, benchmarks, or internal guidance for multi-image medical use cases?

Use Case Summary

• Task: Breast cancer detection from thermography
• Input: 5 fixed views per patient
• Dataset size: ~3,000 cases
• Output: Binary classification + localization

Any confirmation or guidance on this would help ensure correct and safe use of MedGemma in multi-view medical workflows.

Thank you.

[mgultekin]

Hi Ninad! I have the similar issue as you do. Could you find an answer for your question? I am also trying to find out how to use multiple images coming from the same patient for a single prompt and single prediction.

[srikantak-221]

Hey @blackpearl006 ,
Thank you for the insightful question. For a fixed multi-view setup such as yours, which involves 5 correlated thermography views per patient, feature-level late fusion represents the most principled and stable approach. The methodology involves independently encoding each image using a MedGemma / MedSigLIP vision encoder with shared weights. Subsequently, the resulting per-view embeddings are fused, typically via concatenation + MLP, attention pooling, or a small transformer operating over the 5 tokens and then utilised to train a lightweight task-specific prediction head. This method preserves per-view semantics, eliminates spatial distortion associated with montage-style early fusion, and aligns with standard multi-view practices in medical imaging.
Utilising the vision encoder as a frozen feature extractor is a highly recommended starting point for a dataset comprising approximately 3,000 cases. It is advisable to train the fusion module and prediction head first; if necessary, one can later explore partial fine-tuning for the final transformer blocks or low-rank adaptation using a conservative learning rate. Although multi-image prompting is feasible at the multimodal level, its joint reasoning primarily occurs within the language model layers, making it better suited for generative/reporting tasks rather than structured classification and localization. For your specific use case, explicit feature-level aggregation will provide superior control, data efficiency, and interpretability.
Thank you!