ValueError: Input image size (112*1036) doesn't match model ([112, 1036]*[112, 1036]).

[JeffRody]

使用最新的transformers 4.47.0.dev0
删除 improt _expand_mask 改为自定义
def _expand_mask(mask: torch.Tensor, dtype: torch.dtype, tgt_len: Optional[int] = None):
"""
Expands attention_mask from [bsz, seq_len] to [bsz, 1, tgt_seq_len, src_seq_len].
"""
bsz, src_len = mask.size()
tgt_len = tgt_len if tgt_len is not None else src_len
expanded_mask = mask[:, None, None, :].expand(bsz, 1, tgt_len, src_len).to(dtype)
inverted_mask = 1.0 - expanded_mask
return inverted_mask.masked_fill(inverted_mask.to(torch.bool), torch.finfo(dtype).min)

其它模态可以运行，但是modeling_audio出现以下报错，请教一下应该如何修改：
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Traceback (most recent call last):
File "/home/wanglch/projects/LanguageBind/inference.py", line 43, in
embeddings = model(inputs)
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/home/wanglch/projects/LanguageBind/languagebind/init.py", line 78, in forward
value = self.modality_encoderkey[1]
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/home/wanglch/projects/LanguageBind/languagebind/audio/modeling_audio.py", line 656, in forward
hidden_states = self.embeddings(pixel_values)
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 244, in forward
raise ValueError(
ValueError: Input image size (1121036) doesn't match model ([112, 1036][112, 1036]).

[SoyeonHH]

I ran into the same issue, but it was probably due to a version difference.

I made the following changes to the modeling_audio file and it fixed the problem.

from transformers.modeling_attn_mask_utils import _prepare_4d_attention_mask

attention_mask = _prepare_4d_attention_mask(attention_mask, hidden_states.dtype)

[1579364808]

你们解决这个问题了吗?我这样设置还是没有解决

I ran into the same issue, but it was probably due to a version difference.

I made the following changes to the modeling_audio file and it fixed the problem.

from transformers.modeling_attn_mask_utils import _prepare_4d_attention_mask

attention_mask = _prepare_4d_attention_mask(attention_mask, hidden_states.dtype)

[1579364808]

我通过修改CLIPVisionEmbeddings的forward方法解决了这个问题

在 CLIPVisionEmbeddings 类的 forward 方法里

def forward(self, pixel_values: torch.FloatTensor, interpolate_pos_encoding=False) -> torch.Tensor:
batch_size, _, height, width = pixel_values.shape

# --- 开始修改 ---
# 检查 self.image_size 是列表（如 [H, W]）还是整数（如 224）
if isinstance(self.image_size, (list, tuple)):
    # 如果是列表，分别比较高和宽
    expected_height, expected_width = self.image_size
    if not interpolate_pos_encoding and (height != expected_height or width != expected_width):
        raise ValueError(
            f"Input image size ({height}*{width}) doesn't match model ({expected_height}*{expected_width})."
        )
else:
    # 如果是整数，保持原来的逻辑（适用于正方形图像）
    if not interpolate_pos_encoding and (height != self.image_size or width != self.image_size):
        raise ValueError(
            f"Input image size ({height}*{width}) doesn't match model ({self.image_size}*{self.image_size})."
        )
# --- 结束修改 ---

target_dtype = self.patch_embedding.weight.dtype
patch_embeds = self.patch_embedding(pixel_values.to(dtype=target_dtype))  # shape = [*, width, grid, grid]
patch_embeds = patch_embeds.flatten(2).transpose(1, 2)

class_embeds = self.class_embedding.expand(batch_size, 1, -1)
embeddings = torch.cat([class_embeds, patch_embeds], dim=1)
if interpolate_pos_encoding:
    embeddings = embeddings + self.interpolate_pos_encoding(embeddings, height, width)
else:
    embeddings = embeddings + self.position_embedding(self.position_ids)
return embeddings