Missing attention mask for input images

[AmitMY]

When collating different words, their image size is variable.
Currently, it is 0-padded, but this is not something the model should care about.
Instead, we need to add an attention mask for the pixels as well, to prevent the model from behaving differently when the text is batched with longer or shorter texts

[AmitMY]

Huggingface issue: https://discuss.huggingface.co/t/image-encoder-inference-with-variable-size-imags/166910

[AmitMY]

Addressed the determinism issue in 0f41645 - but my solution is slow.

Should use parallelizable models like dino 3 to solve that issue. facebookresearch/dinov3#27 (comment)

For now the solution is to render all images in the same size

[AmitMY]

Ideally, our collator should collate all images into a tensor of [B, L, H, W, 3] - this will prevent current issues with data transfer (such as relieved here c19b7e6)

Currently, we pass a list of a nested tensor.

Once we can pass a [B, L, H, W, 3] tensor, padded with 0, and that is deterministic (0s are ignored correctly) we can:

1. Use stack tensors instead of torch.nested from images https://github.com/sign/image-latent-transformer/blob/c19b7e67216fa541dac2142d941dcdc4337a94e1/image_latent_transformer/processor.py#L136-L138
2. Remove this type: https://github.com/sign/image-latent-transformer/blob/c19b7e67216fa541dac2142d941dcdc4337a94e1/image_latent_transformer/batch_image_encoder.py#L11
3. Remove this hack: https://github.com/sign/image-latent-transformer/blob/c19b7e67216fa541dac2142d941dcdc4337a94e1/image_latent_transformer/batch_image_encoder.py#L69-L70
4. Never encode images sequentially https://github.com/sign/image-latent-transformer/blob/c19b7e67216fa541dac2142d941dcdc4337a94e1/image_latent_transformer/batch_image_encoder.py#L133C5-L147
5. Remove fixed renderer size https://github.com/sign/image-latent-transformer/blob/c19b7e67216fa541dac2142d941dcdc4337a94e1/image_latent_transformer/renderer.py#L71-L73
6. Remove special case for collator https://github.com/sign/image-latent-transformer/blob/c19b7e67216fa541dac2142d941dcdc4337a94e1/image_latent_transformer/collator.py#L89-L92

[AmitMY]

This costs about 1.6x in training performance which is huge.
After merging this: #17
The only thing that is left is to figure out how to create proper attention masks
and then this implementation gets us our speedup:
https://github.com/sign/image-latent-transformer/blob/735fb1daf7e6f1f3e6fbea7238b0af01cf891ac9/image_latent_transformer/batch_image_encoder.py#L11-L22

[AmitMY]

I have now created two implementations:

`modeling_vit.py` with attention masking

# coding=utf-8
# Copyright 2021 Google AI, Ross Wightman, The HuggingFace Inc. team. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""PyTorch ViT model."""

import collections.abc
import math
from typing import Callable, Optional, Union

import torch
import torch.utils.checkpoint
from torch import nn

from ...activations import ACT2FN
from ...modeling_layers import GradientCheckpointingLayer
from ...modeling_outputs import (
    BaseModelOutput,
    BaseModelOutputWithPooling,
    ImageClassifierOutput,
    MaskedImageModelingOutput,
)
from ...modeling_utils import ALL_ATTENTION_FUNCTIONS, PreTrainedModel
from ...processing_utils import Unpack
from ...pytorch_utils import find_pruneable_heads_and_indices, prune_linear_layer
from ...utils import TransformersKwargs, auto_docstring, logging, torch_int
from ...utils.generic import can_return_tuple, check_model_inputs
from .configuration_vit import ViTConfig


logger = logging.get_logger(__name__)


class ViTEmbeddings(nn.Module):
    """
    Construct the CLS token, position and patch embeddings. Optionally, also the mask token.
    """

    def __init__(self, config: ViTConfig, use_mask_token: bool = False):
        super().__init__()

        self.cls_token = nn.Parameter(torch.randn(1, 1, config.hidden_size))
        self.mask_token = nn.Parameter(torch.zeros(1, 1, config.hidden_size)) if use_mask_token else None
        self.patch_embeddings = ViTPatchEmbeddings(config)
        num_patches = self.patch_embeddings.num_patches
        self.position_embeddings = nn.Parameter(torch.randn(1, num_patches + 1, config.hidden_size))
        self.dropout = nn.Dropout(config.hidden_dropout_prob)
        self.patch_size = config.patch_size
        self.config = config
        self.mask_padded = False  # opt-in from ViTModel via model.mask_padded

    def interpolate_pos_encoding(self, embeddings: torch.Tensor, height: int, width: int) -> torch.Tensor:
        """
        This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution
        images. This method is also adapted to support torch.jit tracing.

        Adapted from:
        - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and
        - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211
        """

        num_patches = embeddings.shape[1] - 1
        num_positions = self.position_embeddings.shape[1] - 1

        # always interpolate when tracing to ensure the exported model works for dynamic input shapes
        if not torch.jit.is_tracing() and num_patches == num_positions and height == width:
            return self.position_embeddings

        class_pos_embed = self.position_embeddings[:, :1]
        patch_pos_embed = self.position_embeddings[:, 1:]

        dim = embeddings.shape[-1]

        new_height = height // self.patch_size
        new_width = width // self.patch_size

        sqrt_num_positions = torch_int(num_positions**0.5)
        patch_pos_embed = patch_pos_embed.reshape(1, sqrt_num_positions, sqrt_num_positions, dim)
        patch_pos_embed = patch_pos_embed.permute(0, 3, 1, 2)

        patch_pos_embed = nn.functional.interpolate(
            patch_pos_embed,
            size=(new_height, new_width),
            mode="bicubic",
            align_corners=False,
        )

        patch_pos_embed = patch_pos_embed.permute(0, 2, 3, 1).view(1, -1, dim)

        return torch.cat((class_pos_embed, patch_pos_embed), dim=1)

    def forward(
        self,
        pixel_values: torch.Tensor,
        bool_masked_pos: Optional[torch.BoolTensor] = None,
        interpolate_pos_encoding: bool = False,
    ) -> torch.Tensor:
        batch_size, num_channels, height, width = pixel_values.shape
        embeddings = self.patch_embeddings(pixel_values, interpolate_pos_encoding=interpolate_pos_encoding)

        if bool_masked_pos is not None:
            seq_length = embeddings.shape[1]
            mask_tokens = self.mask_token.expand(batch_size, seq_length, -1)
            # replace the masked visual tokens by mask_tokens
            mask = bool_masked_pos.unsqueeze(-1).type_as(mask_tokens)
            embeddings = embeddings * (1.0 - mask) + mask_tokens * mask

        # add the [CLS] token to the embedded patch tokens
        cls_tokens = self.cls_token.expand(batch_size, -1, -1)
        embeddings = torch.cat((cls_tokens, embeddings), dim=1)

        # add positional encoding to each token
        if interpolate_pos_encoding:
            if getattr(self, "mask_padded", False):
                # --- tight-grid positional encodings (content-only), then scatter into big grid ---
                dim = embeddings.shape[-1]
                ph, pw = self.patch_size if isinstance(self.patch_size, collections.abc.Iterable) else (self.patch_size,
                                                                                                        self.patch_size)
                Hpatch, Wpatch = height // ph, width // pw

                # Detect fully-zero patches
                patches = nn.functional.unfold(pixel_values, kernel_size=(ph, pw), stride=(ph, pw))  # (B, C*ph*pw, N)
                zero_patch = (patches == 0).all(dim=1)  # (B, N)
                valid_patch = ~zero_patch  # (B, N)
                valid_grid = valid_patch.view(batch_size, Hpatch, Wpatch)

                row_has = valid_grid.any(dim=2)  # (B, Hpatch)
                col_has = valid_grid.any(dim=1)  # (B, Wpatch)
                hs = row_has.sum(dim=1).tolist()  # per-sample content height (in patches)
                ws = col_has.sum(dim=1).tolist()  # per-sample content width  (in patches)
                h_small_max = max(hs) if hs else 1
                w_small_max = max(ws) if ws else 1

                # Ask for small-grid positions once at (h_small_max, w_small_max), then slice per-sample
                dummy = embeddings.new_zeros(1, 1 + h_small_max * w_small_max, dim)
                pos_small_max = self.interpolate_pos_encoding(dummy, ph * h_small_max, pw * w_small_max)  # (1, 1+.., D)
                cls_pos = pos_small_max[:, :1, :]  # (1,1,D)
                patch_pos_small = pos_small_max[:, 1:, :].reshape(1, h_small_max, w_small_max, dim)

                pos_big = embeddings.new_zeros(batch_size, Hpatch, Wpatch, dim)
                for b in range(batch_size):
                    hs_b, ws_b = hs[b], ws[b]
                    if hs_b > 0 and ws_b > 0:
                        pos_big[b, :hs_b, :ws_b, :] = patch_pos_small[0, :hs_b, :ws_b, :]
                pos_big = pos_big.view(batch_size, Hpatch * Wpatch, dim)

                embeddings = embeddings + torch.cat((cls_pos.expand(batch_size, -1, -1), pos_big), dim=1)
            else:
                embeddings = embeddings + self.interpolate_pos_encoding(embeddings, height, width)
        else:
            embeddings = embeddings + self.position_embeddings
        embeddings = self.dropout(embeddings)

        return embeddings


class ViTPatchEmbeddings(nn.Module):
    """
    This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial
    `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a
    Transformer.
    """

    def __init__(self, config: ViTConfig):
        super().__init__()
        image_size, patch_size = config.image_size, config.patch_size
        num_channels, hidden_size = config.num_channels, config.hidden_size

        image_size = image_size if isinstance(image_size, collections.abc.Iterable) else (image_size, image_size)
        patch_size = patch_size if isinstance(patch_size, collections.abc.Iterable) else (patch_size, patch_size)
        num_patches = (image_size[1] // patch_size[1]) * (image_size[0] // patch_size[0])
        self.image_size = image_size
        self.patch_size = patch_size
        self.num_channels = num_channels
        self.num_patches = num_patches

        self.projection = nn.Conv2d(num_channels, hidden_size, kernel_size=patch_size, stride=patch_size)

    def forward(self, pixel_values: torch.Tensor, interpolate_pos_encoding: bool = False) -> torch.Tensor:
        batch_size, num_channels, height, width = pixel_values.shape
        if num_channels != self.num_channels:
            raise ValueError(
                "Make sure that the channel dimension of the pixel values match with the one set in the configuration."
                f" Expected {self.num_channels} but got {num_channels}."
            )
        if not interpolate_pos_encoding:
            if height != self.image_size[0] or width != self.image_size[1]:
                raise ValueError(
                    f"Input image size ({height}*{width}) doesn't match model"
                    f" ({self.image_size[0]}*{self.image_size[1]})."
                )
        embeddings = self.projection(pixel_values).flatten(2).transpose(1, 2)
        return embeddings


def eager_attention_forward(
    module: nn.Module,
    query: torch.Tensor,
    key: torch.Tensor,
    value: torch.Tensor,
    attention_mask: Optional[torch.Tensor],
    scaling: float,
    dropout: float = 0.0,
    **kwargs,
):
    # Take the dot product between "query" and "key" to get the raw attention scores.
    attn_weights = torch.matmul(query, key.transpose(-1, -2)) * scaling

    # Allow modules to request pre-softmax masking (logits = -inf where masked)
    if getattr(module, "_use_pre_softmax_mask", False) and attention_mask is not None:
        mask_bool = attention_mask.bool() if attention_mask.dtype != torch.bool else attention_mask
        attn_weights = attn_weights.masked_fill(~mask_bool, torch.finfo(attn_weights.dtype).min)

    # Normalize the attention scores to probabilities.
    attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)

    # This is actually dropping out entire tokens to attend to, which might
    # seem a bit unusual, but is taken from the original Transformer paper.
    attn_weights = nn.functional.dropout(attn_weights, p=dropout, training=module.training)

    # Mask heads if we want to
    if attention_mask is not None and not getattr(module, "_use_pre_softmax_mask", False):
        attn_weights = attn_weights * attention_mask

    attn_output = torch.matmul(attn_weights, value)
    attn_output = attn_output.transpose(1, 2).contiguous()

    return attn_output, attn_weights


class ViTSelfAttention(nn.Module):
    def __init__(self, config: ViTConfig):
        super().__init__()
        if config.hidden_size % config.num_attention_heads != 0 and not hasattr(config, "embedding_size"):
            raise ValueError(
                f"The hidden size {config.hidden_size} is not a multiple of the number of attention "
                f"heads {config.num_attention_heads}."
            )

        self.config = config
        self.num_attention_heads = config.num_attention_heads
        self.attention_head_size = int(config.hidden_size / config.num_attention_heads)
        self.all_head_size = self.num_attention_heads * self.attention_head_size
        self.dropout_prob = config.attention_probs_dropout_prob
        self.scaling = self.attention_head_size**-0.5
        self.is_causal = False

        self.query = nn.Linear(config.hidden_size, self.all_head_size, bias=config.qkv_bias)
        self.key = nn.Linear(config.hidden_size, self.all_head_size, bias=config.qkv_bias)
        self.value = nn.Linear(config.hidden_size, self.all_head_size, bias=config.qkv_bias)

    def forward(
        self, hidden_states: torch.Tensor, head_mask: Optional[torch.Tensor] = None
    ) -> tuple[torch.Tensor, torch.Tensor]:
        batch_size = hidden_states.shape[0]
        new_shape = batch_size, -1, self.num_attention_heads, self.attention_head_size

        key_layer = self.key(hidden_states).view(*new_shape).transpose(1, 2)
        value_layer = self.value(hidden_states).view(*new_shape).transpose(1, 2)
        query_layer = self.query(hidden_states).view(*new_shape).transpose(1, 2)

        attention_interface: Callable = eager_attention_forward
        if self.config._attn_implementation != "eager":
            attention_interface = ALL_ATTENTION_FUNCTIONS[self.config._attn_implementation]

        context_layer, attention_probs = attention_interface(
            self,
            query_layer,
            key_layer,
            value_layer,
            head_mask,
            is_causal=self.is_causal,
            scaling=self.scaling,
            dropout=0.0 if not self.training else self.dropout_prob,
        )

        new_context_layer_shape = context_layer.size()[:-2] + (self.all_head_size,)
        context_layer = context_layer.reshape(new_context_layer_shape)

        return context_layer, attention_probs


class ViTSelfOutput(nn.Module):
    """
    The residual connection is defined in ViTLayer instead of here (as is the case with other models), due to the
    layernorm applied before each block.
    """

    def __init__(self, config: ViTConfig):
        super().__init__()
        self.dense = nn.Linear(config.hidden_size, config.hidden_size)
        self.dropout = nn.Dropout(config.hidden_dropout_prob)

    def forward(self, hidden_states: torch.Tensor, input_tensor: torch.Tensor) -> torch.Tensor:
        hidden_states = self.dense(hidden_states)
        hidden_states = self.dropout(hidden_states)
        return hidden_states


class ViTAttention(nn.Module):
    def __init__(self, config: ViTConfig):
        super().__init__()
        self.attention = ViTSelfAttention(config)
        self.output = ViTSelfOutput(config)
        self.pruned_heads = set()

    def prune_heads(self, heads: set[int]):
        if len(heads) == 0:
            return
        heads, index = find_pruneable_heads_and_indices(
            heads, self.attention.num_attention_heads, self.attention.attention_head_size, self.pruned_heads
        )

        # Prune linear layers
        self.attention.query = prune_linear_layer(self.attention.query, index)
        self.attention.key = prune_linear_layer(self.attention.key, index)
        self.attention.value = prune_linear_layer(self.attention.value, index)
        self.output.dense = prune_linear_layer(self.output.dense, index, dim=1)

        # Update hyper params and store pruned heads
        self.attention.num_attention_heads = self.attention.num_attention_heads - len(heads)
        self.attention.all_head_size = self.attention.attention_head_size * self.attention.num_attention_heads
        self.pruned_heads = self.pruned_heads.union(heads)

    def forward(self, hidden_states: torch.Tensor, head_mask: Optional[torch.Tensor] = None) -> torch.Tensor:
        self_attn_output, _ = self.attention(hidden_states, head_mask)
        output = self.output(self_attn_output, hidden_states)
        return output


class ViTIntermediate(nn.Module):
    def __init__(self, config: ViTConfig):
        super().__init__()
        self.dense = nn.Linear(config.hidden_size, config.intermediate_size)
        if isinstance(config.hidden_act, str):
            self.intermediate_act_fn = ACT2FN[config.hidden_act]
        else:
            self.intermediate_act_fn = config.hidden_act

    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
        hidden_states = self.dense(hidden_states)
        hidden_states = self.intermediate_act_fn(hidden_states)
        return hidden_states


class ViTOutput(nn.Module):
    def __init__(self, config: ViTConfig):
        super().__init__()
        self.dense = nn.Linear(config.intermediate_size, config.hidden_size)
        self.dropout = nn.Dropout(config.hidden_dropout_prob)

    def forward(self, hidden_states: torch.Tensor, input_tensor: torch.Tensor) -> torch.Tensor:
        hidden_states = self.dense(hidden_states)
        hidden_states = self.dropout(hidden_states)
        hidden_states = hidden_states + input_tensor
        return hidden_states


class ViTLayer(GradientCheckpointingLayer):
    """This corresponds to the Block class in the timm implementation."""

    def __init__(self, config: ViTConfig):
        super().__init__()
        self.chunk_size_feed_forward = config.chunk_size_feed_forward
        self.seq_len_dim = 1
        self.attention = ViTAttention(config)
        self.intermediate = ViTIntermediate(config)
        self.output = ViTOutput(config)
        self.layernorm_before = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
        self.layernorm_after = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)

    def forward(self, hidden_states: torch.Tensor, head_mask: Optional[torch.Tensor] = None) -> torch.Tensor:
        hidden_states_norm = self.layernorm_before(hidden_states)
        attention_output = self.attention(hidden_states_norm, head_mask)

        # first residual connection
        hidden_states = attention_output + hidden_states

        # in ViT, layernorm is also applied after self-attention
        layer_output = self.layernorm_after(hidden_states)
        layer_output = self.intermediate(layer_output)

        # second residual connection is done here
        layer_output = self.output(layer_output, hidden_states)

        return layer_output


class ViTEncoder(nn.Module):
    def __init__(self, config: ViTConfig):
        super().__init__()
        self.config = config
        self.layer = nn.ModuleList([ViTLayer(config) for _ in range(config.num_hidden_layers)])
        self.gradient_checkpointing = False

    def forward(self, hidden_states: torch.Tensor, head_mask: Optional[torch.Tensor] = None) -> BaseModelOutput:
        for i, layer_module in enumerate(self.layer):
            layer_head_mask = head_mask[i] if head_mask is not None else None
            hidden_states = layer_module(hidden_states, layer_head_mask)

        return BaseModelOutput(last_hidden_state=hidden_states)


@auto_docstring
class ViTPreTrainedModel(PreTrainedModel):
    config: ViTConfig
    base_model_prefix = "vit"
    main_input_name = "pixel_values"
    supports_gradient_checkpointing = True
    _no_split_modules = ["ViTEmbeddings", "ViTLayer"]
    _supports_sdpa = True
    _supports_flash_attn = True
    _supports_flex_attn = True
    _supports_attention_backend = True
    _can_record_outputs = {
        "hidden_states": ViTLayer,
        "attentions": ViTSelfAttention,
    }

    def _init_weights(self, module: Union[nn.Linear, nn.Conv2d, nn.LayerNorm]):
        """Initialize the weights"""
        if isinstance(module, (nn.Linear, nn.Conv2d)):
            # Upcast the input in `fp32` and cast it back to desired `dtype` to avoid
            # `trunc_normal_cpu` not implemented in `half` issues
            module.weight.data = nn.init.trunc_normal_(
                module.weight.data.to(torch.float32), mean=0.0, std=self.config.initializer_range
            ).to(module.weight.dtype)
            if module.bias is not None:
                module.bias.data.zero_()
        elif isinstance(module, nn.LayerNorm):
            module.bias.data.zero_()
            module.weight.data.fill_(1.0)
        elif isinstance(module, ViTEmbeddings):
            module.position_embeddings.data = nn.init.trunc_normal_(
                module.position_embeddings.data.to(torch.float32),
                mean=0.0,
                std=self.config.initializer_range,
            ).to(module.position_embeddings.dtype)

            module.cls_token.data = nn.init.trunc_normal_(
                module.cls_token.data.to(torch.float32),
                mean=0.0,
                std=self.config.initializer_range,
            ).to(module.cls_token.dtype)

            if module.mask_token is not None:
                module.mask_token.data.zero_()


@auto_docstring
class ViTModel(ViTPreTrainedModel):
    def __init__(self, config: ViTConfig, add_pooling_layer: bool = True, use_mask_token: bool = False):
        r"""
        add_pooling_layer (bool, *optional*, defaults to `True`):
            Whether to add a pooling layer
        use_mask_token (`bool`, *optional*, defaults to `False`):
            Whether to use a mask token for masked image modeling.
        """
        super().__init__(config)
        self.config = config

        self.embeddings = ViTEmbeddings(config, use_mask_token=use_mask_token)
        self.encoder = ViTEncoder(config)

        self.layernorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
        self.pooler = ViTPooler(config) if add_pooling_layer else None

        self.mask_padded = False # opt-in flag to mask padding patches

        # Initialize weights and apply final processing
        self.post_init()

    def get_input_embeddings(self) -> ViTPatchEmbeddings:
        return self.embeddings.patch_embeddings

    def _prune_heads(self, heads_to_prune: dict[int, list[int]]):
        """
        Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base
        class PreTrainedModel
        """
        for layer, heads in heads_to_prune.items():
            self.encoder.layer[layer].attention.prune_heads(heads)

    @check_model_inputs
    @auto_docstring
    def forward(
        self,
        pixel_values: Optional[torch.Tensor] = None,
        bool_masked_pos: Optional[torch.BoolTensor] = None,
        head_mask: Optional[torch.Tensor] = None,
        interpolate_pos_encoding: Optional[bool] = None,
        **kwargs: Unpack[TransformersKwargs],
    ) -> BaseModelOutputWithPooling:
        r"""
        bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*):
            Boolean masked positions. Indicates which patches are masked (1) and which aren't (0).
        """

        if pixel_values is None:
            raise ValueError("You have to specify pixel_values")

        # Mirror opt-in flag to embeddings
        self.embeddings.mask_padded = getattr(self, "mask_padded", False)

        # Prepare head mask if needed
        # 1.0 in head_mask indicate we keep the head
        # attention_probs has shape bsz x n_heads x N x N
        # input head_mask has shape [num_heads] or [num_hidden_layers x num_heads]
        # and head_mask is converted to shape [num_hidden_layers x batch x num_heads x seq_length x seq_length]
        head_mask = self.get_head_mask(head_mask, self.config.num_hidden_layers)

        # TODO: maybe have a cleaner way to cast the input (from `ImageProcessor` side?)
        expected_dtype = self.embeddings.patch_embeddings.projection.weight.dtype
        if pixel_values.dtype != expected_dtype:
            pixel_values = pixel_values.to(expected_dtype)

        # ensure our pre-softmax mask codepath is used
        if getattr(self, "mask_padded", False) and getattr(self.config, "_attn_implementation", "eager") != "eager":
            self.config._attn_implementation = "eager"

        # If requested, build a token-pair mask that blocks fully-zero patches
        token_pair_mask = None
        if getattr(self, "mask_padded", False):
            ph, pw = self.embeddings.patch_size if isinstance(self.embeddings.patch_size,
                                                              collections.abc.Iterable) else (
                self.embeddings.patch_size, self.embeddings.patch_size)
            B, C, H, W = pixel_values.shape
            Hpatch, Wpatch = H // ph, W // pw

            patches = nn.functional.unfold(pixel_values, kernel_size=(ph, pw), stride=(ph, pw))  # (B, C*ph*pw, N)
            zero_patch = (patches == 0).all(dim=1)  # (B, N)
            valid_patch = ~zero_patch  # (B, N)

            # pairwise mask over tokens incl. CLS
            valid_with_cls = torch.cat([torch.ones(B, 1, dtype=torch.bool, device=pixel_values.device), valid_patch],
                                       dim=1)  # (B, L)
            pair = (valid_with_cls[:, None, :, None] & valid_with_cls[:, None, None, :])  # (B, 1, L, L)
            token_pair_mask = pair.to(dtype=pixel_values.dtype)

        embedding_output = self.embeddings(
            pixel_values, bool_masked_pos=bool_masked_pos, interpolate_pos_encoding=interpolate_pos_encoding
        )

        # Merge head_mask with token_pair_mask (broadcast to [LAYER, B, H, L, L])
        if token_pair_mask is not None:
            num_layers = self.config.num_hidden_layers
            num_heads = self.config.num_attention_heads
            B, _, L, _ = token_pair_mask.shape
            token_mask_full = token_pair_mask.expand(B, num_heads, L, L).unsqueeze(0).expand(
                num_layers, B, num_heads, L, L
            )

            # Normalize head_mask => tensor with broadcastable shape [LAYER, B, H, L, L]
            hm = head_mask
            if isinstance(hm, (list, tuple)):
                if len(hm) == 0 or all(h is None for h in hm):
                    hm = None
                else:
                    first = next(h for h in hm if h is not None)
                    ones_like = torch.ones_like(first)
                    hm = torch.stack([h if h is not None else ones_like for h in hm], dim=0)  # [LAYER, ...]
            if hm is None:
                head_mask = token_mask_full
            else:
                # Ensure 5D by adding trailing singleton dims, then broadcast
                while hm.dim() < 5:
                    hm = hm.unsqueeze(-1)  # add dims at the end
                if hm.shape[0] != num_layers:
                    # If it's e.g. [H] or [LAYER,H], expand to [LAYER,1,H,1,1]
                    if hm.dim() == 2:  # [LAYER, H]
                        hm = hm[:, None, :, None, None]
                    elif hm.dim() == 1:  # [H]
                        hm = hm[None, None, :, None, None].expand(num_layers, 1, num_heads, 1, 1)
                head_mask = hm.to(token_mask_full.dtype) * token_mask_full

            # Enable pre-softmax masking on all attention blocks
            for lyr in self.encoder.layer:
                lyr.attention.attention._use_pre_softmax_mask = True

        encoder_outputs: BaseModelOutput = self.encoder(embedding_output, head_mask=head_mask)

        sequence_output = encoder_outputs.last_hidden_state
        sequence_output = self.layernorm(sequence_output)

        # Zero-out final hidden states at padded patches (keep CLS)
        if token_pair_mask is not None:
            # recover per-token keep mask from pairwise diagonal
            keep = valid_with_cls  # (B, L) from above scope
            sequence_output = sequence_output * keep[:, :, None].to(sequence_output.dtype)

        pooled_output = self.pooler(sequence_output) if self.pooler is not None else None

        return BaseModelOutputWithPooling(last_hidden_state=sequence_output, pooler_output=pooled_output)


class ViTPooler(nn.Module):
    def __init__(self, config: ViTConfig):
        super().__init__()
        self.dense = nn.Linear(config.hidden_size, config.pooler_output_size)
        self.activation = ACT2FN[config.pooler_act]

    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
        # We "pool" the model by simply taking the hidden state corresponding
        # to the first token.
        first_token_tensor = hidden_states[:, 0]
        pooled_output = self.dense(first_token_tensor)
        pooled_output = self.activation(pooled_output)
        return pooled_output


@auto_docstring(
    custom_intro="""
    ViT Model with a decoder on top for masked image modeling, as proposed in [SimMIM](https://huggingface.co/papers/2111.09886).

    <Tip>

    Note that we provide a script to pre-train this model on custom data in our [examples
    directory](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining).

    </Tip>
    """
)
class ViTForMaskedImageModeling(ViTPreTrainedModel):
    def __init__(self, config: ViTConfig):
        super().__init__(config)

        self.vit = ViTModel(config, add_pooling_layer=False, use_mask_token=True)

        self.decoder = nn.Sequential(
            nn.Conv2d(
                in_channels=config.hidden_size,
                out_channels=config.encoder_stride**2 * config.num_channels,
                kernel_size=1,
            ),
            nn.PixelShuffle(config.encoder_stride),
        )

        # Initialize weights and apply final processing
        self.post_init()

    @can_return_tuple
    @auto_docstring
    def forward(
        self,
        pixel_values: Optional[torch.Tensor] = None,
        bool_masked_pos: Optional[torch.BoolTensor] = None,
        head_mask: Optional[torch.Tensor] = None,
        interpolate_pos_encoding: Optional[bool] = None,
        **kwargs: Unpack[TransformersKwargs],
    ) -> MaskedImageModelingOutput:
        r"""
        bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`):
            Boolean masked positions. Indicates which patches are masked (1) and which aren't (0).

        Examples:
        ```python
        >>> from transformers import AutoImageProcessor, ViTForMaskedImageModeling
        >>> import torch
        >>> from PIL import Image
        >>> import requests

        >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
        >>> image = Image.open(requests.get(url, stream=True).raw)

        >>> image_processor = AutoImageProcessor.from_pretrained("google/vit-base-patch16-224-in21k")
        >>> model = ViTForMaskedImageModeling.from_pretrained("google/vit-base-patch16-224-in21k")

        >>> num_patches = (model.config.image_size // model.config.patch_size) ** 2
        >>> pixel_values = image_processor(images=image, return_tensors="pt").pixel_values
        >>> # create random boolean mask of shape (batch_size, num_patches)
        >>> bool_masked_pos = torch.randint(low=0, high=2, size=(1, num_patches)).bool()

        >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos)
        >>> loss, reconstructed_pixel_values = outputs.loss, outputs.reconstruction
        >>> list(reconstructed_pixel_values.shape)
        [1, 3, 224, 224]
        ```"""

        if bool_masked_pos is not None and (self.config.patch_size != self.config.encoder_stride):
            raise ValueError(
                "When `bool_masked_pos` is provided, `patch_size` must be equal to `encoder_stride` to ensure that "
                "the reconstructed image has the same dimensions as the input. "
                f"Got `patch_size` = {self.config.patch_size} and `encoder_stride` = {self.config.encoder_stride}."
            )

        outputs: BaseModelOutputWithPooling = self.vit(
            pixel_values,
            bool_masked_pos=bool_masked_pos,
            head_mask=head_mask,
            interpolate_pos_encoding=interpolate_pos_encoding,
            **kwargs,
        )

        sequence_output = outputs.last_hidden_state

        # Reshape to (batch_size, num_channels, height, width)
        sequence_output = sequence_output[:, 1:]
        batch_size, sequence_length, num_channels = sequence_output.shape
        height = width = math.floor(sequence_length**0.5)
        sequence_output = sequence_output.permute(0, 2, 1).reshape(batch_size, num_channels, height, width)

        # Reconstruct pixel values
        reconstructed_pixel_values = self.decoder(sequence_output)

        masked_im_loss = None
        if bool_masked_pos is not None:
            size = self.config.image_size // self.config.patch_size
            bool_masked_pos = bool_masked_pos.reshape(-1, size, size)
            mask = (
                bool_masked_pos.repeat_interleave(self.config.patch_size, 1)
                .repeat_interleave(self.config.patch_size, 2)
                .unsqueeze(1)
                .contiguous()
            )
            reconstruction_loss = nn.functional.l1_loss(pixel_values, reconstructed_pixel_values, reduction="none")
            masked_im_loss = (reconstruction_loss * mask).sum() / (mask.sum() + 1e-5) / self.config.num_channels

        return MaskedImageModelingOutput(
            loss=masked_im_loss,
            reconstruction=reconstructed_pixel_values,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
        )


@auto_docstring(
    custom_intro="""
    ViT Model transformer with an image classification head on top (a linear layer on top of the final hidden state of
    the [CLS] token) e.g. for ImageNet.

    <Tip>

        Note that it's possible to fine-tune ViT on higher resolution images than the ones it has been trained on, by
        setting `interpolate_pos_encoding` to `True` in the forward of the model. This will interpolate the pre-trained
        position embeddings to the higher resolution.

    </Tip>
    """
)
class ViTForImageClassification(ViTPreTrainedModel):
    def __init__(self, config: ViTConfig):
        super().__init__(config)

        self.num_labels = config.num_labels
        self.vit = ViTModel(config, add_pooling_layer=False)

        # Classifier head
        self.classifier = nn.Linear(config.hidden_size, config.num_labels) if config.num_labels > 0 else nn.Identity()

        # Initialize weights and apply final processing
        self.post_init()

    @can_return_tuple
    @auto_docstring
    def forward(
        self,
        pixel_values: Optional[torch.Tensor] = None,
        head_mask: Optional[torch.Tensor] = None,
        labels: Optional[torch.Tensor] = None,
        interpolate_pos_encoding: Optional[bool] = None,
        **kwargs: Unpack[TransformersKwargs],
    ) -> ImageClassifierOutput:
        r"""
        labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
            Labels for computing the image classification/regression loss. Indices should be in `[0, ...,
            config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
            `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
        """

        outputs: BaseModelOutputWithPooling = self.vit(
            pixel_values,
            head_mask=head_mask,
            interpolate_pos_encoding=interpolate_pos_encoding,
            **kwargs,
        )

        sequence_output = outputs.last_hidden_state
        pooled_output = sequence_output[:, 0, :]
        logits = self.classifier(pooled_output)

        loss = None
        if labels is not None:
            loss = self.loss_function(labels, logits, self.config, **kwargs)

        return ImageClassifierOutput(
            loss=loss,
            logits=logits,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
        )


__all__ = ["ViTForImageClassification", "ViTForMaskedImageModeling", "ViTModel", "ViTPreTrainedModel"]


if __name__ == "__main__":
    model = ViTModel.from_pretrained("WinKawaks/vit-tiny-patch16-224")
    model.eval()
    torch.manual_seed(0)  # optional, for full determinism

    example_image = torch.randn(3, 64, 64)
    example_image_padded = torch.zeros(3, 128, 128)
    example_image_padded[:, :64, :64] = example_image

    example_image = example_image.unsqueeze(0)
    example_image_padded = example_image_padded.unsqueeze(0)
    
    args = {
        "interpolate_pos_encoding": True,
        "output_hidden_states": True,
    }
    
    # Observe the first patch
    print("Without patching should be different:")
    print("no pad", model(example_image, **args).pooler_output[0, :5])
    print("yes pad", model(example_image_padded, **args).pooler_output[0, :5])

    model.mask_padded = True

    print("With patching should be the same:")
    print("no pad", model(example_image, **args).pooler_output[0, :5])
    print("yes pad", model(example_image_padded, **args).pooler_output[0, :5])

And code to patch the model instance similarly

import collections.abc

# add imports near the top of your patch file
import types

import torch
import transformers.models.vit.modeling_vit as hvit
from torch import nn
from transformers import AutoModelForImageClassification
from transformers.models.vit.modeling_vit import ViTEmbeddings, ViTModel

from image_latent_transformer.collator import stack_pad_tensors
from image_latent_transformer.vision_utils import model_args_dict


def patch_vit_zero_pad_mask(model: ViTModel, *, pad_value: float = 0.0, tol: float = 0.0) -> None:
    """
    Make ViT invariant to zero-padding and zero out final hidden states at padded patches.

    Guarantees:
      1) Padded patches are excluded from attention via pre-softmax logits masking.
      2) Positional encodings for content patches are computed on the tight (content-only) grid,
         then scattered into the big sequence -> shared patches get identical pos-enc.
      3) At the end of the forward pass, hidden vectors at padded patches are set to 0 for:
           - outputs.last_hidden_state
           - outputs.hidden_states[-1] (if output_hidden_states=True)

    Assumptions:
      - Padding is zeros (tune `pad_value`/`tol` otherwise).
      - Content is a single top-left rectangle after padding (common case). If not, tell me and
        I’ll generalize the scatter to an arbitrary mask.
    """
    if getattr(model, "_zero_pad_mask_patched", False):
        return

    # -------- helpers --------
    def _patch_size_tuple(vit_model: ViTModel) -> tuple[int, int]:
        ps = vit_model.embeddings.patch_size
        if isinstance(ps, collections.abc.Iterable):
            ph, pw = int(ps[0]), int(ps[1])
        else:
            ph = pw = int(ps)
        return ph, pw

    def _valid_patch_grid(vit_model: ViTModel, pixel_values: torch.Tensor) -> tuple[torch.BoolTensor, int, int]:
        """
        Returns:
          valid_grid: (B, Hpatch, Wpatch) bool where True means "non-zero patch"
          Hpatch, Wpatch: patch grid size
        """
        B, C, H, W = pixel_values.shape
        ph, pw = _patch_size_tuple(vit_model)
        Hpatch, Wpatch = H // ph, W // pw
        patches = nn.functional.unfold(pixel_values, kernel_size=(ph, pw), stride=(ph, pw))  # (B, C*ph*pw, N)
        if tol > 0.0:
            zero_patch = (patches - pad_value).abs().le(tol).all(dim=1)  # (B, N)
        else:
            zero_patch = (patches == pad_value).all(dim=1)  # (B, N)
        valid_patch = ~zero_patch  # (B, N)
        return valid_patch.view(B, Hpatch, Wpatch), Hpatch, Wpatch

    def _compute_pairwise_token_mask(vit_model: ViTModel, pixel_values: torch.Tensor) -> torch.Tensor:
        """
        (B,1,L,L) float mask: 1.0 allowed, 0.0 blocked. CLS always valid.
        """
        valid_grid, Hpatch, Wpatch = _valid_patch_grid(vit_model, pixel_values)
        B = valid_grid.size(0)
        valid_flat = valid_grid.view(B, -1)  # (B, N)
        cls = torch.ones(B, 1, dtype=torch.bool, device=valid_flat.device)
        valid_with_cls = torch.cat([cls, valid_flat], dim=1)  # (B, L)
        pair = (valid_with_cls[:, None, :, None] & valid_with_cls[:, None, None, :])  # (B,1,L,L)
        return pair.to(dtype=pixel_values.dtype)

    # -------- patch eager attention to support pre-softmax masking --------
    model.config._attn_implementation = "eager"
    if not hasattr(hvit, "_orig_eager_attention_forward"):
        hvit._orig_eager_attention_forward = hvit.eager_attention_forward

        def _eager_attention_forward_premask(
                module: nn.Module,
                query: torch.Tensor,
                key: torch.Tensor,
                value: torch.Tensor,
                attention_mask: torch.Tensor | None,
                scaling: float,
                dropout: float = 0.0,
                **kwargs,
        ):
            logits = torch.matmul(query, key.transpose(-1, -2)) * scaling  # (B,H,L,L)
            if getattr(module, "_use_pre_softmax_mask", False) and attention_mask is not None:
                mask_bool = attention_mask.bool() if attention_mask.dtype != torch.bool else attention_mask
                logits = logits.masked_fill(~mask_bool, torch.finfo(logits.dtype).min)
            attn = torch.softmax(logits, dim=-1, dtype=torch.float32).to(query.dtype)
            if not getattr(module, "_use_pre_softmax_mask", False) and attention_mask is not None:
                attn = attn * attention_mask
            attn = nn.functional.dropout(attn, p=dropout, training=module.training)
            out = torch.matmul(attn, value).transpose(1, 2).contiguous()
            return out, attn

        hvit.eager_attention_forward = _eager_attention_forward_premask

    # -------- override ViTEmbeddings.forward to inject invariant positional encodings --------
    orig_embed_forward = model.embeddings.forward

    def embeddings_forward_with_invariant_pos(
            self: ViTEmbeddings,
            pixel_values: torch.Tensor,
            bool_masked_pos: torch.BoolTensor | None = None,
            interpolate_pos_encoding: bool | None = None,
    ) -> torch.Tensor:
        B, C, H, W = pixel_values.shape
        ph, pw = _patch_size_tuple(model)

        # Always allow arbitrary size; we'll handle positions ourselves
        patch_tokens = self.patch_embeddings(pixel_values, interpolate_pos_encoding=True)  # (B, N, D)
        N, D = patch_tokens.shape[1], patch_tokens.shape[2]

        # SimMIM mask token (remains valid)
        if bool_masked_pos is not None:
            mask_tokens = self.mask_token.expand(B, N, -1)
            mask = bool_masked_pos.unsqueeze(-1).type_as(mask_tokens)
            patch_tokens = patch_tokens * (1.0 - mask) + mask_tokens * mask

        # CLS
        cls_tokens = self.cls_token.expand(B, -1, -1)  # (B,1,D)

        # Tight content block
        valid_grid, Hpatch, Wpatch = _valid_patch_grid(model, pixel_values)  # (B,Hpatch,Wpatch)
        row_has = valid_grid.any(dim=2)  # (B,Hpatch)
        col_has = valid_grid.any(dim=1)  # (B,Wpatch)

        def last_true_id(x: torch.Tensor) -> int:
            if not x.any():
                return 0
            idxs = torch.arange(x.numel(), device=x.device)
            return int((x.long() * idxs).max().item())

        hs, ws = [], []
        for b in range(B):
            hs.append(last_true_id(row_has[b]) + 1)
            ws.append(last_true_id(col_has[b]) + 1)
        h_small_max, w_small_max = max(hs), max(ws)

        # Ask HF for pos-enc at that small grid
        dummy = torch.zeros(1, 1 + h_small_max * w_small_max, D, device=pixel_values.device, dtype=patch_tokens.dtype)
        pos_small_max = self.interpolate_pos_encoding(dummy, ph * h_small_max, pw * w_small_max)  # (1,1+..,D)
        cls_pos = pos_small_max[:, :1, :]  # (1,1,D)
        patch_pos_small = pos_small_max[:, 1:, :].reshape(1, h_small_max, w_small_max, D)  # (1,hs,ws,D)

        # Scatter small-grid positions into big grid (top-left block) per sample
        pos_big = torch.zeros(B, Hpatch, Wpatch, D, device=pixel_values.device, dtype=patch_tokens.dtype)
        for b in range(B):
            pos_big[b, :hs[b], :ws[b], :] = patch_pos_small[0, :hs[b], :ws[b], :]
        pos_big = pos_big.view(B, Hpatch * Wpatch, D)  # (B,N,D)

        # Add positions
        tokens = torch.cat([cls_tokens, patch_tokens], dim=1)  # (B,1+N,D)
        pos_cat = torch.cat([cls_pos.expand(B, -1, -1), pos_big], dim=1)  # (B,1+N,D)
        tokens = tokens + pos_cat
        tokens = self.dropout(tokens)
        return tokens

    model.embeddings.forward = types.MethodType(embeddings_forward_with_invariant_pos, model.embeddings)

    # -------- wrap encoder.forward to merge masks + enable pre-softmax masking --------
    orig_enc_forward = model.encoder.forward

    def enc_forward_with_zero_pad_mask(self, hidden_states, head_mask=None, *args, **kwargs):
        token_mask = getattr(model, "_zero_pad_token_mask", None)  # (B,1,L,L) or None

        # Normalize head_mask => tensor or None
        if isinstance(head_mask, (list, tuple)):
            if len(head_mask) == 0 or all(h is None for h in head_mask):
                head_mask = None
            else:
                first = next(h for h in head_mask if h is not None)
                ones_like = torch.ones_like(first)
                head_mask = torch.stack([h if h is not None else ones_like for h in head_mask], dim=0)

        if token_mask is not None:
            num_layers = len(self.layer)
            B, _, L, _ = token_mask.shape
            H = model.config.num_attention_heads
            token_mask_full = token_mask.expand(B, H, L, L).unsqueeze(0).expand(num_layers, B, H, L, L)
            head_mask = token_mask_full if head_mask is None else (
                        head_mask.to(token_mask_full.dtype) * token_mask_full)

        # Enable pre-softmax masking
        for lyr in self.layer:
            lyr.attention.attention._use_pre_softmax_mask = True

        return orig_enc_forward(hidden_states, head_mask=head_mask, *args, **kwargs)

    model.encoder.forward = types.MethodType(enc_forward_with_zero_pad_mask, model.encoder)

    # -------- wrap model.forward: compute mask, call orig, then zero-out padded outputs --------
    orig_model_forward = model.forward

    def forward_with_zero_pad_mask(self: ViTModel, *args, **kwargs):
        pixel_values = kwargs.get("pixel_values", None)
        if pixel_values is None and len(args) > 0:
            pixel_values = args[0]

        # Build masks for this call
        valid_grid, Hpatch, Wpatch = _valid_patch_grid(self, pixel_values) if pixel_values is not None else (None, None,
                                                                                                             None)
        self._zero_pad_token_mask = _compute_pairwise_token_mask(self,
                                                                 pixel_values) if pixel_values is not None else None

        # Run original forward
        outputs = orig_model_forward(*args, **kwargs)

        # Post-mask: zero-out padded tokens at the end
        if valid_grid is not None:
            B = valid_grid.size(0)
            valid_flat = valid_grid.view(B, -1)  # (B,N)
            keep = torch.cat(
                [torch.ones(B, 1, dtype=torch.bool, device=valid_flat.device), valid_flat], dim=1
            )  # (B,L) with CLS=True

            keep_f = keep[:, :, None].to(outputs.last_hidden_state.dtype)  # (B,L,1)

            # last_hidden_state
            if hasattr(outputs, "last_hidden_state") and outputs.last_hidden_state is not None:
                outputs.last_hidden_state = outputs.last_hidden_state * keep_f

            # hidden_states[-1]
            if hasattr(outputs, "hidden_states") and outputs.hidden_states is not None and len(
                    outputs.hidden_states) > 0:
                hs = list(outputs.hidden_states)
                hs[-1] = hs[-1] * keep_f
                outputs.hidden_states = tuple(hs)

        # cleanup
        if hasattr(self, "_zero_pad_token_mask"):
            delattr(self, "_zero_pad_token_mask")

        return outputs

    model.forward = types.MethodType(forward_with_zero_pad_mask, model)
    model._zero_pad_mask_patched = True


def call_vit(model, image):
    model_args = model_args_dict(model)

    # Run the forward pass through the image encoder
    encoded_image = model(image.unsqueeze(0), **model_args)

    if hasattr(encoded_image, "pooler_output"):
        return encoded_image.pooler_output

    # Extract the final layer's hidden states (shape varies by model architecture)
    return encoded_image.hidden_states[-1]


if __name__ == "__main__":
    model = AutoModelForImageClassification.from_pretrained("WinKawaks/vit-tiny-patch16-224")
    model.eval()
    torch.manual_seed(0)  # optional, for full determinism

    example_image = torch.randn(3, 64, 64)
    large_zeros = torch.zeros(3, 128, 128)
    example_image_padded = stack_pad_tensors([example_image, large_zeros])[0]

    # Observe the first patch
    print("Without patching should be different:")
    print("no pad", call_vit(model, example_image)[0, 0, :5])
    print("yes pad", call_vit(model, example_image_padded)[0, 0, :5])

    if hasattr(model, "vit"):
        patch_vit_zero_pad_mask(model.vit)
    else:
        patch_vit_zero_pad_mask(model)

    print("With patching should be the same:")
    print("no pad", call_vit(model, example_image)[0, 0, :5])
    res = call_vit(model, example_image_padded)
    print("yes pad", res[0, 0, :5])

    for patch in range(len(res[0])):
        print(patch, res[0, patch, :5])

But actually, it seems like we could just use NaVIT https://github.com/lucidrains/vit-pytorch#navit which is already implemented that way.

We would probably still pass padded tensors across different processes, but after unpacking them we can send them to NaVIT directly

images = [
    torch.randn(3, 256, 256),
    torch.randn(3, 128, 128),
    torch.randn(3, 128, 256),
    torch.randn(3, 256, 128),
    torch.randn(3, 64, 256)
]

preds = v(
    images,
    group_images = True,
    group_max_seq_len = 64
) # (5, 1000)

[AmitMY]

We now patch ViT models: 6469e5f

I'd be happier if we also supported NaViT for the simplest and probably fastest approach.

[AmitMY]

We now have NaViT support, but it is also too slow.
Benchmark:
https://github.com/sign/image-latent-transformer/blob/main/examples/benchmark_image_encoder.py
Results (see lucidrains/vit-pytorch#347):

1. ViT Model with padding (inconsistent results) - 1 second
2. Vit Model grouped into batches of same size images - 13 seconds
3. Vit Model patched (consistent results) - 34 second
4. NaVit Model - 39 seconds

Is the best solution to run with padding, but then "fine tune" using grouping?