RuntimeError: The size of tensor a (32) must match the size of tensor b (64) at non-singleton dimension 3

[kanishkau93]

!!! Exception during processing !!! The size of tensor a (32) must match the size of tensor b (64) at non-singleton dimension 3
Traceback (most recent call last):
File "D:\ComfyUI_windows_portable\ComfyUI\execution.py", line 496, in execute
output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\ComfyUI\execution.py", line 315, in get_output_data
return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\ComfyUI\execution.py", line 289, in _async_map_node_over_list
await process_inputs(input_dict, i)
File "D:\ComfyUI_windows_portable\ComfyUI\execution.py", line 277, in process_inputs
result = f(**inputs)
^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_DiffRhythm\DiffRhythmNode.py", line 271, in diffrhythmgen
generated_songs = inference(
^^^^^^^^^^
File "D:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_DiffRhythm\DiffRhythmNode.py", line 92, in inference
latents, _ = cfm_model.sample(
^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\utils_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_DiffRhythm\diffrhythm\model\cfm.py", line 249, in sample
trajectory = odeint(fn, y0, t, **self.odeint_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchdiffeq_impl\odeint.py", line 80, in odeint
solution = solver.integrate(t)
^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchdiffeq_impl\solvers.py", line 114, in integrate
dy, f0 = self._step_func(self.func, t0, dt, t1, y0)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchdiffeq_impl\fixed_grid.py", line 10, in _step_func
f0 = func(t0, y0, perturb=Perturb.NEXT if self.perturb else Perturb.NONE)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchdiffeq_impl\misc.py", line 197, in forward
return self.base_func(t, y)
^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_DiffRhythm\diffrhythm\model\cfm.py", line 214, in fn
pred = self.transformer(
^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1762, in call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_DiffRhythm\diffrhythm\model\dit.py", line 211, in forward
x, * = block(x, attention_mask=attention_mask, position_embeddings=rotary_embed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\modeling_layers.py", line 94, in call
return super().call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\models\llama\modeling_llama.py", line 289, in forward
hidden_states, _ = self.self_attn(
^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\models\llama\modeling_llama.py", line 237, in forward
query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\transformers\models\llama\modeling_llama.py", line 135, in apply_rotary_pos_emb
q_embed = (q * cos) + (rotate_half(q) * sin)
~~^~~~~
RuntimeError: The size of tensor a (32) must match the size of tensor b (64) at non-singleton dimension 3

[tomtvgraphic-stack]

The same thing occurred with me

[oumad]

same issue here

[kanishkau93]

couldn't find a fix yet

[oumad]

nothing yet, could this be related to some comfyui update? maybe we need to try with a much older version?

[Maelstrom2014]

same problem

[troed]

NOT A SOLUTION! Just more information:

The error is caused by dit.py line 213, while enumerating the transformer blocks. Every time that loop executes, blocks 9-15 only have two dimensions instead of three, which is what throws the error down in modeling_llama.py

Avoiding the call to block() for blocks that don't have three dimensions causes the code to run to completion, but the resulting audio sounds like a modem from the 80s. The actual fix will be to figure out why only half the blocks have been loaded with correct (?) data.

I don't know enough myself about torch to immediately spot why, but I wanted to post this in the meantime while I continue looking at it.

The blocks are loaded here:

        self.transformer_blocks = nn.ModuleList(
            [LlamaDecoderLayer(llama_config, layer_idx=i) for i in range(depth)]
        )

... and it does fit somewhat with depth being 16 in config.json, I guess.

[troed]

Workaround that seems to produce correct(?) results with v1.2 of the model:

In dit.py, change:

        for i, block in enumerate(self.transformer_blocks):
            x, *_ = block(x, attention_mask=attention_mask, position_embeddings=rotary_embed)
            if i < self.depth // 2:
                x = x + self.text_fusion_linears[i](text_embed)

into:

        for i, block in enumerate(self.transformer_blocks):
            if x.dim() <= 2:
                x.unsqueeze_(0)
            x, *_ = block(x, attention_mask=attention_mask, position_embeddings=rotary_embed)
            if i < self.depth // 2:
                x = x + self.text_fusion_linears[i](text_embed)

A real fix would be to figure out why this needs to be done, of course.

[troed]

A different workaround which will also allow to use quality instead of speed is to remove the // 2 for the text_fusion_linears. Now I'm not an AI developer, just a regular developer, and cannot say why they're limited to half the layers originally. Maybe this workaround increases the memory requirements - I don't know.

For this, change the following (still in dit.py) instead:

               nn.Sequential(
                   nn.Linear(cond_dim, dim),
                   nn.SiLU()
               ) for i in range(depth // 2)

to

                nn.Sequential(
                    nn.Linear(cond_dim, dim),
                    nn.SiLU()
                ) for i in range(depth)

and:

            x, *_ = block(x, attention_mask=attention_mask, position_embeddings=rotary_embed)
            if i < self.depth // 2:
                x = x + self.text_fusion_linears[i](text_embed)

to

            x, *_ = block(x, attention_mask=attention_mask, position_embeddings=rotary_embed)
            x = x + self.text_fusion_linears[i](text_embed)

I'm starting to think the changed behavior somewhere is an automatic broadcast between tensors of different sizes that's not longer automatic. If so, a manual broadcast of x to still match pos_ids even without adding the text_fusion_linears would allow the previous behavior of only adding them for half the layers without throwing an error. But, again, it's better if someone who understands this type of development better than I do steps in here.

[lishuai1993]

The same thing occurred with me，I solved the problem by installing transformers of the version specified in requirement.txt. At first, transformers==4.57 was already installed in the local environment. Then, I reinstalled version 4.49.0 in requirement.txt, and it worked fine.

[spark752]

Testing with the "text-music-gen.json" sample workflow and cfm_model_v1_2.pt, the issue first occurs with transformers 4.54 (released in July). The error does not occur with the previous version, transformers 4.53.3. Of course this is not claiming that it is a bug in transformers, but looking at what changed in 4.54 might be helpful in solving the issue.

[mkat2]

Workaround that seems to produce correct(?) results with v1.2 of the model:

In dit.py, change:

        for i, block in enumerate(self.transformer_blocks):
            x, *_ = block(x, attention_mask=attention_mask, position_embeddings=rotary_embed)
            if i < self.depth // 2:
                x = x + self.text_fusion_linears[i](text_embed)

into:

        for i, block in enumerate(self.transformer_blocks):
            if x.dim() <= 2:
                x.unsqueeze_(0)
            x, *_ = block(x, attention_mask=attention_mask, position_embeddings=rotary_embed)
            if i < self.depth // 2:
                x = x + self.text_fusion_linears[i](text_embed)

A real fix would be to figure out why this needs to be done, of course.

This does create a audio file, but the output is rubbish, no speak and just same toneish sound, no music.... do you have a good output??

[troed]

This does create a audio file, but the output is rubbish, no speak and just same toneish sound, no music.... do you have a good output??

When I wrote this comment it worked perfectly fine. I haven't looked at this recently if there's now been other changes.

[mkat2]

strange, i have transformers 4.57.1, but your fix just outputs rubbish sound.... any change in transformers again? i cannot get it working now.