"ValueError: Expected 1 columns but got 8 columns." when quantizing Llama3.2-1B

[SunXT-0719]

Thank you for providing such a great repo, but i can't run quantization for other model, even it's llama.
The command is:

python run_vptq.py \
        --model_name meta-llama/Llama-3.2-1B \
        --output_dir /data/sunxuetin/Llama-3.2-1B-VPTQ \
        --vector_lens -1 8 \
        --group_num 1 \
        --num_centroids -1 65536 \
        --num_res_centroids -1 256 \
        --blocksize 128 \
        --new_eval \
        --seq_len 8192 \
        --kmeans_mode hessian \
        --num_gpus 1 \
        --enable_perm \
        --enable_norm \
        --save_model \
        --save_packed_model \
        --hessian_path /data/sunxuetin/hessian/llama3.2-1B \
        --inv_hessian_path /data/sunxuetin/hessian_inv/llama3.2-1B \
        --ktol 1e-5 --kiter 100

I got hessian via https://github.com/VPTQ/hessian_collector/blob/master/hessian_collector.py as you say in other issues,
The complete error message is:

args: VPTQArguments(model_name='meta-llama/Llama-3.2-1B', seq_len=8192, quant_step=1, percdamp=0.01, blocksize=128, output_dir='/data/sunxuetin/Llama-3.2-1B-VPTQ/2025-10-21-10-09-01', seed=42, eval=False, new_eval=True, save_model=True, save_packed_model=True, disable_actorder=False, hessian_path='/data/sunxuetin/hessian/llama3.2-1B', inv_hessian_path='/data/sunxuetin/hessian_inv/llama3.2-1B', num_gpus=1, eval_nsamples=128, save_qlinear=True, absorb_perm=False)
quant_args: QuantizationArguments(vector_lens=[-1, 8], num_centroids=[-1, 65536], num_res_centroids=[-1, 256], npercent=0, group_num=1, group_size=-1, kiter=100, ktol=1e-05, kseed=0, kmeans_mode='hessian', kmeans_alpha=0, enable_norm=True, norm_dim=0, enable_perm=True)
Starting VPTQ...
model dtype: torch.bfloat16
----quantization start ...---- 2025-10-21 10:09:04
gpu 0 tasks: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
INFO - ----Quantizing on cuda:0----
INFO - ----Quantizing layer 0 ...---- 2025-10-21 10:09:09 on cuda:0 dtype torch.bfloat16
INFO - dict_keys(['self_attn.q_proj', 'self_attn.k_proj', 'self_attn.v_proj', 'self_attn.o_proj', 'mlp.gate_proj', 'mlp.up_proj', 'mlp.down_proj'])
INFO - load Hessian from /data/sunxuetin/hessian/llama3.2-1B/0_qkv.pt
INFO - load inv Hessian from /data/sunxuetin/hessian_inv/llama3.2-1B/0_qkv.pt
INFO - ----Quantizing llama ...---- 2025-10-21 10:09:09 0.self_attn.q_proj
INFO - enabling norm dim 0, layer_name:0.self_attn.q_proj, scale:torch.Size([2048]), bias:torch.Size([2048])
INFO - kmeans_mode: hessian, enable_perm: True, enable_norm: True,
INFO - data shape: torch.Size([2048, 2048]), weights shape: torch.Size([2048, 2048])
INFO - group_size: 2048 number of groups: 1
INFO - idx: 0, num_centroids: -1, skip
INFO - cuml kmeans 23 iterations, error 1998.8406982421875
INFO - idx: 1, quant_data shape: torch.Size([524288, 8])
INFO - idx: 1, quant_data shape: torch.Size([2048, 2048])
INFO - quantized_data shape: torch.Size([2048, 2048])
INFO - 0.self_attn.q_proj 1st kmeans time: 2.732691526412964
INFO - 0.self_attn.q_proj qweight init shape: torch.Size([2048, 2048]), weight shape: torch.Size([2048, 2048])
INFO - 0.self_attn.q_proj proxy error before VPTQ: 0.0001238604891113937, 0.03463037312030792, 0.0035766432993113995
INFO - 0.self_attn.q_proj 1st error time: 1.029219150543213
INFO - 0.self_attn.q_proj proxy error after VPTQ: 6.773966742912307e-05, 0.03463037312030792, 0.0019560768269002438
INFO - group_size: 2048 number of groups: 1
INFO - idx: 0, num_centroids: -1, skip
INFO - kmeans_mode: hessian, cuml kmeans, 256 clusters
Traceback (most recent call last):
File "/data/sunxuetin/projects/VPTQ/run_vptq.py", line 86, in
model, quantizers = quant_llama(model, args, quant_args)
File "/data/sunxuetin/projects/VPTQ/vptq/models/llama.py", line 122, in quant_llama
layer_state_dicts, layer_qlinear_args = quantize_executer(0, tasks[0], args, quant_args, None, None)
File "/data/sunxuetin/projects/VPTQ/vptq/quantize_executer.py", line 69, in quantize_executer
layer, qlinear_args = layer_quantizer(
File "/data/sunxuetin/projects/VPTQ/vptq/layer_quantizer.py", line 105, in layer_quantizer
_vptq.fast_vector_quant()
File "/data/sunxuetin/projects/VPTQ/vptq/vptq.py", line 261, in fast_vector_quant
qweight_residual = self.quantizer.init_res_centroids_indices(qerror, kmeans_weight)
File "/data/sunxuetin/projects/VPTQ/vptq/quantizer.py", line 403, in init_res_centroids_indices
_kmeans.fit(sub_vectors, sample_weight=vector_weights)
File "/data/sunxuetin/anaconda3/envs/vptq-algo/lib/python3.10/site-packages/cuml/internals/api_decorators.py", line 211, in wrapper
ret = func(*args, **kwargs)
File "cuml/cluster/kmeans.pyx", line 520, in cuml.cluster.kmeans.KMeans.fit
File "/data/sunxuetin/anaconda3/envs/vptq-algo/lib/python3.10/site-packages/cuml/internals/api_decorators.py", line 211, in wrapper
ret = func(*args, **kwargs)
File "cuml/cluster/kmeans.pyx", line 535, in cuml.cluster.kmeans.KMeans._fit
File "/data/sunxuetin/anaconda3/envs/vptq-algo/lib/python3.10/site-packages/cuml/internals/input_utils.py", line 358, in input_to_cuml_array
arr = CumlArray.from_input(
File "/data/sunxuetin/anaconda3/envs/vptq-algo/lib/python3.10/site-packages/cuml/internals/memory_utils.py", line 74, in cupy_rmm_wrapper
return func(*args, **kwargs)
File "/data/sunxuetin/anaconda3/envs/vptq-algo/lib/python3.10/site-packages/cuml/internals/array.py", line 1216, in from_input
raise ValueError(
ValueError: Expected 1 columns but got 8 columns.

I've printed the shape of sub_vectors, it's (524288, 8).
Have anyone meets the same problem? Thanks a lot if you can give me a solution!

[YangWang92]

Hi @SunXT-0719 , I haven't tested the llama3.2 1b model yet. Previously, we had an example with the llama 3 8b model, which you can try out. There might be slight differences with the smaller 1b model. You may need to update some configurations for the 1B model.

[SunXT-0719]

Hi @SunXT-0719 , I haven't tested the llama3.2 1b model yet. Previously, we had an example with the llama 3 8b model, which you can try out. There might be slight differences with the smaller 1b model. You may need to update some configurations for the 1B model.

Thank you for reply, now I have finished quantizing the llama 1b model, while there is a new problem. When I try to run the model with the commond python -m vptq --prompt="Explain: Do Not Go Gentle into That Good Night" --model=/data/sunxuetin/Llama-3.2-1B-VPTQ/2025-10-21-15-24-13/packed_model . there occurs an error:

/data/sunxuetin/anaconda3/envs/transformers/lib/python3.13/site-packages/torch/cuda/init.py:61: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
Successfully loaded VPTQ CUDA kernels.
Namespace(model='/data/sunxuetin/Llama-3.2-1B-VPTQ/2025-10-21-15-24-13/packed_model', tokenizer='', prompt='Explain: Do Not Go Gentle into That Good Night', chat=False, chat_system_prompt='you are a math teacher.')
Replacing linear layers...: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 215/215 [00:00<00:00, 4014.71it/s]
2025-10-24 13:32:32,159 - accelerate.utils.modeling - WARNING - The model weights are not tied. Please use the tie_weights method before using the infer_auto_device function.
2025-10-24 13:32:32,161 - accelerate.utils.modeling - WARNING - The model weights are not tied. Please use the tie_weights method before using the infer_auto_device function.
OrderedDict({'': 0})
device: 0
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/data/sunxuetin/anaconda3/envs/transformers/lib/python3.13/site-packages/vptq/main.py", line 8, in
main()
~~~~^^
File "/data/sunxuetin/anaconda3/envs/transformers/lib/python3.13/site-packages/vptq/app_utils.py", line 182, in main
model = VQAutoModelQuantization.from_pretrained(
args.model, device_map="auto", **hf_args
)
File "/data/sunxuetin/anaconda3/envs/transformers/lib/python3.13/site-packages/vptq/layers/model_base.py", line 185, in from_pretrained
model = accelerate.load_checkpoint_and_dispatch(
model,
...<5 lines>...
preload_module_classes=["VQuantLinear"],
)
File "/data/sunxuetin/anaconda3/envs/transformers/lib/python3.13/site-packages/accelerate/big_modeling.py", line 632, in load_checkpoint_and_dispatch
return dispatch_model(
model,
...<5 lines>...
force_hooks=force_hooks,
)
File "/data/sunxuetin/anaconda3/envs/transformers/lib/python3.13/site-packages/accelerate/big_modeling.py", line 500, in dispatch_model
model.to(device)
~~~~~~~~^^^^^^^^
File "/data/sunxuetin/projects/transformers_sketch/src/transformers/modeling_utils.py", line 3160, in to
return super().to(*args, **kwargs)
~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/data/sunxuetin/anaconda3/envs/transformers/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1343, in to
return self._apply(convert)
~~~~~~~~~~~^^^^^^^^^
File "/data/sunxuetin/anaconda3/envs/transformers/lib/python3.13/site-packages/torch/nn/modules/module.py", line 903, in _apply
module._apply(fn)
~~~~~~~~~~~~~^^^^
File "/data/sunxuetin/anaconda3/envs/transformers/lib/python3.13/site-packages/torch/nn/modules/module.py", line 930, in _apply
param_applied = fn(param)
File "/data/sunxuetin/anaconda3/envs/transformers/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1336, in convert
raise NotImplementedError(
...<2 lines>...
) from None
NotImplementedError: Cannot copy out of meta tensor; no data! Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() when moving module from meta to a different device.

Is that mean the model isn't quantized successfully? Or do you have any idea about this problem?

[LiangPeng03]

I also encountered the same problem: "ValueError: Expected 1 columns but got 8 columns.".
I tried using this method: " _kmeans.fit(sub_vectors, sample_weight=vector_weights### .mean(dim=1))",
but the result was not satisfactory. The quantified effect was poor.
Could you please tell me how you solved it?

[SunXT-0719]

I also encountered the same problem: "ValueError: Expected 1 columns but got 8 columns.". I tried using this method: " _kmeans.fit(sub_vectors, sample_weight=vector_weights### .mean(dim=1))", but the result was not satisfactory. The quantified effect was poor. Could you please tell me how you solved it?

I used thetransformers.AutoModelForCausalLM.from_pretrainedto directly load the model, then it works on my testing script, I'm not sure if it can work on any other testing tasks.
Hope this tip helps you.

[LiangPeng03]

I also encountered the same problem: "ValueError: Expected 1 columns but got 8 columns.". I tried using this method: " _kmeans.fit(sub_vectors, sample_weight=vector_weights### .mean(dim=1))", but the result was not satisfactory. The quantified effect was poor. Could you please tell me how you solved it?

I used thetransformers.AutoModelForCausalLM.from_pretrainedto directly load the model, then it works on my testing script, I'm not sure if it can work on any other testing tasks. Hope this tip helps you.

Thank you for your response. May I ask if you obtained reasonable quantitative results through the method you described? And approximately how much was the difference in PPL before and after the quantification?

[SunXT-0719]

Oh, I'm sorry I misread the problem, for the method you mentioned, I just deleted the code which raise the error because the error doesn't seem to have any impact. after quantization I loaded the model by the method I mentioned to test it on some downstream tasks, and it has pretty well results.

[LiangPeng03]

OK. Thank you for your sharing.