ValueError: Schema and number of arrays unequal during training

[hulianyuyy]

Many thanks for your contributions for origanizing such a wonderful work! When running the training script, i encounter the following issue. I want to inquire if you have met the same issue? Thanks.

[rank0]: multiprocess.pool.RemoteTraceback:                                                                                                                                  
[rank0]: """                                                                                                                                                                 
[rank0]: Traceback (most recent call last):                                                                                                                                  
[rank0]:   File "/root/anaconda3/envs/llamafactory/lib/python3.11/site-packages/multiprocess/pool.py", line 125, in worker                                                   
[rank0]:     result = (True, func(*args, **kwds))                                                                                                                            
[rank0]:                     ^^^^^^^^^^^^^^^^^^^                                                                                                                             
[rank0]:   File "/root/anaconda3/envs/llamafactory/lib/python3.11/site-packages/datasets/utils/py_utils.py", line 586, in _write_generator_to_queue                          
[rank0]:     for i, result in enumerate(func(**kwargs)):                                                                                                                     
[rank0]:   File "/root/anaconda3/envs/llamafactory/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 3689, in _map_single                                        
[rank0]:     writer.write_batch(batch, try_original_type=try_original_type)                                                                                                  
[rank0]:   File "/root/anaconda3/envs/llamafactory/lib/python3.11/site-packages/datasets/arrow_writer.py", line 630, in write_batch                                          
[rank0]:     pa_table = pa.Table.from_arrays(arrays, schema=schema)                                                                                                          
[rank0]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                          
[rank0]:   File "pyarrow/table.pxi", line 4851, in pyarrow.lib.Table.from_arrays                                                                                             
[rank0]:   File "pyarrow/table.pxi", line 1603, in pyarrow.lib._sanitize_arrays                                                                                              
[rank0]: ValueError: Schema and number of arrays unequal                                                                                                                     
[rank0]: """                                                                                                                                                                 
                                                      
[rank0]: Traceback (most recent call last):                                                                                                                                  
[rank0]:   File "/mnt/tidal-alsh01/usr/dawo/qinshengqian/c1/OneThinker/LLaMA-Factory/src/llamafactory/data/loader.py", line 316, in _get_preprocessed_dataset                
[rank0]:     dataset = dataset.map(                                                                                                                                          
[rank0]:               ^^^^^^^^^^^^                                                                                                                                          
[rank0]:   File "/root/anaconda3/envs/llamafactory/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 560, in wrapper                                             
[rank0]:     out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)                                                                                              
[rank0]:                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                              
[rank0]:   File "/root/anaconda3/envs/llamafactory/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 3309, in map                                                
[rank0]:     for rank, done, content in iflatmap_unordered(                                                                                                                  
[rank0]:   File "/root/anaconda3/envs/llamafactory/lib/python3.11/site-packages/datasets/utils/py_utils.py", line 626, in iflatmap_unordered                                 
[rank0]:     [async_result.get(timeout=0.05) for async_result in async_results]                                                                                              
[rank0]:   File "/root/anaconda3/envs/llamafactory/lib/python3.11/site-packages/datasets/utils/py_utils.py", line 626, in <listcomp>                                         
[rank0]:     [async_result.get(timeout=0.05) for async_result in async_results]                                                                                              
[rank0]:      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                 
[rank0]:   File "/root/anaconda3/envs/llamafactory/lib/python3.11/site-packages/multiprocess/pool.py", line 774, in get                                                      
[rank0]:     raise self._value                                                                                                                                               
[rank0]: ValueError: Schema and number of arrays unequal                                                                                                                     
                                                                                                                                                                             
[rank0]: During handling of the above exception, another exception occurred:                                                                                                 
                                                                                                                                                                             
[rank0]: Traceback (most recent call last):                                                                                                                                  
[rank0]:   File "/mnt/tidal-alsh01/usr/dawo/qinshengqian/c1/OneThinker/LLaMA-Factory/src/llamafactory/launcher.py", line 57, in <module>                                     
[rank0]:     run_exp()                                                                                                                                                       
[rank0]:   File "/mnt/tidal-alsh01/usr/dawo/qinshengqian/c1/OneThinker/LLaMA-Factory/src/llamafactory/launcher.py", line 41, in run_exp                                      
[rank0]:     return _run_exp()  # use absolute import                                                                                                                        
[rank0]:            ^^^^^^^^^^                                                                                                                                               
[rank0]:   File "/mnt/tidal-alsh01/usr/dawo/qinshengqian/c1/OneThinker/LLaMA-Factory/src/llamafactory/train/tuner.py", line 110, in run_exp                                  
[rank0]:     _training_function(config={"args": args, "callbacks": callbacks})                                                                                               
[rank0]:   File "/mnt/tidal-alsh01/usr/dawo/qinshengqian/c1/OneThinker/LLaMA-Factory/src/llamafactory/train/tuner.py", line 72, in _training_function                        
[rank0]:     run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)                                                                      
[rank0]:   File "/mnt/tidal-alsh01/usr/dawo/qinshengqian/c1/OneThinker/LLaMA-Factory/src/llamafactory/train/sft/workflow.py", line 51, in run_sft                            
[rank0]:     dataset_module = get_dataset(template, model_args, data_args, training_args, stage="sft", **tokenizer_module)                                                   
[rank0]:                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                   
[rank0]:   File "/mnt/tidal-alsh01/usr/dawo/qinshengqian/c1/OneThinker/LLaMA-Factory/src/llamafactory/data/loader.py", line 391, in get_dataset                              
[rank0]:     dataset = _get_preprocessed_dataset(                                                                                                                            
[rank0]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                            
[rank0]:   File "/mnt/tidal-alsh01/usr/dawo/qinshengqian/c1/OneThinker/LLaMA-Factory/src/llamafactory/data/loader.py", line 327, in _get_preprocessed_dataset                
[rank0]:     raise ValueError(f"e : {e}")
[rank0]: ValueError: e : Schema and number of arrays unequal

[tulerfeng]

Hi, thanks for the issue.

This error is usually related to inconsistencies in the dataset schema during the pre-processing (tokenization) stage, for example when different samples or batches produce different output fields or when some samples fail to be processed correctly.

Are you encountering this error during the data pre-processing / tokenization stage, or during the actual training stage?

In our case, we observed that when pre-processing a large number of video samples, the original data loader logic could occasionally run into such issues. The original implementation in Llamafactory (https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/data/loader.py) was:

dataset = dataset.map(
    dataset_processor.preprocess_dataset,
    batched=True,
    batch_size=data_args.preprocessing_batch_size,
    remove_columns=column_names,
    **kwargs,
)

To make the pre-processing more robust to problematic samples, we introduced a safety wrapper that handles per-sample failures and explicitly enforces a consistent output schema:

dataset = dataset.map(
    _safe_batch_wrapper(dataset_processor.preprocess_dataset),
    batched=True,
    batch_size=data_args.preprocessing_batch_size,
    remove_columns=column_names,
    features=features,
    writer_batch_size=8,
    **kwargs,
)

You may change our implementation to the original versions and checking whether this issue can be addressed.

[ddz16]

I have encountered the same error during the data pre-processing / tokenization stage.

And I try to use the original versions:

dataset = dataset.map(
    dataset_processor.preprocess_dataset,
    batched=True,
    batch_size=data_args.preprocessing_batch_size,
    remove_columns=column_names,
    feaures=features,
    **kwargs,
)

The error can be solved.

[tulerfeng]

@ddz16 Thank you for sharing this !

[Fu-Fu-Fu-Fu]

@ddz16 Thank you for sharing this !

您好，我在预处理数据的时候发现报错：ValueError: Processor was not found, please check and update your model file.
这似乎是transformers<=4.57.0版本不支持Qwen3VL，但是根据您项目中指引来安装llamafactory又要求transformers<=4.57.0，这个您是怎么解决的呀

已经解决，运行脚本前加一个export DISABLE_VERSION_CHECK=1，避开版本检查

[tulerfeng]

@Fu-Fu-Fu-Fu 4.57.0是支持qwen3-vl的，可以参考 https://github.com/huggingface/transformers/releases/tag/v4.57.0

[hulianyuyy]

I notice that you run experiments on 32 GPUs. If i perform experiments on 8 GPUs, should i decrease the learning rate by 4 during SFT? Thank you.

[tulerfeng]

@hulianyuyy Hi, I think instead of decrease learning rate, increasing gradient_accumulation_steps=4 in the yaml file maybe a better choice. Because batchsize=8 may be too small.

[hulianyuyy]

Many thanks for your kind reply!

[hulianyuyy]

I want to ask if you have explored the effects of different data combinations on the performance after SFT training, e.g. using image-only, video-only or both for training. Would combining both lead to substantial performance boost?

[tulerfeng]

@hulianyuyy Hi, I think it depends on your tasks. In my case, combing image data : video data = 1 : 1 yields the best results.

I have not tested the image-only, video-only SFT checkpoints before.