Is the qnn model use mixed precision quantization?

[zhouyongxyz]

Hi, I find when I use the python converter.py
--model-folder /work/.cache/modelscope/hub/qwen/Qwen2___5-0___5B-Instruct/
--model-name qwen2_0.5b
--system-prompt-file /work/code/projects/PowerServe/assets/system_prompts/qwen2.txt
--prompt-file /work/code/projects/PowerServe/assets/prompts/long_prompt.txt
--batch-sizes 1 128
--artifact-name qwen2_0.5b
--n-model-chunk 1
--output-folder ./qwen2.5-0.5b-QNN
--build-folder ./qwen2.5-0.5b-QNN-tmp
--soc sa8775

try to convert the Qwen2___5-0___5B-Instruct/ to qnn target. but I find the messy result. Did you use fp16 and int4 weight when convert onnx to qnn bin (use qnn default quant method)?

Thanks

[SorryMaker2022]

Hi. Please attach your full execution log and environment settings (in model conversion or inference phase).
We're using W4A16 per-channel quantization with some ops specified as fp16.

To make a quick guess, here are some possible causes:

• Qwen series of models are very sensitive to quantization. The QNN per-channel quantization is just not enough.
• Automotive SoCs has a different set of SDKs though they look the same with the mobile ones, they are actually different. Downloading a SDK from Qualcomm Software Center may cause unexpected problems. You can ask your SoC vendor for the right SDK.
• This converter uses Matmul op instead of Conv2D op. According to Qualcomm docs Conv2D op is better-supported compared with Matmul while they are mathematically equivalent. The Matmul ops with quantization on prior SoCs may be buggy internally.
• Qualcomm states that there is a feature on HMX to enhance Conv2D quantization precisions starting in HTP V73. But we currently have no further information about it. This setting may be in the device config stage.

Note that you can explicitly specify several layers as fp16:

PowerServe/tools/qnn_converter/model_params.py

Lines 34 to 55 in 5ef333a

	class Llama3_1_8B_Params(ModelParams):
	has_qkv_bias = False
	use_drelu = False
	tie_embedding = False
	n_layers = 32
	vocab_size = 128256
	ffn_hidden_dim = 14336
	head_dim = 128
	n_heads = 32
	n_kv_heads = 8
	rope_theta = 5e5
	rms_norm_eps = 1e-5
	attention_mask_value = -1e5
	fp16_attention_layers = []
	fp16_ffn_layers = []
	fp16_rope = False
	fp16_qkv_heads = False
	n_fp16_heads = 0
	n_fp16_neurons = 0

You can try to specify fp16_layers first (we suggest trying to start with the first and last layers, for example, layers 0, 1, 2, 21, 22, 23).
If layers don't work, you can then try the fp16_heads and fp16_neurons.

If you are willing to contribute to this project, you can also try to replace the Matmul ops with Conv2D ops in the converter, or support the W4FP16 (Weights stored as INT4 but dequantized to FP16 in inference) in the converter.

[zhuipiaochen]

Hi. Please attach your full execution log and environment settings (in model conversion or inference phase). We're using W4A16 per-channel quantization with some ops specified as fp16.

To make a quick guess, here are some possible causes:

• Qwen series of models are very sensitive to quantization. The QNN per-channel quantization is just not enough.
• Automotive SoCs has a different set of SDKs though they look the same with the mobile ones, they are actually different. Downloading a SDK from Qualcomm Software Center may cause unexpected problems. You can ask your SoC vendor for the right SDK.
• This converter uses Matmul op instead of Conv2D op. According to Qualcomm docs Conv2D op is better-supported compared with Matmul while they are mathematically equivalent. The Matmul ops with quantization on prior SoCs may be buggy internally.
• Qualcomm states that there is a feature on HMX to enhance Conv2D quantization precisions starting in HTP V73. But we currently have no further information about it. This setting may be in the device config stage.

Note that you can explicitly specify several layers as fp16:

PowerServe/tools/qnn_converter/model_params.py

Lines 34 to 55 in 5ef333a

class Llama3_1_8B_Params(ModelParams):
has_qkv_bias = False
use_drelu = False
tie_embedding = False

 n_layers = 32 
 vocab_size = 128256 
 ffn_hidden_dim = 14336 
 head_dim = 128 
 n_heads = 32 
 n_kv_heads = 8 

 rope_theta = 5e5 
 rms_norm_eps = 1e-5 
 attention_mask_value = -1e5 

 fp16_attention_layers = [] 
 fp16_ffn_layers = [] 
 fp16_rope = False 
 fp16_qkv_heads = False 
 n_fp16_heads = 0 
 n_fp16_neurons = 0 

You can try to specify fp16_layers first (we suggest trying to start with the first and last layers, for example, layers 0, 1, 2, 21, 22, 23). If layers don't work, you can then try the fp16_heads and fp16_neurons.

If you are willing to contribute to this project, you can also try to replace the Matmul ops with Conv2D ops in the converter, or support the W4FP16 (Weights stored as INT4 but dequantized to FP16 in inference) in the converter.

@SorryMaker2022 Hi，I have a 7GEN3 device, it do not support FP16, So what should I change to make it work？

Design objective: modularity, scalability, extensibility, and portability
:Features
Supports SM7675P and other devices in multiple tiers
Hardware Acceleration of Al/ML through Kryo CPU (INT8,FP32), Adreno GPU (FP16 and FP32), HexagonTensor ProcessorINT(INT4,INT8 and INT16)
INT4 support is dependent on layer configuration , it may vary from model to model.
Supports mixed INT precision (INT4,INT8 and IN16)Enablement
Requires QNN 2.x
General public availability by early 2024.
·MigrationAny FP16 models from legacy chipsets need to be quantized to INT16 for HTP backend.Models needs to prepared for SM7675P SOC number Document and resources
Qualcomm Neural Network SDK: Available on Qualcomm Package Manager for downloadContain extensive SDK documentation with examplesChipset-specific features information and updates on SDK will be available in time for FC

[SorryMaker2022]

Hi @zhuipiaochen ,

Thanks for your interest in deploying this project on Snapdragon 7 Gen 3. However, the hardware specs you provided seem a bit unclear and don’t fully match the public documentation (see https://docs.qualcomm.com/bundle/publicresource/[87-73943-1_REV_E_Snapdragon_7__Gen_3_Mobile_Platform_Product_Brief.pdf page 2 Artificial Intelligence section). Could you double-check the details to see whether it supports FP16 or not? Please attach the HTP architecture version of your SoC, too.

Having accurate and consistent information will help us determine if and how the project can be adapted for this hardware. Looking forward to your update!

Also, there is an ambiguity in your problem: I am not sure whether you want to run an LLM on Snapdragon 7 Gen 3 without FP16 support, or you want to run the Qwen series (in the models we tested, only the Qwen series models presented sensitivity to QNN quantization) model with fair precision on Snapdragon 7 Gen 3. This will lead to different workarounds.

BTW, to keep discussions organized and ensure better tracking, could you please open a new issue for this? That way, we can properly address it without mixing topics.

Thanks for your understanding!

[zhuipiaochen]

Hi @zhuipiaochen ,

Thanks for your interest in deploying this project on Snapdragon 7 Gen 3. However, the hardware specs you provided seem a bit unclear and don’t fully match the public documentation (see [https://docs.qualcomm.com/bundle/publicresource/87-73943-1_REV_E_Snapdragon_7__Gen_3_Mobile_Platform_Product_Brief.pdf page 2 Artificial Intelligence section). Could you double-check the details to see whether it supports FP16 or not? Please attach the HTP architecture version of your SoC, too.

Having accurate and consistent information will help us determine if and how the project can be adapted for this hardware. Looking forward to your update!

Also, there is an ambiguity in your problem: I am not sure whether you want to run an LLM on Snapdragon 7 Gen 3 without FP16 support, or you want to run the Qwen series (in the models we tested, only the Qwen series models presented sensitivity to QNN quantization) model with fair precision on Snapdragon 7 Gen 3. This will lead to different workarounds.

BTW, to keep discussions organized and ensure better tracking, could you please open a new issue for this? That way, we can properly address it without mixing topics.

Thanks for your understanding!

@SorryMaker2022 Hi, I have created a new issue about 7+gen3 :#8