How to run this project on 7+gen3（soc=70, arch=V73）

[zhuipiaochen]

Hi，We tried to run this project on 7+gen3 and found the following error:

[INFO ] Compiled on: Mar  7 2025 at 12:05:31
[INFO ] Load model qwen2 ...
[INFO ] after model init: [Memory(MB)] VMS: 11028, RSS: 522
[INFO ] QNN core API version: 2.22.0
[INFO ] QNN backend API version: 5.29.0
[INFO ] QNN system API version: 1.2.0
[INFO ] QNN backend properties:
[INFO ] - Create context from binary list: Yes
[INFO ] - Dynamic batch: Yes
[INFO ] - Early termination: No
[INFO ] - Dynamic dimensions: Yes
[INFO ] - Blockwise quantization: Unknown
[INFO ] - Blockwise quantization with expansion: Unknown
[INFO ] - Vector quantization: Unknown
[INFO ] - Tensor sparsity: Yes
[INFO ] - Updateable application tensor: Yes
[INFO ] - Updateable native tensor: Yes
[INFO ] - Updateable static tensor: Yes
[INFO ] Hardware device information:
[INFO ] [0] id=0, type=0, num_cores=1, ext_type=0
[INFO ] [0] core[0]: id=0, type=0
[INFO ] [0] on_chip: soc=70, arch=73, dlbc=true, signed_pd=true, vtcm_size=4
[INFO ] Load model model-2025_03_10 from /data/local/tmp/qwen2.5-0.5b-instruct/qwen2.5-0.5b-instruct-model/qnn
[INFO ] Loading "qwen2_0.5b_0.bin"...
[INFO ] load binary file /data/local/tmp/qwen2.5-0.5b-instruct/qwen2.5-0.5b-instruct-model/qnn/qwen2_0.5b_0.bin (size: 730027424) into address space [0x6dafc44000, 0x6ddb4795a0)
QnnDsp <E> Request feature fp16 enable with value 1 unsupported
QnnDsp <E> Failed to register context to device and backend
QnnDsp <E> Failed to create context from binary with err 0x138d
libc++abi: terminating due to uncaught exception of type powerserve::EnvironmentException: [Exception][QNN] BasicException: EnvironmentException
[Exception][QNN] EnvironmentException: failed to create QNN context from binary: /data/local/tmp/qwen2.5-0.5b-instruct/qwen2.5-0.5b-instruct-model/qnn/qwen2_0.5b_0.bin
[Exception][QNN] System error: Success
Aborted

However, the documentation for 7+gen3 shows that its NPU supports fp16:

Qualcomm® AI Engine
• Qualcomm® Adreno™ GPU
• Qualcomm® Kryo™ CPU
• Qualcomm® Hexagon™ NPU
• Fused AI accelerator architecture
• Hexagon scalar, vector, and tensor accelerators
• Hexagon Direct Link
• Support for mix precision (INT8+INT16)
• Support for all precisions (INT4, INT8, INT16, FP16)

https://docs.qualcomm.com/bundle/publicresource/87-73943-1_REV_E_Snapdragon_7__Gen_3_Mobile_Platform_Product_Brief.pdf

[SorryMaker2022]

It seems your SoC does not support FP16. You can try to fall back to Q8 or Q4 quantization types. You can also ask your vendor or Qualcomm for further information (like whether there is a workaround to run fp16 or how to migrate models, as in issue#7 there seemed to be some proprietary documents in your organization).

Migration: Any FP16 models from legacy chipsets need to be quantized to INT16 for HTP backend. Models needs to prepared for SM7675P SOC number Document and resources
Qualcomm Neural Network SDK: Available on Qualcomm Package Manager for downloadContain extensive SDK documentation with examplesChipset-specific features information and updates on SDK will be available in time for FC

You can modify this part of code to quantize the model into the data type you like:

PowerServe/tools/qnn_converter/export_to_onnx.py

Lines 594 to 690 in 5ef333a

	def export_quantization_config(self):
	class Encoding(NamedTuple):
	category: Literal["activation", "param"]
	bitwidth: int
	dtype: Literal["float", "int"]
	encoding_map: Dict[str, Encoding] = {}
	def update_encoding(name: str, encoding: Encoding):
	"""Set encoding for name. Only update if the bitwidth is larger than the previous setting."""
	if name not in encoding_map or encoding.bitwidth > encoding_map[name].bitwidth:
	encoding_map[name] = encoding
	def encode_activation(node: str | onnx.ValueInfoProto, bitwidth: int):
	if not isinstance(node, str):
	node = node.name
	update_encoding(node, Encoding("activation", bitwidth, "float"))
	def encode_output(node: onnx.NodeProto, bitwidth: int):
	for name in node.output:
	update_encoding(name, Encoding("activation", bitwidth, "float"))
	def encode_param(node: str | onnx.NodeProto, bitwidth: Encoding, dtype: Literal["float", "int"]):
	if not isinstance(node, str):
	node = node.name
	update_encoding(node, Encoding("param", bitwidth, dtype))
	def match(target: str | onnx.NodeProto | onnx.TensorProto | onnx.ValueInfoProto, pattern: str):
	if not isinstance(target, str):
	target = target.name
	return re.fullmatch(pattern, target) is not None
	graph = self.onnx_model.graph
	# Inputs
	encode_activation("x", 32)
	encode_activation("attn_bias", 16)
	# KV cache: FP16
	for node in graph.input:
	if match(node, "layer_[0-9]+_(key_t|value)_cache_[0-9]+"):
	encode_activation(node, 16)
	for node in graph.output:
	if match(node, "layer_[0-9]+_(key|value)_[0-9]+"):
	encode_activation(node, 16)
	# Attention core: FP16
	for node in graph.node:
	if match(node, "/layers\.[0-9]+/attn/core.*"):
	encode_output(node, 16)
	# Residual connection: FP32
	for node in graph.node:
	if match(node, "/layers\.[0-9]+/(attn|ffn|ffn/down_proj)/Add(_[0-9]+|)"):
	encode_output(node, 32)
	# RMSNorm: FP32
	for node in graph.initializer:
	if match(node, "layers\.[0-9]+\.(attn|ffn)\.norm\.weight"):
	encode_param(node, 32, "float")
	for node in graph.node:
	# print(node)
	if match(node, "/layers\.[0-9]+/(attn|ffn)/norm.*"):
	encode_output(node, 32)
	elif match(node, "/norm.*"):
	encode_output(node, 32)
	if self.use_fp16:
	print("NOTE: Use FP16 for output embedding.")
	for node in graph.initializer:
	encode_param(node, 16, "float")
	for node in graph.node:
	encode_output(node, 16)
	# Generate config
	config = {
	"version": "0.6.1",
	"quantizer_args": {
	"activation_bitwidth": 16,
	"param_bitwidth": 4,
	"dtype": "int",
	"per_channel_quantization": True,
	"quant_scheme": "post_training_tf",
	},
	"activation_encodings": {},
	"param_encodings": {},
	}
	for name, encoding in sorted(encoding_map.items(), key=lambda item: item[0]):
	config[f"{encoding.category}_encodings"][name] = [{
	"bitwidth": encoding.bitwidth,
	"dtype": encoding.dtype,
	"is_symmetric": str(encoding.category == "param"),
	}]
	export_json(config, self.output_folder / f"{self.graph_name}.encodings")

You can refer to this fork:
https://github.com/shuojiangliu/PowerServe/blob/a6ca19cd5a970dcf2969585438a31ec2217c11a5/tools/qnn_converter/export_to_onnx.py#L610-L723

Note that using per-channel or per-tensor Q8 or Q4 quantization may cause loss in precision. You can try out llama series of models first if qwen models output gibberish. Theoretically decreasing the attention mask value (modify from -1e5 to -1e2) may help with precision on llama models, but this does not work on qwen models because the activation magnitude of qwen models are larger.

I appreciate your enthusiasm for bringing this project to older Qualcomm platforms, however, at this point, it seems like I’ve accidentally been cast as your personal migration consultant. While I’d love to help (will be better as your colleague in your startup, just kidding :D ), navigating Qualcomm's environment does require some self-study.

I'd recommend diving into the official Qualcomm documentation and your proprietary documentation—it's a treasure trove of information (and way more comprehensive than my replies). If you hit specific roadblocks after that, feel free to open a well-scoped issue that offers clear questions and well-organized information.

Best of luck, and may the migration odds be ever in your favor.