[v2] Missing demo scripts for Android QNN backend execution (was available in v1)

[OujaKivi]

In the current mllm_v2 version, I can see that custom tasks can be run via task.py. However, I have not found a one-click execution script like run_qwen_qnn.sh which was available in the v1 version.
I have already completed the compilation following the Android section in the README file, so how can I quickly start an inference test for a specific LLM (e.g., Qwen)?

[chenghuaWang]

Hello OujaKivi,

Thank you for using the mllm framework.

1. We will soon add a one-click tutorial for using NPU Qwen at https://ubiquitouslearning.github.io/mllm/qnn_backend/setup_env.html
2. task.py is primarily intended to help you compile projects, not run them
3. Running Qwen on Arm CPU is very simple. You can use task.py tasks/adb_push.py to upload the binary files to your phone, and then run it with the command ./mllm-qwen-service -m ../qwen3-0.6B-xxxx.

[OujaKivi]

Hello OujaKivi,

Thank you for using the mllm framework.

1. We will soon add a one-click tutorial for using NPU Qwen at https://ubiquitouslearning.github.io/mllm/qnn_backend/setup_env.html
2. task.py is primarily intended to help you compile projects, not run them
3. Running Qwen on Arm CPU is very simple. You can use task.py tasks/adb_push.py to upload the binary files to your phone, and then run it with the command ./mllm-qwen-service -m ../qwen3-0.6B-xxxx.

Hello ChenghuaWang,

Thank you for your warm reply.

I'd like to check if there's an estimated timeline for the NPU Qwen tutorial.
If possible, could you attach a draft version to this issue?

Thanks a lot!

[chenghuaWang]

Tonight, we will upload a streamlined Qwen 1.5 1.8B NPU model to ModelScope, complete with usage instructions. The methods for rotation quantization training will be shared soon. (We are downloading the model from our internal server, it takes lots of time... waiting for internet to finish its coffee break...)

[guoguo1314]

I'm really looking forward to this. could you please also open-source the code, models, and usage instructions for the paper titled "Accelerating Mobile Language Model via Speculative Decoding and NPU-Coordinated Execution"? thank you !

[csAugust]

I'm really looking forward to this. could you please also open-source the code, models, and usage instructions for the paper titled "Accelerating Mobile Language Model via Speculative Decoding and NPU-Coordinated Execution"? thank you !

Hi guoguo1314, thanks a lot for your interest in our work!

At the moment, the codebase for this paper was developed on top of the early internal v1 framework, and it requires some refactoring and cleanup before it can be released. Since the framework has now evolved to v2, a straightforward merge is also non-trivial, so I’ll need some time to reorganize the repository properly.

I'm considering implementing the system atop the newer v2 mllm. However, this may take a bit of time. Thank you for your understanding.

[Sp0tless]

Here is how you can build and run the current main branch with the mllm-qwen-npu example.

1. Download and push the model to the device

   • Download the Qwen1.5 1.8B models, tokenizer and config files from the ModelScope org:

     • https://www.modelscope.cn/organization/mllmTeam

   • Push these files to your target device (e.g. /data/local/tmp/).

2. Update paths in the example and build

   • Open examples/qwen_npu/main.cpp and replace the hard-coded paths for model / tokenizer / config with the actual paths on your device.
   • Build the example to get the mllm-qwen-npu executable.
   • Note: in the future we plan to remove these hard-coded paths and pass them via command-line arguments, so this manual edit step will no longer be needed.

3. Extract required libraries from the QNN SDK

   • Assume your QNN SDK is installed under QNN_SDK_ROOT (typically something like /opt/qcom/aistack/qairt/2.xxxxxx).

   • First, check your device SoC and the corresponding NPU version (e.g. v75). You can refer to:

     • https://docs.qualcomm.com/nav/home/QNN_general_overview.html?product=1601111740009302#supported-snapdragon-devices

   Taking NPU v75 as an example, copy the following libraries:

   • From $QNN_SDK_ROOT/lib/aarch64-android/:

     • libQnnHtp.so
     • libQnnHtpV75Stub.so
     • libQnnHtpPrepare.so
     • libQnnHtpProfilingReader.so
     • libQnnHtpOptraceProfilingReader.so
     • libQnnHtpV75CalculatorStub.so
     • Important: you also need libQnnSystem.so from this aarch64-android directory.
       Do not use the libQnnSystem.so under unsigned/lib.

   • From $QNN_SDK_ROOT/lib/hexagon-v75/unsigned/:

     • libQnnHtpV75.so
     • libQnnHtpV75Skel.so

4. Prepare the custom op package libraries (LLaMAPackage)

   • In your built QNN custom op package directory:
     mllm/backends/qnn/custom-op-package/LLaMAPackage
     locate the libQnnLLaMAPackage.so libraries.

   • Take the one compiled for CPU, copy it and rename it to:

     • libQnnLLaMAPackage_CPU.so

   • Take the one compiled for NPU (HTP), copy it and rename it to:

     • libQnnLLaMAPackage_HTP.so

5. Prepare additional runtime dependencies from this repo and the Android NDK

   • After building this repository, you will also get a set of libMllm*.so shared libraries (in the same output directory as mllm-qwen-npu). These are required at runtime as well, so please copy them to the device together with the binary.
   • In addition, the project depends on libomp.so from the Android NDK. Please locate the appropriate aarch64 libomp.so in your Android NDK and copy it to the device as well.

6. Copy everything to the device and run

   • Copy all the mentioned .so files (from the QNN SDK, LLaMAPackage, libMllm*.so, and libomp.so) plus the mllm-qwen-npu executable to some directory on the device, e.g.:

     • /data/local/tmp/

   • On the device, change to that directory and run with the library path set, e.g.:

     LD_LIBRARY_PATH=/data/local/tmp ./mllm-qwen-npu

   • If you still see errors about missing libraries, try adding ADSP_LIBRARY_PATH as well:

     ADSP_LIBRARY_PATH=/data/local/tmp LD_LIBRARY_PATH=/data/local/tmp ./mllm-qwen-npu

[guoguo1314]

Hello Sp0tless,
the mllmTeam on ModelScope doesn't have the Qwen1.5 1.8B models, tokenizer, and config files.

[Sp0tless]

we have uploaded them just now
https://www.modelscope.cn/models/mllmTeam/Qwen1.5-1.8B-Chat

[guoguo1314]

thanks, Sp0tless
I think I was the first to download it（占坑）！

[guoguo1314]

When I use the command LD_LIBRARY_PATH=/data/local/tmp ./mllm-qwen-npu on the 8gen3 to infer the qwen1.5-1.8b model downloaded from ModelScope, why does it take 84.27 seconds to process 194 tokens during prefill, with an average speed of only 2.31 tokens per second? This seems too slow.

[KarthikL1729]

Hi, I've tried running v2 on the NPU of a SD 8 Elite (Hexagon v79) and the dynamic cast operations all fail when the prefill stage begins and the executable segfaults. I'm not sure why, since the ops were compiled for htp_v79. Are there other op libs that I need to push to the device to get this working? Pasting a snippet of the logs below. This is for Qwen1.5 1.8B chat from modelscope. Could it be a backend mismatch with the .mllm model? I am using QNN SDK version 2.40.0.251030.

[INFO] /root/mllm/examples/qwen_npu/main.cpp:33 Mixed inference mode: NPU prefill + CPU decode
[INFO] /root/mllm/examples/qwen_npu/main.cpp:55 CPU decode model loaded from: /data/local/tmp/Qwen1.5-1.8B-Chat/qwen1.5-1.8b-chat-rot_q4_0.mllm
[INFO] /root/mllm/examples/qwen_npu/main.cpp:59 Loading QNN model for prefill...
[INFO] /root/mllm/mllm/backends/qnn/QNNUtils.cpp:23 QNN Backend Lib: libQnnHtp.so
[INFO] /root/mllm/mllm/backends/qnn/QNNBackend.cpp:306 Registered Op Package: libQnnLLaMAPackage_CPU.so and interface provider: LLaMAPackageInterfaceProvider
[INFO] /root/mllm/mllm/backends/qnn/QNNBackend.cpp:306 Registered Op Package: libQnnLLaMAPackage_HTP.so and interface provider: LLaMAPackageInterfaceProvider
[INFO] /root/mllm/mllm/backends/qnn/QNNBackend.cpp:47 QNN Backend Build Id: v2.40.0.251030114326_189385
[INFO] /root/mllm/mllm/backends/qnn/QNNBackend.cpp:49 QNN backend supports tensor sparsity
[INFO] /root/mllm/mllm/backends/qnn/QNNBackend.cpp:52 QNN backend supports dynamic dimensions
[INFO] /root/mllm/mllm/backends/base/PluginSystem.cpp:89 Register customized op: DequantizeAdd:4097 -> QNN
[INFO] /root/mllm/examples/qwen_npu/main.cpp:66 Created shared StaticCache with 24 layers
[INFO] /root/mllm/examples/qwen_npu/main.cpp:71 QNN prefill model loaded from: /data/local/tmp/Qwen1.5-1.8B-Chat/qwen1.5-1.8b-chat-rot-qnn.mllm
[INFO] /root/mllm/examples/qwen_npu/main.cpp:81 Configured 24 QNN KVCache layers to use shared StaticCache
[INFO] /root/mllm/examples/qwen_npu/main.cpp:91 Configured 24 CPU KVCache layers to use shared StaticCache
[INFO] /root/mllm/examples/qwen_npu/main.cpp:103 Input tokens: 194 tokens
[INFO] /root/mllm/examples/qwen_npu/main.cpp:161 Starting QNN prefill...
[ERROR] /root/mllm/mllm/backends/qnn/op/QNNRMSNormOp.cpp:34 Failed to cast to QNNRMSNormOp
[ERROR] /root/mllm/mllm/backends/qnn/passes/QNNGraphBuildPass.cpp:160 Failed to add node for op type: RMSNorm in graph 'model.layers.0_1'
[ERROR] /root/mllm/mllm/backends/qnn/op/QNNLinearOp.cpp:135 Failed to cast to QNNLinearOp
[ERROR] /root/mllm/mllm/backends/qnn/passes/QNNGraphBuildPass.cpp:160 Failed to add node for op type: Linear in graph 'model.layers.0_2'
... <repeats for every layer>
[ERROR] /root/mllm/mllm/backends/qnn/op/QNNRMSNormOp.cpp:34 Failed to cast to QNNRMSNormOp
[ERROR] /root/mllm/mllm/backends/qnn/passes/QNNGraphBuildPass.cpp:160 Failed to add node for op type: RMSNorm in graph 'model.layers.23_1'
[ERROR] /root/mllm/mllm/backends/qnn/op/QNNLinearOp.cpp:135 Failed to cast to QNNLinearOp
[ERROR] /root/mllm/mllm/backends/qnn/passes/QNNGraphBuildPass.cpp:160 Failed to add node for op type: Linear in graph 'model.layers.23_2'
[WARN] /root/mllm/mllm/backends/cpu/kernels/common/ggml/vec_dot_type.hpp:181 Unsupported DataType Int8
[WARN] /root/mllm/mllm/backends/cpu/kernels/common/ggml/vec_dot_type.hpp:181 Unsupported DataType Int8
Segmentation fault 

[oreomaker]

When I use the command LD_LIBRARY_PATH=/data/local/tmp ./mllm-qwen-npu on the 8gen3 to infer the qwen1.5-1.8b model downloaded from ModelScope, why does it take 84.27 seconds to process 194 tokens during prefill, with an average speed of only 2.31 tokens per second? This seems too slow.

The qnn in v2 is a basic migration from v1, which only supports the whole process of online graph building and execution. The demo will perform QNNGraphBuildPass online, which takes a lot of time. Load the qnn graph context from binary file will save some time. We will improve this later.

[OujaKivi]

When I use the command LD_LIBRARY_PATH=/data/local/tmp ./mllm-qwen-npu on the 8gen3 to infer the qwen1.5-1.8b model downloaded from ModelScope, why does it take 84.27 seconds to process 194 tokens during prefill, with an average speed of only 2.31 tokens per second? This seems too slow.

The qnn in v2 is a basic migration from v1, which only supports the whole process of online graph building and execution. The demo will perform QNNGraphBuildPass online, which takes a lot of time. Load the qnn graph context from binary file will save some time. We will improve this later.

Thanks, @oreomaker and @Sp0tless !

I'm wondering if there's a straightforward way to get the elapsed time of QNNGraphBuildPass, so that the real TTFT can be calculated after then. BTW, do you have a rough timeline for the release of the binary graph loading version?

---

I added detailed elapsed time LOG of createQNNXxxxPass and npu_model->forward(past, prefill_args). Can I equivalently achieve the expected prefill performance data by forward_duration?

[INFO] /home/wjw/mllm/examples/qwen_npu/main.cpp:161 Starting QNN prefill...
[INFO] /home/wjw/mllm/examples/qwen_npu/main.cpp:196 QNN prefill completed: seq_len=194, time=83.50s
[INFO] /home/wjw/mllm/examples/qwen_npu/main.cpp:199 createQNNGraphIOTensorPass+createQNNOpNamingPass duration: 0.00s
[INFO] /home/wjw/mllm/examples/qwen_npu/main.cpp:200 createQNNGraphBuildPass_duration: 79.69s
[INFO] /home/wjw/mllm/examples/qwen_npu/main.cpp:201 forward_duration: 3.75s

[chenghuaWang]

I can provide a rough timeline—we will deliver the AOT's Qnn mode using LPBQ quantization around the first week of January. The PR can be found at #563.