RyzenAI 1.6.1 - Linux Feedback

[X-Ryl669]

After installing the 1.6.1 version of the package that only available through the wallgarden, I'd like to thank you for your work, since it actually make us closer to be able to use the linux environment with your technologies.

Here is the feedback and issues I've found to actually get a working system:

Dependencies

The fact that you're limiting to outdated linux distribution is a pain. I can understand you're not delivering open source libraries, but please follow the usual guideline for delivering binaries. Typically, your libraries are linked with 2 problematic dependencies:

• boost filesystem
• stdc++

For the former, you should link statically with it, or simply switch to C++'s standard std::filesystem code. One could solve the issue with downloading the libboost-filesystem-1.74.so file from debian repository and installing it in the lib/python/site-packages/.../voe/libs folder, but it's only a matter of time until debian bumps the version and your code will again fail to link dynamically.

For the latter, it's not a good idea to distribute it, since the other libraries on the system will likely link with the system's libstdc++, thus fail to link when your library make use of the version you're distributing (only a single libstdc++ is allowed in a program's shared library resolution). GCC developpers make great care to ensure libstdc++ is backward compatible, so the system's version will work with your libraries. If it doesn't, providing a libstdc++ that's built for a newer system will cause trouble anyway, since it might use syscall and so on that aren't present on the system. So in short, don't deliver a libstdc++ in your package. I had to delete your lib for shared linking to work.

Mess in virtual environment

You've hand edited the bin/activate script, but forget to apply the same change to the other shell's scripts. I can provide a patch for the shell script I'm using if required.

This is very fragile anyway, since in order to activate/setup the shell is typically hard to get right. One has to first source the /opt/xilinx/xrt/setup before this one, because both modify the LD_LIBRARY_PATH variable in order to find their library and this one doesn't do it right.

The activate script also doesn't add the dependency/lib folder to the LD_LIBRARY_PATH so the model_benchmark binary has to be tweaked to run.

In the end, the use of the environment variable for storing path is IMHO, clumsy. I don't see how the lemonade server could use the package with all this messing environment variables. Maybe you should use a config file instead that's specifying where the other files are, something like "/etc/amd/ryzenai.conf", so a third-party software isn't polluted with a longish LD_LIBRARY_PATH that would break its libraries.
Even better, create a /lib/ryzenai folder to store your libraries and add the path to a ldconfig script (in the usual /etc/ld.so.conf.d/ folder). So it doesn't require a messy hack to run them.

Missing actual software in the package

One of the main use of this package is to be able to use the NPU on linux. Yet, there are no software making this possible in the package. The only software (quicktest, model_benchmark) are both working (great!) but are useless from the user point of view. The LLM's run_model.py doesn't accept a OGA model (which is the point of the package/NPU, right?) even after hand editing the genai_config.json to replace the Windows's dll by a linux .so file (while model_benchmark works fine with the same model!).

So after all the trouble of installing and fixing the mess, we have nothing to actually play with, except a benchmark that just tease us but with no outside world use.

OGA's genai_config.json is windows only

There are too many windowsity in the downloaded model. Lemonade-server can pull the model just fine, but it can't run it (it's not even listed). It has to be hand edited to remove the ".dll" in the file and replace them with the equivalent ".so" file. There's also a "directx_xrt" lurking around I don't know if I need to change it or not. Maybe the base of the library should be used instead (so "onnx_custom_ops" instead of "onnx_custom_ops.dll") and let the actual software which knows what platform it is running on, sort out the actual filename, it's not hard to append a prefix and a suffix to a string.

Also the delivered python code (like run_model.py) doesn't understand the genai_config structure and chokes on unexpected keys in the json object. This means it wasn't even tested before delivery with an actual model 😢

No stable diffusion example

In the previous version and in the documentation, there's a stable diffusion example to use GenAI. But the delivered package doesn't contain it anymore. Can I use the SD models ? If so, how?

Ignored by lemonade_server

I'm not sure I've done everything right with this one, but it seems lemonade_server is completely ignoring the package. If I made a mistake in configuring it, please tell me how to fix it?

[wc2333]

(ryzen-ai) root@Default-string:/run_llm# ls
amd_genai_prompt.txt deployment model_benchmark Phi-3.5-mini-instruct-onnx-ryzenai-npu
(ryzen-ai) root@Default-string:/run_llm# mv /root/ryzen_ai-1.6.1/venv/lib/python3.10/site-packages/flexml/flexml_extras/lib/libstdc++.so.6.bak /root/ryzen_ai-1.6.1/venv/lib/python3.10/site-packages/flexml/flexml_extras/lib/libstdc++.so.6
(ryzen-ai) root@Default-string:~/run_llm# ./model_benchmark -i Phi-3.5-mini-instruct-onnx-ryzenai-npu/ -l 128 -f amd_genai_prompt.txt
./model_benchmark: /root/ryzen_ai-1.6.1/venv/lib/python3.10/site-packages/flexml/flexml_extras/lib/libstdc++.so.6: version GLIBCXX_3.4.32' not found (required by ./model_benchmark) ./model_benchmark: /root/ryzen_ai-1.6.1/venv/lib/python3.10/site-packages/flexml/flexml_extras/lib/libstdc++.so.6: version CXXABI_1.3.15' not found (required by ./model_benchmark)
./model_benchmark: /root/ryzen_ai-1.6.1/venv/lib/python3.10/site-packages/flexml/flexml_extras/lib/libstdc++.so.6: version GLIBCXX_3.4.31' not found (required by ./model_benchmark) ./model_benchmark: /root/ryzen_ai-1.6.1/venv/lib/python3.10/site-packages/flexml/flexml_extras/lib/libstdc++.so.6: version GLIBCXX_3.4.32' not found (required by deployment/lib/libonnxruntime-genai.so)
./model_benchmark: /root/ryzen_ai-1.6.1/venv/lib/python3.10/site-packages/flexml/flexml_extras/lib/libstdc++.so.6: version GLIBCXX_3.4.31' not found (required by deployment/lib/libonnxruntime-genai.so) ./model_benchmark: /root/ryzen_ai-1.6.1/venv/lib/python3.10/site-packages/flexml/flexml_extras/lib/libstdc++.so.6: version CXXABI_1.3.15' not found (required by deployment/lib/libonnxruntime-genai.so)
./model_benchmark: /root/ryzen_ai-1.6.1/venv/lib/python3.10/site-packages/flexml/flexml_extras/lib/libstdc++.so.6: version GLIBCXX_3.4.32' not found (required by deployment/lib/libryzen_mm.so.1) ./model_benchmark: /root/ryzen_ai-1.6.1/venv/lib/python3.10/site-packages/flexml/flexml_extras/lib/libstdc++.so.6: version GLIBCXX_3.4.31' not found (required by deployment/lib/libryzen_mm.so.1)
./model_benchmark: /root/ryzen_ai-1.6.1/venv/lib/python3.10/site-packages/flexml/flexml_extras/lib/libstdc++.so.6: version `CXXABI_1.3.15' not found (required by deployment/lib/libryzen_mm.so.1)

I should have similar issues, but my Ubuntu skills are too poor to solve them. Could you please provide some solutions? Thank you！！！ @X-Ryl669

[X-Ryl669]

Can you use the backtick "`" operator for posting code, it's hard to read the way you did it?

I don't understand if your libstdc++ is either too old or too recent?

Please post the LD_LIBRARY_PATH you're using, and the version of your system libstdc++ (you can get the latter by: strings /lib/libstdc++.so | grep GLIBCXX | sort and find the highest number in the top of the list). Then do the same for the provided libstdc++ in the package. If your system lib is too old, you'd use the provided version, but it'll fail anyway since it's very likely other libraries in the program will load your system's libstdc++ first. In that case, you'll need to prefix your command with "LD_PRELOAD=/path/to/provided/libstdc++" so you force using the provided package version even before any system library is loaded.

Everything here is too fragile, it's a dependency mess.

There are only 2 possible solutions:

Static libraries (preferred) for their .so code

AMD should release its binary-only code as statically linked for the base libstdc++. That way, it can't interfere with the system's libstdc++. It's done by linking with a -static-libstdc++ -static-libgcc if I remember correctly. However, the drawback is that you can't load any std's C++ class dynamically from a .so in your code, since the layout will likely break.

Dynamic libraries

First state what is the minimum libstdc++ version that's required to run the code.

This is a huge maintenance work on their side, since they need to maintain a build with an antique system (with a very old libstdc++) so it fit with as much distribution as possible. This means rebuilding everything with that old version (and hope that it still work).

There are always woes with this issue as your are experiencing currently.

[wc2333]

Yes, it's my first time replying on GitHub, so there were some formatting issues. The current situation is that neither local nor Venv libstd c++works, only the error message is different. I have now abandoned this example on the official website (although soon) because I just discovered that Lemonade can be used and made some changes to the source code to support NPU LLM under Linux, which also works well.Thank you anyway！

[X-Ryl669]

I didn't know lemonade could use the NPU on linux (I haven't found how to do that). Would you mind explaining your changes ?

[duanearnett]

@wc2333 also curious as to how you were able to do that?

[wc2333]

Code Change Summary

File Path

src/lemonade/tools/oga/load.py

Specific Changes

diff --git a/src/lemonade/tools/oga/load.py b/src/lemonade/tools/oga/load.py
index aa4ac66..f30a2c3 100644
--- a/src/lemonade/tools/oga/load.py
+++ b/src/lemonade/tools/oga/load.py
@@ -341,7 +341,17 @@ class OgaLoad(FirstTool):
     """

     # For RyzenAI 1.6.0, check NPU driver version for NPU and hybrid devices
-    if device in ["npu", "hybrid"]:
+    if device not in ["npu", "hybrid"]:
+        return

+    if not sys.platform.startswith("win"):
+        printing.log_info(
+            f"[OGA] Skipping Windows-only NPU dependency checks "
+            f"for device={device} on platform={sys.platform}."
+        )
+        return
+
+    else:
         required_driver_version = REQUIRED_NPU_DRIVER_VERSION
         current_driver_version = _get_npu_driver_version()



These are the main parts, of course, these changes came from a friend of mine. Currently, only these (signed documents) have been obtained. I will try my best to practice them myself in the future (about a week). If it can be fully implemented, I will make a simple document and put it here

[X-Ryl669]

Please have a look at this PR for an updated lemonade server that runs model on the NPU.

In the description, I've explained the steps required to get everything working.

[lakshay048]

Hi @X-Ryl669

Thanks for the detailed feedback. We are reviewing it and surely get back to you shortly.