Skip to content

Commit 4c2dd2d

Browse files
tomicetomice
committed
Add Five Years and No Solution
1 parent 74aa7db commit 4c2dd2d

1 file changed

Lines changed: 315 additions & 0 deletions

File tree

‎_posts/2025-06-08-five-years-and-no-solution.md‎

Lines changed: 315 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,315 @@
1+
---
2+
title: "Five Years and No Solution"
3+
description: Revisiting an old forum post
4+
date: 2025-06-08 00:00:00 -0400
5+
categories: [Programming, Virtualization, ARM, Xilinx, AMD, AI]
6+
tags: [programming, virtualization, compilers, ai, forums]
7+
---
8+
9+
While logging into work early in the morning last week, I came across an
10+
interesting email that hit my inbox from the AMD SoC/FPGA support forum.
11+
In the email was someone asking if I had ever solved an issue that I ran into
12+
about five years ago. Quick spoiler alert - I did not. After resetting my
13+
password after long forgetting it, I jumped in to reply stating that I,
14+
unfortunately, did not ever figure out a true solution to the problem, but I
15+
did extend some advice I learned as I was debugging the issue.
16+
17+
For anyone curious, you can see the issue
18+
[here](https://adaptivesupport.amd.com/s/question/0D52E00006hpX90SAE/mouse-and-keyboard-inputs-in-qemu-with-zynqmpsoc-zcu106?language=en_US).
19+
20+
In today's post, we'll be diving into this issue a bit more, setting up the
21+
problem space (as best as I can talk about it given that it is a work-related
22+
topic, after all), and peering into what the future might look like as the
23+
world rapidly shifts thanks to our friendly new Clippy-esque companion - AI.
24+
25+
## The Task and Background
26+
27+
The company I work for specializes in building out software solutions for
28+
embedded platforms, mostly focusing on the defense side of the house. Our
29+
main specialty is building custom Linux-based operating systems for embedded
30+
devices. This is probably of no surprise if you have read my previous blog
31+
posts where we dive into the internals of Yocto and how it works. Essentially,
32+
we do the same thing for major corporations that want to fully control their
33+
operating system and the experience the end-user receives.
34+
35+
While working with a customer, we were tasked to build out a custom variant of
36+
AMD's
37+
[PetaLinux](https://www.amd.com/en/products/software/adaptive-socs-and-fpgas/embedded-software/petalinux-sdk.html).
38+
This is a front end that AMD (previously Xilinx) created to help make working
39+
with Yocto-based systems easier on the developer who might not be as
40+
well-versed in the Yocto ecosystem as a normal Linux BSP developer would.
41+
PetaLinux is nothing more than a simple front end to managing a Yocto-built
42+
operating system for whatever AMD/Xilinx BSP you happen to be targeting.
43+
44+
Not all of our developers have access to secure labs where we have target
45+
hardware readily available for testing. Those of us who are working remote also
46+
need to be able to write software against this, so we needed to come up with a
47+
solution for those devs (aka me) so we can all work on this task of building up
48+
a custom OS for the customer together. The solution comes in the form of our
49+
emulation friend [QEMU](https://www.qemu.org/).
50+
51+
### QEMU Intro
52+
53+
QEMU (Quick EMUlator) is an open-source machine emulator and virtualizer that
54+
has become the go to standard of the virtualization and embedded development
55+
world. It was originally written by the famous French programmer
56+
[Fabrice Bellard](https://bellard.org/) in 2003. If you haven't heard of him, I
57+
highly recommend reading what he has worked on because there's a good chance
58+
you have either used his tech, or used a program that leveraged software he
59+
invented. Thanks to its flexibility, speed, and ease of use, QEMU quickly
60+
gained popularity and started obtaining broad hardware support, making it the
61+
de facto standard for virtualization.
62+
63+
At its core, QEMU allows users to emulate various hardware platforms. This is
64+
everything that a board manufacturer might happen to support, from x86 and ARM,
65+
to PowerPC, MIPS, and more. This makes it possible to run operating systems and
66+
applications designed for one architecture on a completely different one.
67+
68+
You will find QEMU used all over the place, too. For example, it is used as the
69+
underlying engine for numerous higher-level virtualization tools, including
70+
[KVM](https://www.redhat.com/en/topics/virtualization/what-is-KVM),
71+
[libvirt](https://libvirt.org/), and a plethora of cloud infrastructure
72+
platforms. It's also the go-to tool for embedded system developers, CI/CD
73+
testing pipelines, operating system maintainers, and anyone needing to simulate
74+
hardware they don't physically own.
75+
76+
As you probably guessed, this means QEMU is the perfect solution to our problem
77+
that we were facing at our company. So what went wrong? Before getting into that,
78+
I first should showcase common uses and the problem I was running into. So
79+
first, let's take a mini lesson on how we would use QEMU in a practical manner.
80+
81+
### QEMU Mini Lesson
82+
83+
QEMU has two main modes:
84+
85+
* Full system emulation: QEMU can simulate a complete machine, including CPU,
86+
memory, disk, and network interfaces. This enables
87+
running entire guest operating systems in a sandboxed
88+
environment.
89+
90+
* User-mode emulation: QEMU can execute single programs compiled for one
91+
architecture on another, translating system calls and
92+
CPU instructions as needed.
93+
94+
For our use case, we will be focusing on full system emulation because we want
95+
to be able to emulate a piece of hardware that not all of us have access to.
96+
We are, after all, wanting to be able to load our custom Linux on it and take
97+
control of everything.
98+
99+
First, you want to specify what kind of hardware you want to emulate. This is
100+
more than just the CPU architecture or a simple distro select menu that you
101+
tend to see on most virtualization platforms. We need to be explicit about RAM,
102+
storage, USB devices, and so on. Each one of these is defined by passing
103+
options to QEMU via the command line.
104+
105+
Basic Command Structure:
106+
107+
```sh
108+
qemu-system-ARCH [machine/board options] \
109+
[CPU/ram/dtb options] \
110+
[kernel/u-boot/image options] \
111+
[device options] \
112+
[drive/storage/network options] \
113+
[other flags]
114+
```
115+
116+
* `qemu-system-ARCH` - Specifies the architecture to emulate
117+
(e.g., `qemu-system-x86_64` for a 64-bit PC,
118+
`qemu-system-aarch64` for ARM64, etc.)
119+
120+
* Machine/board options - Tell QEMU what kind of board or machine to emulate
121+
122+
* CPU/ram/dtb options - Lets you choose CPU type, RAM amount, and provide a
123+
device tree, if applicable
124+
125+
* Kernel/U-Boot/image options - Point QEMU at the software images you want to
126+
run (e.g., kernel, device loader, file, etc.)
127+
128+
* Device options - Add devices like USB, network cards, or display hardware
129+
130+
* Drive/storage/network options - Control how disks and network interfaces
131+
should be configured for the board
132+
133+
* Other flags options - Things like display settings, GDB debugging, etc.
134+
135+
### ARM Mini Lesson
136+
137+
ARM (Advanced RISC Machine) processors are at the heart of most modern embedded
138+
systems and SoCs. Unlike x86 processors found in most desktops and laptops, ARM
139+
CPUs are designed with efficiency and low power consumption in mind, making them
140+
ideal for everything from smartphones to FPGAs and development boards. Recently,
141+
they have gotten so powerful that major manufacturers like Apple are even using
142+
them for their desktops and laptops.
143+
144+
In the Xilinx world, the Zynq UltraScale+ MPSoC (Multi-Processor
145+
System-on-Chip) family combines programmable logic with powerful ARM Cortex-A53
146+
application processors. Developing for these chips often means targeting
147+
unique hardware setups, custom peripherals, and deeply tuned Linux environments
148+
to propagate all these features to userland.
149+
150+
Now that we know roughly how QEMU works, let's walk through a stripped-down
151+
version of the kind of command you might use to bring up an ARM-based board:
152+
153+
```sh
154+
qemu-system-aarch64 \
155+
-M virt \
156+
-cpu cortex-a53 \
157+
-m 2048 \
158+
-kernel Image \
159+
-dtb board.dtb \
160+
-drive file=rootfs.img,if=none,format=raw,id=hd0 \
161+
-device virtio-blk-device,drive=hd0 \
162+
-append "root=/dev/vda console=ttyAMA0" \
163+
-serial mon:stdio \
164+
-nographic
165+
```
166+
167+
Explanation of commands:
168+
169+
* `-M virt` - Tells QEMU to use a generic ARM "virt" board
170+
171+
* `-cpu cortex-a53` - Sets the CPU model, matching real-world ARM equivalent
172+
173+
* `-m 2048` - Gives the guest 2GB of RAM to work with
174+
175+
* `-kernel Image` - Loads the Linux kernel image
176+
177+
* `-dtb board.dtb` - Loads a device tree describing the virtual hardware
178+
179+
* `-drive file=rootfs.img,if=none,format=raw,id=hd0` - Connect a raw root
180+
filesystem image to the
181+
VM at hd0
182+
183+
* `-append "root=/dev/vda console=ttyAMA0"` - Passes kernel command-line
184+
args telling Linux where its
185+
root filesystem is and which
186+
console to use
187+
188+
* `-serial mon:stdio` - Ties the guest's serial console to your terminal
189+
190+
* `-nographic` - Disables the graphical window so everything is in the terminal
191+
192+
## The Problem
193+
194+
At my company, we wanted to run and interact with our custom Linux images for
195+
the ZynqMP ZCU106 board using QEMU, allowing multiple developers to test and
196+
validate their work without requiring access to hardware. The idea was simple:
197+
boot our custom Linux in QEMU, and use it just like the real thing, including
198+
interacting with the VM using a mouse and keyboard, just as we would with a
199+
real development board that had a keyboard, monitor, mouse, etc.
200+
201+
However, it wasn't quite that simple...
202+
203+
Despite QEMU's reputation for broad hardware support, I hit a wall when it came
204+
to accessing peripherals in the QEMU environment. No matter what options or
205+
device flags I used, QEMU refused to recognize any mouse or keyboard input.
206+
I tried a variety of USB controller emulations (`qemu-xhci`, `usb-ehci`,
207+
`usb-uhci`, and more), as well as different HID device options, cycling through
208+
every reasonable combination the documentation recommended. I swapped out real
209+
hardware, thinking maybe it might have been my devices themselves. Different
210+
mice, keyboards, USB ports, laptops were all tested. Still, QEMU's guest OS sat
211+
there never registering any of my USB devices.
212+
213+
For my specific use case, it wasn't a true showstopper. I could offload any of
214+
that testing to my wonderful and very patient on-site tester. Still, it was
215+
frustrating having to essentially throw test code over the wall to someone else
216+
when QEMU should be able to handle such a simple use case.
217+
218+
One subtle wrinkle in embedded development is that companies like Xilinx often
219+
maintain their own vendor forks of tools like QEMU. These forks include custom
220+
patches, board support, and device models that are not yet available (or
221+
sometimes never make it) in the official, "upstream" QEMU project.
222+
223+
In my case, the official upstream QEMU had very limited support for the Zynq
224+
UltraScale+ MPSoC family that I was targeting. To emulate the board's unique
225+
hardware and boot flows, you were required to use the version of QEMU provided
226+
by Xilinx—often distributed as part of the PetaLinux SDK.
227+
228+
Here's the catch:
229+
230+
The Xilinx fork of QEMU lagged behind the upstream project in features, device
231+
emulation, and bug fixes. Some features that worked in mainstream QEMU, such as
232+
certain USB controllers or advanced input options, simply weren't present
233+
enabled, or just didn't work as expected in the vendor fork.
234+
235+
Documentation from Xilinx sometimes referenced features that didn't actually
236+
exist in the shipped binaries, or the functionality wasn't working as expected.
237+
238+
On top of that, if you strayed from the "golden path" like we were, you were
239+
pretty much on your own.
240+
241+
## The Solution
242+
243+
The truth is, I never fully solved the problem; we only had a workaround. After
244+
years of silence, when someone finally reached out to resurrect my old support
245+
thread, I realized just how few details I had remembered about the scenario,
246+
as well as what exactly went down to workaround this. Five years is a long time
247+
in tech, and in the case of Xilinx's QEMU fork, things may have shifted under
248+
the hood more than once since then. Hell, Xilinx was bought out by AMD after I
249+
reached out for support in the forums.
250+
251+
To help out the fellow developer also struggling with the same problem, I threw
252+
in my two cents, hoping it would lead to some sort of avenue to help them solve
253+
their problem. It was generic advice, but it boiled down to:
254+
255+
* Double-check your device tree: Any USB controllers or input devices you
256+
configure in QEMU must actually exist in your device tree. If you're using a
257+
`.dtb` file, convert it to a `.dts` with `dtc` and look for xhci or usb nodes.
258+
259+
* Be explicit with device mappings: If you're getting errors about missing USB
260+
buses, try explicitly assigning the device to a known bus. For example:
261+
`-device usb-mouse,bus=xhci.0`
262+
263+
* Use the QEMU monitor: The built-in QEMU monitor can help you inspect what
264+
devices are visible to the emulated guest which is useful for debugging why
265+
input isn't working. Attach it with `-serial mon:stdio` (as shown above) and
266+
issue monitor commands.
267+
268+
* Verify kernel configuration: Sometimes it's not QEMU at all. Make sure your
269+
guest kernel is configured with support for the right USB controllers and HID
270+
devices.
271+
272+
## The Future
273+
274+
This got me thinking about today's world where everything is shifting to AI.
275+
When I wrote this originally, there were no public AI models for anybody to
276+
use. It was pre-AI revolution. Stack Overflow was still alive and well.
277+
Google still worked (mostly) as expected. You could find solutions on the
278+
Internet via forums, mailing lists, and more.
279+
280+
This is rapidly changing.
281+
282+
Today, people have AI to turn to. It's a void you type into, attempt to solve
283+
problems with, and hope whatever it returns back to you works. It's an endless
284+
feedback loop where you never know if you're going to get a correct answer.
285+
Sure, you'll get an answer, but it'll be difficult to verify if it's correct
286+
or not.
287+
288+
However, there's an even more interesting aspect about this that doesn't seem
289+
to be discussed: the community banding together to solve issues. If
290+
questions are never posted publicly, people might not know others are going
291+
through similar issues, causing them to pursue a path that will surely lead
292+
them to a roadblock. Manufacturers might not know there are issues with
293+
their custom implementations. After all, many bug reports out there come about
294+
from users attempting to utilize the components and them not behaving as
295+
expected. In short, people can't learn from other real people.
296+
297+
Forums are already in decline thanks to closed gardens like Discord, Facebook,
298+
and more. It makes me wonder what the future will look like when everyone has
299+
their own little AI version of Clippy, and nobody is reaching out to others to
300+
see if a solution is readily in front of them just a simple search away...
301+
302+
I'm sure someone could come up with scenarios where such support groups are no
303+
longer needed, as well as counter arguments for everything stated above, but
304+
the old grey beard in me remembers what life was like in the past and worries
305+
what life might be like in the future...
306+
307+
## Further Reading
308+
309+
Some useful links on virtualization and emulation:
310+
311+
* [QEMU Master Documentation](https://www.qemu.org/docs/master/)
312+
* [Configuring and Managing Virtualization](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/configuring_and_managing_virtualization/index)
313+
* [libvirt - Ubuntu Server Documentation](https://documentation.ubuntu.com/server/how-to/virtualisation/libvirt/index.html)
314+
315+
Happy virtualizing!

0 commit comments

Comments
 (0)