Best way to get remote imgui for python?

[taylorh140]

So, I have a imgui_bundle python application. And I would like to be able to run it remotely. in my use case I just have one instance that i need to worry about, so shared input would be fine. I looked over the imgui_bundle and saw some things that looked closeish, but not quite there. eg the stuff here:

imgui_bundle/bindings/imgui_bundle/hello_imgui.pyi

Line 2415 in f1f7827

class RemoteParams:

But I'm not sure how it is supposed to be activated.

There is also the pyodide backend that can now run on the web which makes me wonder if it would be trivial to have a websocket share the drawlist/UserInput using the same version of imgui for max compatibility.

If there is a way to get this working in a stable manner it would be nice to know. I have reviewed some of the repos for remoting, but they seem pretty tied to the c++ implementation, and perhaps there is a way to jump that hurdle, but I'm not sure what it is.

[pthom]

Hum, remoting is a complex subject. It only works in C++ at the moment, and its support is experimental within Hello ImGui, even in C++.

[taylorh140]

Well, I'm going to try and make it work, at least get a feel for how much work it would be.

1. I'm going for the easiest solution possible. So, no data compression or anything.
2. I'm not making a custom renderer, or anything fancy.
3. I'm assuming that the draw-list is the same regardless of the renderer used.
4. The goal is to simply transfer the draw-list over a web-socket for starters.
5. I assume the wasm version can handle the same draw-list as the desktop version.
6. I assume that the extern C/C++ structure can be iterated via python, to extract the draw-list (otherwise how would a pure python render work)

And yes, this is not simple, I guess I just wanted to ask someone if it was even feasible, aka a good general approach or not.

[taylorh140]

Using the transfer of stuff seems to show at least some promise. I need to align/transfer the textures somehow but check it :

pretty nice right.

[taylorh140]

This is all python with a bit of js as a glue

[pthom]

Well done!

I had explored remoting inside helloimgui, prior to having a pyodide version.
The use case you are using is close to what imgui-ws provides : see https://github.com/ggerganov/imgui-ws

I had worked on providing a way to use imgui-ws from hello imgui. See
https://github.com/pthom/hello_imgui/blob/902dd7d8229b0f396a7f5d4e0a61486825b69c4a/src/hello_imgui/internal/backend_impls/remote_display_handler.cpp#L219
However it only works in C++ (no python bindings) and to be honest it is not considered ready for production. I stopped working on it when I saw that the pyodide version was going to work.

However I am interested in your use case and in seeing your progress, since it might be useful when heavy competition is required on the server side.

Transfering the font texture needs to be done at the backend level I’d say and it should probably follow the new texture API introduced in ImGui V 1.92

Can you tell me more about the stack you are using?

• I guess it is using python on the server side. Are you sure using hello imgui or immapp or are you using a pure python backend?
• and on the client side, are you rendering using js, or are you rendering using ImGui bundle via pyodide ?

Thanks

[taylorh140]

it's pretty basic at the moment:

server.py =execute with=> uv run server.py

#!/usr/bin/env uv run
# /// script
# requires-python = ">=3.13"
# dependencies = [
#     "aiohttp",
#     "imgui_bundle",
#     "msgspec",
#     "janus"
# ]
# ///



from serialize import PyImDrawData,replay_draw_data

import asyncio
import janus
from aiohttp import web
from imgui_bundle import imgui, immapp, hello_imgui

dQ = janus.Queue()

# ---------- WebSocket handler ----------
async def websocket_handler(request):
    ws = web.WebSocketResponse()
    await ws.prepare(request)

    
    while True:
        mybytes = await dQ.async_q.get()
        await ws.send_bytes(mybytes,compress=None)
        #await asyncio.sleep(0.1)  # send every second

    return ws

# ---------- HTTP server serving index.html ----------
async def index(request):
    return web.FileResponse("index.html")

# ---------- Build and run app ----------
app = web.Application()
app.router.add_get("/", index)
app.router.add_get("/ws", websocket_handler)

# Serve index.html and allow file access
app.router.add_static("/", ".")
app.router.add_resource("/serialize.py")

# ---------- Gui stuff ------------
from imgui_bundle import implot, imgui, immapp, hello_imgui, icons_fontawesome_4
import time
import numpy as np

# Fill x and y whose plot is a heart
vals = np.arange(0, np.pi * 2, 0.01)
x = np.power(np.sin(vals), 3) * 16
y = 13 * np.cos(vals) - 5 * np.cos(2 * vals) - 2 * np.cos(3 * vals) - np.cos(4 * vals)
# Heart pulse rate and time tracking
phase = 0.0
t0 = time.time() + 0.2
heart_pulse_rate = 80


def gui():
    global heart_pulse_rate, phase, t0, x, y

    imgui.text("Made with " + icons_fontawesome_4.ICON_FA_HEART + " using Dear ImGui and ImPlot")
    imgui.text(f"Running at {hello_imgui.frame_rate():.1f} FPS")

    t = time.time()
    phase += (t - t0) * heart_pulse_rate / (np.pi * 2)
    k = 0.8 + 0.1 * np.cos(phase)
    t0 = t

    implot.begin_plot("Heart", immapp.em_to_vec2(21, 21))
    implot.plot_line("", x * k, y * k)
    implot.end_plot()

    imgui.set_next_item_width(hello_imgui.em_size(10))
    _, heart_pulse_rate = imgui.slider_float("Pulse", heart_pulse_rate, 30, 180)



def pre_imgui_callback():
    dd = imgui.get_draw_data()
    if(dQ.sync_q.empty()):
        data = PyImDrawData.from_imgui(dd).to_bytes()
        print(f"FILLED {len(data)} bytes")
        dQ.sync_q.put_nowait(data)


callbacks = hello_imgui.RunnerCallbacks(
    show_gui = gui,
    after_swap =  pre_imgui_callback,
)

runner = immapp.RunnerParams(
    callbacks=callbacks,
)

addons = immapp.AddOnsParams(
    with_implot=True,
)


# ---------- Run the stuff -------------
taskA = asyncio.to_thread(
    immapp.run,
    runner_params=runner,
    add_ons_params = addons,
    
)
taskB = asyncio.to_thread(
    web.run_app,app, host="0.0.0.0", port=8000
)

async def main():
    async with asyncio.TaskGroup() as tg:
        tg.create_task(taskA)
        # tg.create_task(taskB)

if __name__ == '__main__':
    asyncio.run(main())

Index.html:

<!doctype html>
<html>
    <title>PythonWebSocketTest</title>
<head>
    <meta charset="UTF-8" />
    <style>
        html, body { width: 100%; height: 100%; margin: 0; }
        #canvas { width: 100%; height: 100%; display: block; }
    </style>
    <script src="https://cdn.jsdelivr.net/pyodide/v0.29.0/full/pyodide.js"></script>
</head>

<body>
<canvas id="canvas"></canvas>

<script type="text/javascript">




// ====================== Python code ============================
let pythonCode = `


#################################################
from imgui_bundle import implot, imgui, immapp, hello_imgui
import numpy as np


def gui():
    global In_Q
    if(len(In_Q)!=0):
        value = In_Q[-1]
        In_Q = In_Q[-1:]
        data = PyImDrawData.from_bytes(bytes(value))
        replay_draw_data(data)


if __name__ == "__main__":
    immapp.run(gui, window_size=(300, 200), window_title="WS Demo", with_implot=False, fps_idle=60)
`;

// ====================== MAIN JS START ==========================
async function main() {
    const pyfile = await fetch('/serialize.py');
    const pytxt = await pyfile.text();

    pythonCode = pytxt + pythonCode;

    document.addEventListener('contextmenu', event => event.preventDefault());

    let pyodide = await loadPyodide();
    pyodide.globals.set("In_Q", []);


    await pyodide.loadPackage("msgspec");
    await pyodide.loadPackage("imgui_bundle");

    // Setup WebGL canvas
    let canvas = document.getElementById("canvas");
    pyodide.canvas.setCanvas2D(canvas);

    // Expose ws_counter to Python

    // ---------- WebSocket setup ----------
    let ws = new WebSocket(`ws://${window.location.host}/ws`);

    ws.binaryType = "arraybuffer"; // Tell the web socket that we have bytes

    ws.onmessage = (event) => {
        const in_Q = pyodide.globals.get("In_Q");
        const array = new Uint8Array(event.data);
        console.log("gotNew");
        in_Q.push(array);
    };

    ws.onopen = () => console.log("WebSocket connected.");
    ws.onclose = () => console.log("WebSocket closed.");
    ws.onerror = (e) => console.log("WebSocket error:", e);

    // Run Python app
    await pyodide.runPythonAsync(pythonCode);
}

main();
// ======================= MAIN JS END ===========================

</script>
</body>
</html>

And a poorly slubbed together glue:

serialize.py

from __future__ import annotations
from typing import List, Tuple
import msgspec
from imgui_bundle import imgui

# ------------------------------------------------------------------
# The One True Class — clean, fast, websocket-ready
# ------------------------------------------------------------------
class PyImDrawData(msgspec.Struct, array_like=True):
    total_vtx_count: int
    total_idx_count: int
    cmd_lists_count: int

    # Nested structs — msgspec handles deep encoding automatically
    class Vert(msgspec.Struct, array_like=True):
        pos_x: float
        pos_y: float
        uv_x:  float
        uv_y:  float
        col:   int       # ARGB u32

    class Cmd(msgspec.Struct, array_like=True):
        elem_count: int
        clip_x1: float
        clip_y1: float
        clip_x2: float
        clip_y2: float
        texture_id: int  # u32

    class imList(msgspec.Struct, array_like=True):
        vertices:  List[PyImDrawData.Vert]
        indices:   List[int]      # auto 16/32-bit in real renderer, but we store as int
        commands:  List[PyImDrawData.Cmd]

    lists: List[imList]

    # ------------------------------------------------------------------
    # Classmethods — clean API
    # ------------------------------------------------------------------
    @classmethod
    def from_imgui(cls, dd: imgui.ImDrawData) -> "PyImDrawData":
        """Convert real ImDrawData → pure Python msgspec-ready structure"""
        lists = []
        for draw_list in dd.cmd_lists:
            # Vertices
            verts = []
            for v in draw_list.vtx_buffer:
                pos = v.pos
                uv = v.uv
                verts.append(cls.Vert(
                    pos_x=float(pos.x),
                    pos_y=float(pos.y),
                    uv_x=float(uv.x),
                    uv_y=float(uv.y),
                    col=int(v.col) & 0xFFFFFFFF
                ))

            # Indices
            indices = [int(i) for i in draw_list.idx_buffer]

            # Commands
            cmds = []
            for c in draw_list.cmd_buffer:
                clip = c.clip_rect
                cmds.append(cls.Cmd(
                    elem_count=int(c.elem_count),
                    clip_x1=float(clip.x),
                    clip_y1=float(clip.y),
                    clip_x2=float(clip.z),
                    clip_y2=float(clip.w),
                    texture_id=int(c.get_tex_id())
                ))

            lists.append(cls.imList(verts, indices, cmds))

        return cls(
            total_vtx_count=getattr(dd, "total_vtx_count", dd.total_vtx_count),
            total_idx_count=getattr(dd, "total_idx_count", dd.total_idx_count),
            cmd_lists_count=len(dd.cmd_lists),
            lists=lists
        )

    def to_bytes(self) -> bytes:
        """Ultra-fast compact binary (better than msgpack in practice due to array_like)"""
        return msgspec.msgpack.encode(self)

    @classmethod
    def from_bytes(cls, data: bytes) -> "PyImDrawData":
        """Deserialize from wire"""
        return msgspec.msgpack.decode(data, type=cls)

    # Optional: extract all used texture IDs
    def used_texture_ids(self) -> set[int]:
        ids = set()
        for lst in self.lists:
            for cmd in lst.commands:
                if cmd.texture_id:
                    ids.add(cmd.texture_id)
        return ids
    

def replay_draw_data(pyd: PyImDrawData, texture_map: dict[int, int] | None = None):
    if texture_map is None:
        texture_map = {}
    draw_list = imgui.get_window_draw_list()

    for lst in pyd.lists:
        verts = lst.vertices
        idxs = lst.indices
        idx_offset = 0

        for cmd in lst.commands:
            # THIS IS THE IMPORTANT PART — bind the correct texture!
            tex_id = texture_map.get(cmd.texture_id, imgui.ImTextureRef(imgui.get_io().fonts.python_get_texture_id()))

            draw_list.push_texture(tex_id)   # ← This sets the active texture

            # Clip rect
            draw_list.push_clip_rect((cmd.clip_x1, cmd.clip_y1), (cmd.clip_x2, cmd.clip_y2),True)

            # Draw all triangles in this command
            tri_count = cmd.elem_count // 3
            for _ in range(tri_count):
                if idx_offset + 2 >= len(idxs):
                    break
                i0, i1, i2 = idxs[idx_offset:idx_offset + 3]
                idx_offset += 3

                v0 = verts[i0]
                v1 = verts[i1]
                v2 = verts[i2]

                draw_list.add_triangle_filled( #probably wrong
                    (v0.pos_x, v0.pos_y),
                    (v1.pos_x, v1.pos_y),
                    (v2.pos_x, v2.pos_y),
                    v0.col
                )

            draw_list.pop_clip_rect()
            draw_list.pop_texture()  #  restore previous texture?

What I like about it is how little total code there is here. It's not even close to correct, I need to figure out how to get the cmd stuff to work properly. But it is pretty fun.

[taylorh140]

I suppose if i had to describe it. I would say its a mixed (http server/imgui bundle) program for the server
the server "copies" the drawstate and serializes it, then sends it over a websocket.

The client then deserializes the draw commands and feeds them to the pyoidide imgui.

[taylorh140]

I also have to Admit, I'm very unfamiliar with how the inner workings of imgui do.. stuff. So, please forgive the handling of things so recklessly. Compared to this ^ the imgui-js is quite highly developed.