ComfyUI native API integration with ComfyStream

[BuffMcBigHuge]

One of the primary limitations of building workflows within ComfyStream is the use of the hiddenswitch fork.

Many difficulties arise when particular node packs do not play well with the EmbeddedComfyClient. Careful testing and modifications to existing nodes is usually required to enable full functionality. Nonetheless, there are usually other issues that may come from dependency on the fork, such as delays or limitations with newer ComfyUI features, including native performance optimizations.

I set out to replace the EmbeddedComfyClient with communication directly to a running ComfyUI instance through a native API and Websocket connection.

Sending Data: All data can be sent via REST POST /prompt through the Comfy API. Currently, the method to pass incoming frames is done via the comfyui-tooling-nodes, with ETN_LoadImageBase64, where the input frame is passed as base64 data in the prompt request itself. This node will be migrated to the ComfyStream /nodes pack.

Receiving Data: Message events from the webhook are parsed, and data can be received and added to the image_outputs tensor array for WebRTC frames. The comfyui-tooling-nodes includes ETN_SendImageWebSocket which includes a Blob format with prefix similar to how Comfy sends previews to the UI. Upon successfully capturing the Blob, the prompt can then be called for the next subsequent frame.

It is obvious that this process is not as efficient as the hiddenswitch method of communicating with the ComfyStream tensor_cache directly, however it opens up new opportunities for parallelization through multi-inference gpu scaling as well as multi-gpu scaling, an avenue I'm investigating as a performance increase.

Note that this preliminary DRAFT is very early and the proof of concept was just demonstrated as functional. More work is to be done.

• Test a myriad of existing workflows used with ComfyStream
• Reduce CPU usage of tensor manipulations to WebRTC
• Migrate input and output nodes to /nodes pack, and add auto-replacement
• Investigate alternative lightweight strategies for transferring data to/from Comfy API
• Test and optimize and compare to existing EmbeddedComfyClient method
• Auto start ComfyUI workspaces, and test multi-inference on a single GPU and build a frame buffer handler to reduce jitter and guarantee frame timing delivery
• Deployment planning

[BuffMcBigHuge]

TEST_00002.mp4

EmbeddedComfyClient is slightly faster at the moment. Optimizations still to be developed.

[BuffMcBigHuge]

Closed in lieu of livepeer#59.