Video and photo multimodality

[AMaestroG]

I'm really excited about cannoli, it's something spectacular and incredible, I'm just waiting for the multimodality with it, to be able to put an arrow and put the image or video into the flow.

[blindmansion]

That would be super cool, it's been on my mind. No ETA on it yet, not sure how much work it would take.

There is a workaround for image generation at least, using the http nodes/templates. Here's how someone set it up, you might have to check if the API still looks like this:
#16 (comment)

[ttodosi]

ya i'd like to send to comfyui, will try to get a setup going this week

[ttodosi]

still failing at cannoli college, but I'm getting closer. not sure how the image would come back

[ttodosi]

https://github.com/comfyanonymous/ComfyUI/blob/master/script_examples/basic_api_example.py
endpoints:
@routes.get('/ws') ⇒ Returns the WebSocket object, sends status and executing messages
@routes.post("/prompt") ⇒ Queues prompt to workflow, returns prompt_id or error
@routes.get("/history/{prompt_id}") ⇒ Returns the queue or output for the given prompt_id
@routes.get("/view") ⇒ Returns an Image given a filename, subfolder, and type ("input", "output", "temp")
@routes.post("/upload/image")⇒ Uploads an image to ComfyUI, given image_data and type ("input", "output", "temp")

[blindmansion]

Give this one a try:

[Make image]
{
"url": "http://127.0.0.1:8188/prompt",
"method": "POST",
"headers": {
"Content-Type": "application/json"
},
"bodyTemplate": {
"prompt": {
"3": {
"inputs": {
"seed": 5
}
},
"6": {
"inputs": {
"text": "{{prompt}}"
}
}
}
}
}

[ttodosi]

no dice yet, thanks for looking at it

[AMaestroG]

I loved it, but I'm afraid the translator has not expressed my idea correctly, which I would like, because the confusion thing has blown my mind, but I would like to put an image in cannoli and have GPT4o or Ollama read it to me or interpret it with an arrow and a purple box. In this way give vision to Cannoli, Video, audio, image.

[blindmansion]

Yep, the vision aspect isn't currently implemented, but we'd love to have it too. No ETA on it yet but i'll let you know if we implement that!