Adding CAAL API endpoint for headless usage

[AbdulShahzeb]

Hi
I was wondering if there's a way to set up an endpoint that lets me prompt CAAL and receive its responses (including tool calls, short term memory, and other features) without using the frontend, STT, or TTS. I had a crack at adding one but didn't get very far:

# webhooks.py
class QueryRequest(BaseModel):
    """Request body for /query endpoint."""
    
    message: str
    room_name: str = "voice_assistant_room"
    timeout: int = 60


class QueryResponse(BaseModel):
    """Response body for /query endpoint."""
    
    status: str
    response: str | None = None
    room_name: str


# voice_agent.py
elif action == "query":
    query_id = cmd.get("query_id")
    message = cmd.get("message", "")

    if message and query_id:
        logger.info(f"Processing query {query_id}: {message[:50]}...")

        try:
            from livekit.agents import llm

            # Create a fresh chat context for now (ideally retain existing context)
            chat_ctx = llm.ChatContext()
            chat_ctx.add_message(
                role="user",
                content=message,
            )

            # Call the LLM node
            response_text = ""

            async for chunk in assistant.llm_node(
                chat_ctx=chat_ctx,
                tools=None,                # fix
                model_settings=None,       # fix
            ):
                response_text += chunk

        except Exception as e:
            logger.error(f"Failed to process query {query_id}: {e}", exc_info=True)

Any tips or suggestions on implementing such a feature would be greatly appreciated!

Cheers,
Abdul

[cmac86]

Hey Abdul,

This could be useful. I just made a test script today to test fine-tuned models. It makes separate calls to Ollama API. It only tests the first tool call at this point but was thinking about how to test tool chaining. Connecting directly to CAAL could work, but then we're using production tools and DM'ing and emailing people through tests probably isn't a good thing.

Are you thinking about the same use case? Just piecing it together with your other discussion around fine tuning. We should collab on this. I'll message on the other thread regarding the web search training.

Corey

[AbdulShahzeb]

Hey Corey,

That makes sense, and I agree that directly testing against production tools probably isn’t ideal.

My use case is actually a bit different from testing fine-tunes. I’m not trying to benchmark models through CAAL. What I’d like is a way to run CAAL headlessly and expose it as an endpoint so I can use CAAL itself as a tool from another orchestrator.

My personal LLM (which I don’t want to replace) doesn’t support tool/function calling. So the idea is to “offload” tool-capable tasks to CAAL - essentially treating CAAL as a meta-agent that can handle tool use, chaining, memory, etc., and then return a final response. Kind of like a small council-of-agents setup that PewDiePie talked about.

That’s why I’m looking for a way to prompt CAAL without the frontend, STT, or TTS. I’d prefer not to load the speech models at all since they consume extra VRAM and aren’t needed for this use case.

I completely understand this probably isn’t a common workflow, which is why I didn’t open a formal issue. I’m happy to experiment on my own but I mostly just need guidance on the cleanest way to hook into the existing agent loop without breaking assumptions around rooms, contexts, or tool execution.

If you think there’s a clean architectural entry point for this, I’d appreciate a nudge in the right direction. Additionally, if you think the community would benefit from this feature, I'm happy to open an issue.

Abdul

[AbdulShahzeb]

Great success.

I ended up using FastAPI instead of LiveKit. My orchestrator now has access to caal-lite for tool use with n8n. 30G VRAM usage :)