Openrouter Llama Models Aren't Calling the Execute Tools

[MFA-X-AI]

Hi! I'm investigating agentic behavior across models. Right now I'm trying out llama models. I'm observing that most, if not all, of the llama models aren't able to call the execute tools.

Specifically, I've tested out:

• meta-llama/llama-3.3-70b-instruct
• meta-llama/llama-3.1-70b-instruct
• meta-llama/llama-3.1-8b-instruct

and they all have the same behavior of:

curl -X POST http://127.0.0.1:9001/openai/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "dai-agent",
    "messages": [
      {"role": "user", "content": "Give me the fully diacritized version of he arabic text of the first verse of the first surah."}
    ]
  }'

{"id":"chatcmpl-8a59b034e01f4cb08863aecd2760e722","object":"chat.completion","created":1750932624,"choices":[{"index":0,"message":{"role":"assistant","content":""},"finish_reason":"stop"}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

I've been using the default prompt template, You are D'AI, a helpful Islamic assistant with access to a variety of tools. Could it be a prompt issue?

For some models, such as

• meta-llama/llama-3.2-3b-instruct

I am also seeing it stop at list_model

And it times out:

I've done my due diligence on testing it using a request api directly, and I can see it respond.

If you compare it with gpt-4.1-nano:

curl -X POST http://127.0.0.1:9001/openai/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "dai-agent",
    "messages": [
      {"role": "user", "content": "Give me the fully diacritized version of he arabic text of the first verse of the first surah."}
    ]
  }'
{"id":"chatcmpl-9ec4c41f2fab4c67b3333ce40e542fc8","object":"chat.completion","created":1750932880,"choices":[{"index":0,"message":{"role":"assistant","content":"The fully diacritized version of the first verse of the first surah (Al-Fatiha) is: بسم الله الرحمن الرحيم"},"finish_reason":"stop"}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

Surprisingly meta-llama/llama-3-8b-instruct works very well out of the box. :D

Note:
One variable to note is that OpenRouter has various providers... the issue that I'm seeing where there's no response might be just that. But it doesn't explain why I can get responses on a python hello world and not on xaibo.