Does not use native support for function/tool calling #9
Description
The function calls do not pass in the tools using the tools and tool_choice params and thus we are testing tool calling using only model's normal text generation capability and not the native support for function/tool calling that a lot of models now have.
For example the Berkeley Function-Calling Leaderboard enables both evaluating native support FC and prompt based.
It would be good to have a version of Acebench that implements FC's in the following manner:
tools = [
{
"type": "function",
"function": {
"name": "get_current_weather",
# ...
}
},
{
"type": "function",
"function": {
"name": "get_current_stock_price",
# ...
}
}
]
response = client.chat.completions.create(
model="Qwen/Qwen2.5-7B-Instruct-Turbo",
messages=[
{"role": "user", "content": "What's the current price of Apple's stock?"},
],
tools=tools,
tool_choice={"type": "function", "function": {"name": "get_current_stock_price"}}
)