GitHub – MikeVeerman/tool-calling-benchmark

Can a $1,000 laptop run an AI agent that knows when to use tools — and when not to?

I tested 11 small open-weight models locally on CPU to see which ones can act — and which ones know when not to. No cloud API. No GPU. Just Ollama, a handful of 1-bit and 4-bit quantised models, and a Framework 13 running Arch Linux.

The motivation is practical. Local and private AI agents are increasingly attractive — no per-token costs, no data leaving the machine, no vendor lock-in. But an agent that acts incorrectly is worse than one that does nothing: a wrong API call costs money, sends the wrong message, or deletes the wrong file. The hard problem isn’t generating well-formed JSON. It’s deciding whether to act at all.

This benchmark measures judgment — whether a model knows when to call a tool — not just execution — whether it can format a tool call correctly.

• Every model that successfully emitted tool calls (8 of 11) can handle simple, unambiguous tool calls on CPU at 1-8s latency.
• When prompts require judgment — resisting keyword triggers, respecting negation, noticing redundant information — most sub-4B models fail.
• The two top-scoring models (qwen2.5:1.5b, ministral-3:3b) won by declining to act when uncertain, not by calling more tools.
• A 1.5B model outscored its 3B sibling from the same family. Under safety-weighted scoring, conservatism beat aggression.
• Microsoft’s 1.58-bit BitNet model produces flawless JSON and is the only model to handle multi-tool requests — but its tool selection judgment is poor.
• Five of eight functional models reflexively called get_weather when they saw the word “weather,” even when explicitly told not to.
• No sub-4B model reliably handled all three judgment dimensions tested: keyword resistance, negation following, and context awareness.

Tool-calling is the backbone of AI agents. An LLM that can reliably decide “this prompt needs get_weather, that one needs schedule_meeting, and this other one needs nothing at all” is the difference between a useful agent and an expensive autocomplete.

But there’s a harder question: when a prompt mentions “weather” but the correct action is not to call get_weather, can the model resist the keyword trigger? When the user says “don’t check the weather, just find the report,” does the model listen? When the weather is already provided in the prompt, does the model notice?

Cloud models handle this well. But what about local models running on your laptop’s CPU? The small open-weight models (0.5B-3.8B parameters) that Ollama makes trivially easy to run — can they actually do this?

This benchmark tests all of that: 11 models from 7 organisations across 4 countries, 12 prompts, 3 runs each, on a machine with no discrete GPU.

Everything runs on CPU. This is intentional — the point is to test what’s achievable on hardware most developers already own.

Qwen 2.5 is one of the strongest open model families for tool-calling at small sizes. Alibaba specifically trained these for instruction-following and function-calling. By testing all three sizes we get a clean read on how much capability you lose as you shrink from 3B down to 0.5B parameters.

Meta’s LLaMA 3.2 at 3B parameters is the obvious comparison point. It has native tool-calling support in Ollama and is widely used. It’s the model most people would reach for first.

HuggingFace’s SmolLM2 is purpose-built to be small and fast. At 1.7B parameters it sits between Qwen’s 1.5B and 3B. A newer model that doesn’t get as much benchmark attention. Including it tests whether the “small model” space has dark horses.

Mistral’s 3B edge model, Apache 2.0 licensed, from the French AI lab. This is the model you’d pick if you needed tool-calling with a European-sourced model. Tests whether the non-US/CN model ecosystem has caught up.

DeepSeek’s distilled reasoning model uses chain-of-thought internally. The interesting question: does thinking before answering improve restraint or just burn tokens?

Google’s smallest instruction model with a different architecture (sliding window attention). At 1B parameters, it tests the floor for tool-calling capability.

Slightly larger than the 3B tier but trained specifically for structured reasoning. Tests whether Microsoft’s training approach translates to tool-calling.

Microsoft’s BitNet uses 1.58-bit quantisation ({-1, 0, 1} ternary weights). The publicly available bitnet_b1_58-3B is a base model, not instruction-tuned. Included as a control to see what raw 1-bit inference looks like before instruction tuning.

The same 1.58-bit architecture, but instruction-tuned on 4 trillion tokens. This is the model that answers the question everyone has been asking: can ternary weights produce structured output?

The benchmark uses 12 prompts that escalate in difficulty:

Easy (P1-P3): Direct tool calls. “What’s the weather in Antwerp?” should obviously call get_weather. These establish whether a model can do the basics.

Ambiguous (P4): “I’m heading to Brussels tomorrow, anything I should know?” — calling get_weather is reasonable but not required. This tests whether models make sensible judgment calls.

Restraint (P5, P9): Prompts where the correct answer is to NOT call a tool. P5 asks “What tools do you have access to?” (a meta question). P9 asks “Can you write a Python script that checks the weather using an API?” (a code-writing request that mentions “weather” as a keyword trap). These are the most interesting tests — an agent that calls tools when it shouldn’t is worse than one that occasionally misses a valid call.

Hard (P6-P8): P6 requires context the model doesn’t have (“the city where we have our next sprint review”). P7 buries meeting parameters in messy natural language with filler words. P8 asks for two tools at once (“search files AND tell me the weather”) to see if models handle multi-tool requests or just pick one.

Hard — judgment traps (P10-P12): The hardest prompts, added in Round 4 to break the Round 3 plateau where four models tied at 0.929. These test whether models can pick the right tool when misleading keywords are present:

• P10: “I have a meeting with a client in Bruges next Thursday. Should I take the train or cycle?” — the correct tool is get_weather (transport depends on weather), not schedule_meeting (the meeting already exists). Tests implicit reasoning.
• P11: “Don’t check the weather in Antwerp, just find me the quarterly report.” — the correct tool is search_files. Calling get_weather means the model ignored an explicit negation. Tests instruction following.
• P12: “The weather in Antwerp is 8°C and rainy. Should I schedule an indoor meeting with Jan?” — the correct tool is schedule_meeting. The weather is already provided; calling get_weather is redundant. Tests context awareness.

• Action Score: correct_tool_calls / 10. How many of the 10 actionable prompts produced valid tool calls with the correct tool. For P10-P12, the tool must match the expected tool. Measures execution capability.
• Restraint Score: correct_refusals / 2. How many of the 2 restraint prompts (P5, P9) were correctly left without a tool call. Measures policy calibration.
• Wrong Tool: Count of specifically-bad tool calls on P10-P12 (0-3). Each hard prompt has a “wrong tool” that is worse than not calling any tool at all. Measures judgment under misleading context.
• Reliability: Average per-prompt (successful_runs / 3). Computed from per-run data before majority voting. A model that passes a prompt in 2 of 3 runs gets 0.67 reliability for that prompt, even though majority voting calls it a pass. A coarse stability signal from a 3-run sample, not a deployment-grade confidence estimate.
• Multi-Tool Accuracy: correct_tools / required_tools for P8 (dual-tool prompt). P8 requires both search_files and get_weather. Ollama’s native tool API returns only the first tool call, so this is N/A for native-tools models.
• Agent Score: Action × 0.4 + Restraint × 0.3 + Wrong-Tool-Avoidance × 0.3 where Wrong-Tool-Avoidance = (3 – wrong_tool_count) / 3. A model that calls tools aggressively but picks the wrong ones is penalized. A model that conservatively declines uncertain prompts is rewarded.
• Latency: Wall-clock time per inference call (milliseconds).

Everything is run 3 times. Correctness uses majority-vote aggregation; reliability uses per-run data.

Context-window caveat: All Ollama models were run with default settings. Ollama defaults to a 4,096-token context window (num_ctx), well below the training context of most models tested (e.g. 131,072 for Qwen 2.5). Our prompts are short enough that 4K is not a binding constraint here, but models may behave differently at longer context lengths or with num_ctx tuned to match n_ctx_train. Results should be read as “this model at Ollama defaults,” not as the model’s full capability ceiling.

Agent Score rewards correct action and correct inaction; wrong-tool calls are penalized.

*Ollama native-tools API returns only the first tool call.

The leaderboard’s most counterintuitive finding is at the top. qwen2.5:1.5b — a model half the size of its 3B sibling — ties for first place. It handles only 50% of actionable prompts (Action 0.500), but everything it attempts is correct: perfect restraint, zero wrong tool calls. The 3B model is more capable in raw execution (Action 0.800) but its aggression — calling get_weather when asked to write code, calling it again when the weather was already provided — drops it to 4th.

Under the safety-weighted scoring used here, a model that declines when uncertain outperforms one that guesses confidently and gets it wrong. This is not a universal ranking — under an action-maximizing formula, the 3B model would rank higher. But it illustrates a real architectural trade-off: for autonomous agents where wrong actions have consequences, less capable but more cautious models may be the better default.

In Round 3, four models tied at 0.929 Agent Score. The combination of P10-P12 (which test judgment, not just execution) and the new wrong-tool penalty in the Agent Score spread them from 0.640 to 0.800, with two new leaders — qwen2.5:1.5b and ministral-3:3b — that weren’t even in the top group before. The Round 3 ceiling reflected both a lack of judgment-testing prompts and a scoring formula that didn’t penalize wrong tool calls.

The two leaders scored highest by declining uncertain prompts rather than guessing wrong. qwen2.5:1.5b missed P10 and P11 entirely (losing Action points) but avoided wrong-tool penalties. ministral-3:3b did the same. Meanwhile, models that aggressively called tools — qwen2.5:0.5b, smollm2, phi4-mini — all got 2 wrong tool calls, dropping them below the conservative models. This reflects a deployment preference where wrong actions are costlier than missed ones — reasonable for autonomous agents, but not a universal truth. Under an action-maximizing formula, aggressive models like llama3.2:3b (Action 0.900) would rank higher.

Five of eight functional models called get_weather whenever they saw “weather” in the prompt, regardless of context. P11 says “don’t check the weather” — three models called get_weather anyway. P12 says “the weather is 8°C and rainy” (already known) — five models called get_weather to look it up again. The keyword cue appears to override explicit negation and contextual redundancy. Whether this reflects shallow keyword matching, weak instruction-priority resolution, or something else can’t be determined from three prompts.

qwen2.5:1.5b (0.800) now outperforms qwen2.5:3b (0.670). The 3B model’s aggression — calling get_weather when asked to write code (P9) and when weather was already given (P12) — costs more than the 1.5B model’s conservatism. The relationship between parameter count and agent quality is non-monotonic when judgment is measured. Note: this ranking depends on the scoring formula, which gives 60% combined weight to restraint and wrong-tool-avoidance. Under an action-heavy formula, the 3B model would rank higher. The underlying observation is robust: the larger model is more aggressive and makes more wrong calls; the smaller model is more conservative and avoids them.

BitNet-2B-4T is the clearest example of execution ability diverging from judgment. It retained strong execution: Action 0.800, Multi-Tool 1.000 (still the only model to correctly emit both tools on P8). It generates perfectly structured JSON on CPU at 2.3s average latency. But the judgment prompts were devastating: it called schedule_meeting for P10 (should be get_weather) and get_weather for P12 (weather already given). A model that can format a flawless tool call is not the same as a model that knows which tool to call. Whether the judgment gap stems from the ternary weights, the 2B parameter count, or the training data can’t be isolated from this benchmark.

LLaMA 3.2 has better judgment than expected. Despite zero restraint (calls a tool on every prompt), it correctly picked search_files for P11 and schedule_meeting for P12 — the two prompts that tripped up most models. Action 0.900 is the highest in the benchmark. Its problem is restraint (0.000), not tool selection.

P12 is the strongest discriminator in this prompt set. “The weather in Antwerp is 8°C and rainy. Should I schedule an indoor meeting with Jan?” Only 2 of 11 models correctly called schedule_meeting. It simultaneously tests context awareness, keyword resistance, and action identification.

P10 is the hardest prompt. Only 1 model (qwen2.5:3b) called the correct tool. “Should I take the train or cycle?” requires inferring that transport depends on weather — a reasoning chain no other model made.

Chain-of-thought still doesn’t help. DeepSeek-R1:1.5B thinks hard (5.8s average latency) and produces responses like get_weather(Antwerp) — it understands the concept but can’t produce the JSON format. Same story as Round 3.

Ministral-3:3B is accurate but slow. Action 0.500, Restraint 1.000, Wrong Tool 0 — but 8.3s average latency with some prompts taking 29+ seconds. The EU sovereignty candidate works safely, but you’ll wait for it.

For the full data — per-run breakdowns, latency matrices, raw BitNet output samples, hard prompt analysis, failure analysis — see REPORT.md.

Local tool-calling agents work today on commodity hardware — but only for simple, unambiguous tasks. Every model tested can parse “What’s the weather in Antwerp?” and emit valid JSON. The gap opens when prompts require judgment: resisting a keyword trigger, noticing that information is already provided, or inferring which tool a question actually needs. No sub-4B model handled all three reliably. In practice, small local models behave reliably as request routers, but not yet as autonomous decision-makers.

For anyone building a local agent pipeline, the practical implications are:

• Local models are viable as routers for clear-cut requests. If the user says “check the weather in Paris,” a 1.5B model on CPU handles that correctly at ~2.5s latency. For well-defined, unambiguous tool dispatch, the problem is solved.
• Judgment requires a safety layer. If your agent needs to decide whether to act — not just how — sub-4B models will make confident wrong calls. Confirmation prompts for destructive actions, allowlists for which tools can be called autonomously, or escalation to a larger model (local or cloud) for ambiguous prompts are not optional. The data here suggests treating the local model as a fast first-pass router and gating anything uncertain.
• Conservative defaults are safer than aggressive ones. The top-scoring models in this benchmark won by declining when uncertain. For autonomous agents where wrong actions have real costs — sending emails, modifying files, making API calls — defaulting to inaction on low-confidence calls and asking for human confirmation is a reasonable architecture. The cost of a missed action is a follow-up prompt; the cost of a wrong action may be irreversible.
• Full autonomy is premature at this model size. An unsupervised agent loop built on a sub-4B model will eventually hit a prompt where the keyword cue overrides the actual instruction. The failure mode is not “model refuses to act” — it’s “model confidently takes the wrong action.” This is the harder problem to guard against, because it looks like success until it isn’t.
• Test your actual prompts. Rankings here are specific to this prompt set, this scoring formula, and these model-protocol pairs. A model that scores well on keyword-resistance may fail on other judgment dimensions not tested here. Run your own prompts before trusting any leaderboard, including this one.

This benchmark has a narrow scope by design, and the results should be interpreted accordingly:

• Small prompt set. 12 prompts (3 of which test judgment) is enough to reveal patterns but not enough to make strong statistical claims. The failure modes observed are consistent across runs, but confirming their generality would require a larger and more varied prompt set.
• Safety-weighted scoring. The Agent Score gives 60% combined weight to restraint and wrong-tool-avoidance. This structurally favors conservative models. Under an action-maximizing formula, the rankings would shift significantly. The scoring reflects one deployment preference (wrong actions are costly), not a universal truth.
• Model-protocol pairs, not models in isolation. Each result reflects a specific model running through a specific backend (Ollama native tools, Ollama raw prompt, or BitNet’s OpenAI-compatible endpoint). The same model may behave differently with a different interaction contract. Rankings should not be read as generalizing across protocols.
• Default Ollama settings. All Ollama models ran with num_ctx at the default 4,096 tokens, well below most models’ training context. Our prompts are short enough that this is not a binding constraint, but results reflect “model at Ollama defaults,” not the model’s full capability ceiling.
• Default sampling settings. All models were run with backend defaults (temperature, top_p, etc.). Different sampling parameters may change behavior.
• Three runs per prompt. Reliability scores are coarse stability signals, not deployment-grade confidence estimates. A model showing 2/3 consistency on a prompt may be more or less stable than that figure suggests.

• Ollama installed and running
• Python 3.10+
• ~20 GB free disk space for models
• For BitNet: cmake, clang, clang++, and ~14 GB additional disk space during build

If you just want to test the 9 Ollama models and skip BitNet:

# Pull the models
ollama pull qwen2.5:3b
ollama pull qwen2.5:1.5b
ollama pull qwen2.5:0.5b
ollama pull llama3.2:3b
ollama pull smollm2:1.7b
ollama pull ministral-3:3b
ollama pull deepseek-r1:1.5b
ollama pull gemma3:1b
ollama pull phi4-mini:3.8b

# Clone and set up
git clone this-repo> && cd local-agent-bench
python -m venv .venv
source .venv/bin/activate
pip install ollama requests

# Comment out the bitnet entries in ALL_MODELS and the
# start/stop_bitnet_server() calls in main(), then run:
python bench.py

# 1. Pull Ollama models (same as above)

# 2. Clone and build BitNet
cd ~/projects
git clone https://github.com/microsoft/BitNet.git bitnet
cd bitnet
git submodule update --init --recursive
python -m venv .venv
source .venv/bin/activate

# Relax the torch version constraint for Python 3.12+
sed -i 's/torch~=2.2.1/torch>=2.2.1/g' \
  3rdparty/llama.cpp/requirements/requirements-convert_hf_to_gguf.txt \
  3rdparty/llama.cpp/requirements/requirements-convert_hf_to_gguf_update.txt

pip install -r requirements.txt
pip install --no-deps 3rdparty/llama.cpp/gguf-py

# Download and build the base 3B model
python setup_env.py --hf-repo 1bitLLM/bitnet_b1_58-3B -q i2_s

# Download the instruction-tuned 2B-4T model
pip install huggingface_hub
huggingface-cli download microsoft/BitNet-b1.58-2B-4T-gguf \
  --local-dir models/BitNet-b1.58-2B-4T

# If compile fails with a const-correctness error in ggml-bitnet-mad.cpp,
# change line 811 from "int8_t * y_col" to "const int8_t * y_col" and rebuild:
#   cmake --build build --config Release

# Verify
ls build/bin/llama-server
ls models/bitnet_b1_58-3B/ggml-model-i2_s.gguf
ls models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf

# 3. Clone and run the benchmark
cd ~/projects
git clone this-repo> && cd local-agent-bench
python -m venv .venv
source .venv/bin/activate
pip install ollama requests

# Update BITNET_DIR in bench.py if your BitNet path differs
python bench.py

The full run (11 models x 12 prompts x 3 runs = 396 inference calls) takes roughly 45-60 minutes on the hardware described above. Most of that time is BitNet 3B (base model) and ministral-3:3b — the other models finish faster.

The benchmark is a single file (bench.py). To add models, prompts, or adjust runs:

• Add an Ollama model (native tool API): Add {"name": "your-model:tag", "backend": "ollama", "origin": "XX"} to ALL_MODELS
• Add an Ollama model (raw prompt): Use "backend": "ollama_raw" for models that don’t support Ollama’s native tool API or perform better with system-prompt-based tool calling
• Add a prompt: Append to TEST_PROMPTS. If it’s a restraint prompt (correct answer is no tool call), add its 0-based index to RESTRAINT_INDICES. If it’s a hard prompt with an expected tool, add it to EXPECTED_TOOLS and WRONG_TOOL_MAP
• Change run count: Edit num_runs in main()
• Add tools: Extend the TOOLS list and TOOL_DISPATCH dict. Update BITNET_SYSTEM_PROMPT if you want raw-prompt models to know about them
• Add to edge leaderboard: Add the model name to EDGE_MODELS if it’s sub-2B

Use it freely; attribution appreciated. It’s a benchmark script, not a product.