OpenClaw Model Research

Updated: 2026-04-05

Testing various models for use with OpenClaw.

Test Prompts

Results Summary

Sorted by $/M input (cheapest first). — = not tested (model incompatible with tool calling).

Failed / Not Viable

Model Notes

Grok 4.1 Fast — x-ai/grok-4.1-fast

Score: 4/4 ✅ | Cost: $0.020/run

Strong cost/performance model. Passes all four tests cleanly with no chattiness, correct tool routing, and good response quality. Broadens grocery results to show alternative steak deals. Concise weather formatting. At $0.20/M input and $0.50/M output, it’s 15x cheaper than Claude Sonnet on input and 30x cheaper on output. Displaced as the primary LIGHT/MEDIUM model by MiniMax M2.7, which proved cheaper per query in practice despite slightly higher per-token cost.

Claude Sonnet 4.6 — anthropic/claude-sonnet-4.6 ⭐ Router HEAVY

Score: 4/4 ✅ | Cost: $0.176/run | Current: HEAVY tier

The fastest and highest quality model. Produces the most polished responses — adds context like “solid cut if you’re looking for…” on grocery deals, highlights rocket launch visibility from Jupiter, and gives weather summaries with personality (“Pretty nice out!”). The 2.8s general knowledge response is the fastest of any model. Premium pricing is justified only for complex, multi-step tasks where quality matters most.

Gemini 3 Flash Preview — google/gemini-3-flash-preview

Score: 4/4 ✅ | Cost: $0.052/run

Strong all-rounder and the best backup to Grok 4.1 Fast. Fastest on grocery (12s) and weather (9s). Uses correct tools every time with clean single-message responses. Only weakness is the general knowledge test where it takes 23s — oddly slow for a simple factual question. At $0.50/$3.00 per M, it’s 2.5x more expensive than Grok but still very affordable.

GLM-4.6 — z-ai/glm-4.6

Score: 4/4 ✅ | Cost: $0.043/run

Passes all tests but has a speed problem — grocery took 82s. Response quality is good but not exceptional. Humorously listed “Beefsteak Tomato $1.99” as a steak deal in the grocery results. Extremely concise on general knowledge (just “Copenhagen”, 4 output tokens). Pricing at $0.35/$1.71 is competitive but Grok 4.1 Fast is cheaper and faster.

GLM-4.7 — z-ai/glm-4.7

Score: 3✅ 1⚠️ | Cost: $0.063/run

Slightly cheaper than GLM-4.6 per token but uses significantly more tokens (194K total vs 121K). Chatty on the grocery test (2 messages). Grocery was extremely slow at 92s. Otherwise solid — correct tool usage, good rocket and weather responses. Not worth choosing over GLM-4.6 or Grok.

Claude Haiku 4.5 — anthropic/claude-haiku-4.5

Score: 1✅ 3⚠️ | Cost: $0.058/run

Gets every answer right with correct tool usage, but chatty on every tool-use test (2-4 messages instead of 1). Sends intermediate “thinking” messages like “Let me check…” before the actual answer. The fastest model on tool tasks (7-11s) and general knowledge (3.7s). Sits awkwardly between Grok ($0.20/$0.50) and Sonnet ($3.00/$15.00) — too expensive for LIGHT tier, too chatty for production use.

Gemini 3.1 Pro Preview — google/gemini-3.1-pro-preview

Score: 4/4 ✅ | Cost: $0.250/run

Clean 4/4 pass with excellent response quality, but the most expensive model tested at $0.250 per run. Grocery took 74s — slow. High output token counts (236-1877 per test) drive up cost. At $2.00/$12.00 per M, it’s nearly as expensive as Claude Sonnet but slower. Not cost-effective for any router tier.

GPT-5.3-codex — openai/gpt-5.3-codex

Score: 3✅ 1⚠️ | Cost: $0.074/run

Strong tool calling and fast responses (12.3s average — second only to Claude Sonnet’s 9.3s). Grocery is clean — correctly reports no flank steak, broadens to petite sirloin at Sprouts. Weather and general knowledge both pass cleanly. Rocket gets a minor chattiness ding (2 messages). Response quality is high with good emoji usage and concise formatting. The catch is cost: at $1.75/$14.00 per M it’s nearly as expensive as Claude Sonnet ($3.00/$15.00) but with a lower score. The grocery test alone consumed 19.5K input tokens ($0.034 input) due to skill data volume. Still proves GPT-5.3-codex is a capable agent model — the OAuth integration is the problem when trying to use it through a ChatGPT Plus subscription, not the model itself.

MiniMax M2.5 — minimax/minimax-m2.5

Score: 1✅ 3⚠️ | Cost: $0.051/run

Solid budget option at $0.30/$1.10 per M. Gets every answer right with correct tool usage — grocery correctly reports no flank steak and broadens to alternatives, rocket returns real launch data, weather is accurate. The ⚠️ ratings are all for minor chattiness (2-3 messages instead of 1), not wrong answers. The extra messages are things like brief formatting notes, not the problematic “let me check…” intermediate spam. One quirk: consistently shows the wrong flag for Denmark (🇨🇿 Czech Republic instead of 🇩🇰). At $0.051/run it’s competitive with Gemini 3 Flash ($0.052) but the chattiness and flag issue keep it behind Grok 4.1 Fast.

MiniMax M2 — minimax/minimax-m2

Score: 3✅ 1⚠️ | Cost: $0.056/run

Slightly better than its M2.5 sibling — less chatty overall (3✅ vs 1✅), with weather and grocery both passing clean. Correct tool routing on every test. The rocket test got a minor chattiness ding (2 messages). Same Czech flag bug as M2.5 (🇨🇿 instead of 🇩🇰 for Denmark) and oddly repeated the general knowledge answer twice. At $0.26/$1.00 per M it’s marginally cheaper than M2.5 but uses more input tokens (210K vs 165K total), landing at a similar per-run cost. A decent budget option but still behind Grok 4.1 Fast on both quality and price.

GPT-4o-mini — openai/gpt-4o-mini

Score: 2✅ 2⚠️ | Cost: $0.010/run

Cheapest per-run cost but poor skill routing. Spawned subagents for the grocery test instead of reading SKILL.md, used web_search for weather instead of the weather skill, and got the rocket launch date wrong (Feb 28 vs Feb 27). The older/smaller OpenAI models don’t navigate OpenClaw’s skill framework well. At $0.15/$0.60 per M it’s dirt cheap, but Grok 4.1 Fast ($0.20/$0.50) is barely more expensive and goes 4/4 clean.

Kimi K2.5 — moonshotai/kimi-k2.5

Score: 3✅ 1⚠️ | Cost: $0.041/run

Good all-rounder from Moonshot AI. Correct tool routing on every test — reads SKILL.md then runs the right script. Grocery is the only ding (3 messages — chatty while running multiple searches). Rocket response is nicely formatted with visibility info. Weather is accurate. Gets the Danish flag right (🇩🇰). At $0.45/$2.20 per M it’s mid-range pricing, landing at $0.041/run — cheaper than Gemini 3 Flash ($0.052) due to lower token consumption (83K total). Weather was slow at 41s. A solid option but Grok 4.1 Fast still wins on both price and clean 4/4 score.

Qwen3 Max Thinking — qwen/qwen3-max-thinking

Score: 1✅ 2⚠️ 1❌* | Cost: $0.318/run

The most expensive model tested at $0.318 per run, and it didn’t even pass cleanly. Chatty on 3 of 4 tests (2-4 messages). The grocery ❌ is likely a false positive — the model correctly said “No flank steak is currently on sale” but one of its earlier chatty messages probably contained intent phrasing like “let me check if flank steak is on sale” which triggered the hallucination detector. Correct tool usage throughout. Consumes massive input tokens (261K total) due to the thinking/reasoning overhead. At $1.20/$6.00 per M, it’s 6x more expensive than Grok 4.1 Fast on input and 12x on output with worse results. Not recommended for any router tier.

GPT-oss-120b — openai/gpt-oss-120b

Score: 3✅ 1❌ | Cost: $0.006/run

OpenAI’s larger open-weight MoE model — 117B params, 5.1B active per pass, runs on a single H100. At $0.04/$0.19 per M tokens it’s the second cheapest model tested, only behind its 20b sibling. Rocket launch response is excellent — well-formatted with emoji, visibility notes, all the right data. Weather passes cleanly with proper skill usage (read SKILL.md, ran script). General knowledge is quick and correct with the right flag (🇩🇰). The grocery test is a hard ❌ — it used the right tools (read SKILL.md, ran the compare script) but produced no response text, similar to the 20b’s weather failure. This “silent completion” pattern seems to be a quirk of the GPT-oss family where the model correctly executes tools but then fails to synthesize a response. At $0.006/run and 12.8s average, it’s tantalizing — 3x cheaper than Grok 4.1 Fast and comparable speed. If the grocery failure is a one-off, this could be a serious LIGHT tier contender. Worth retesting.

GPT-oss-20b — openai/gpt-oss-20b

Score: 2✅ 1⚠️ 1❌ | Cost: $0.003/run

The cheapest model tested by a wide margin at $0.03/$0.14 per M tokens — a full 7x cheaper than Grok 4.1 Fast on input. This is OpenAI’s open-weight 21B MoE model (3.6B active params) and it shows both the potential and limits of a small model on OpenClaw’s framework. Rocket launch is clean with nice formatting (visibility notes, emoji). General knowledge is instant at 2.1s. Grocery gets the right answer (no flank steak, broadens correctly) but the accuracy check flagged it ⚠️ — likely a pattern-matching issue since it listed Beefsteak Tomato as a “steak” deal. The weather test is a hard ❌ — it read the SKILL.md but never ran the script, returning no response. At $0.003/run it’s interesting as a potential LIGHT tier model for trivial queries, but the weather failure and grocery accuracy issue make it unreliable for tool-heavy workflows. The 7.0s average time is the second fastest after Claude Sonnet (9.3s).

Devstral Small — mistralai/devstral-small

Score: 1✅ 2❌ 1⚠️ | Cost: $0.093/run

A disaster for agent tasks despite rock-bottom per-token pricing ($0.10/$0.30). Spawned 41 subagents and consumed 357K input tokens on a simple grocery question with no response. Fired 24 web searches (454K tokens) for weather instead of using the skill. Burned 922K total tokens across 4 tests — the most of any model by 5x. Deceptively cheap per-token but ruinously expensive in practice due to uncontrolled tool loops.

Round 3 Results (2026-03-22)

Round 3 extended the benchmark to 5 prompts by adding the stock test:

Round 3 Benchmark Results

GPT-5.4-nano — openai/gpt-5.4-nano

Score: 4/5 (1⚠️) | Avg: 7.6s | Cost: $0.20/$1.25 per M

Fastest model in this round and solid on the benchmark — correctly uses tools (grocery-compare, rocket-launches, weather, stock skills) with concise single-message responses. The only ding is the stock test (2 messages, minor chattiness). No hallucination patterns detected.

However, the natural prompt comparison (see below) revealed a significant gap between benchmark and real-world performance: in practice, GPT-5.4-nano avoids tool use and asks clarifying questions instead of acting on available context. The benchmark’s structured prompts don’t expose this; open-ended real-world queries do.

Mistral Small 2603 — mistralai/mistral-small-2503

Score: 3/5 (2⚠️) | Avg: 12.2s | Cost: $0.15/$0.60 per M

Inexpensive and fast, but inconsistent. Passes rocket and stock cleanly. Grocery and weather both get ⚠️ — chatty responses (2+ messages) with extra context tacked on rather than the problematic “let me check…” intermediate spam. Competitive pricing but the inconsistency makes it a poor fit for the LIGHT tier where predictable, single-message responses matter.

MiniMax M2.7 — minimax/minimax-m2.7

Score: 5/5 ✅ | Avg: 14.2s | Cost: $0.30/$1.20 per M

Clean 5/5 pass with correct tool routing, good response quality, and no chattiness — the best benchmark score of any model in Round 3. The natural prompt comparison confirmed the benchmark: proactively used tools, gave thorough and accurate answers to all three real-world questions, at just $0.014 for 3 queries vs Grok’s $0.035. At $0.30/$1.20 per M it’s 1.5x the input cost of Grok but roughly 2.5x cheaper per query in practice due to lower token consumption. A strong upgrade to the MiniMax M2 and M2.5 predecessors which struggled with chattiness.

See the Real-World Deployment Testing section for production findings.

NVIDIA Nemotron 3 Super 120B — nvidia/llama-3.1-nemotron-ultra-253b-v1

Score: 3/5 (2⚠️) | Avg: 18.0s | Cost: $0.10/$0.50 per M

Attractive per-token pricing but inconsistent results. Grocery and rocket both get ⚠️ — correct answers but tool errors or chattiness along the way. Weather at 47s is notably slow. High token consumption (255K input tokens on complex tasks) means the attractive per-token price doesn’t translate to low per-query cost. At $0.10/$0.50 per M it’s cheaper than Grok on input but more expensive in practice. Not competitive with MiniMax M2.7 on either quality or real-world cost.

Natural Prompt Comparison (2026-03-22)

The benchmark uses structured prompts designed to test specific capabilities. To evaluate real-world suitability, three open-ended personal assistant questions were run through Grok 4.1 Fast, GPT-5.4-nano, and MiniMax M2.7 via compare-models.py:

1. “When is Christine’s flight today and when do you think we should leave to drive to the airport?”
2. “What time tomorrow would it be best for me to go for a run?”
3. “Is there anything interesting happening around here this weekend?”

GPT-5.4-nano’s benchmark score (4/5) doesn’t reflect its real-world behavior. With open-ended prompts, it declines to use context or tools and asks what you mean instead. This makes it unsuitable as a LIGHT tier model despite its speed and low benchmark cost.

MiniMax M2.7 is the standout finding: at $0.014 for 3 queries vs Grok’s $0.035, it’s 2.5x cheaper with answer quality that rivals Grok. See Real-World Deployment Testing for how it held up in production.

Real-World Deployment Testing (2026-03-24)

After the Round 3 benchmark, MiniMax M2.7 was deployed as LIGHT tier for real-world testing in the Slack bot. Key findings:

Context size is not the issue. The earlier timeout when first deploying MiniMax was transient — a service blip, not a fundamental problem. Direct API tests at full OpenClaw context size (64k chars system prompt, 26 tool definitions) consistently return in 4–7s. The router sends requests to OpenRouter’s Anthropic-compatible /v1/messages endpoint, which MiniMax handles without issue.

Tool parameter naming. MiniMax (and Grok) both send emojiName instead of emoji when calling the Slack react tool — a mismatch with OpenClaw’s tool schema. This caused failed reactions that occasionally leaked error messages to the Slack channel. Fixed at the router level with a tool input normalization pass that rewrites emojiName→emoji before the gateway executes the call.

Verdict: back on LIGHT tier. With the router-level fix in place, MiniMax M2.7 is the best cost/performance option for LIGHT — 2.5x cheaper per query than Grok in natural prompt tests with comparable answer quality and a clean 5/5 benchmark.

Round 4 Results (2026-03-24)

Round 4 tested four models at the low-cost end of the OpenRouter catalog (all under $0.10/M input), looking for candidates that could challenge MiniMax M2.7 on the LIGHT tier.

Round 4 Benchmark Results

Mistral Small 3.2 24B — mistralai/mistral-small-3.2-24b-instruct

Score: 2/5 (2⚠️, 1❌) | Avg: 9.3s | Cost: $0.07/$0.20 per M

Tool calling works — a significant fix over 3.1’s complete failure. Weather and general pass cleanly. Grocery ran the right tools (5 turns) but returned no store names. Rocket reached for memory_search instead of the rocket-launches skill. Stock hit 8 consecutive tool errors across 9 turns and gave up with an apology. Fast response time but unreliable tool selection — not competitive with MiniMax M2.7.

Gemini 2.5 Flash Lite — google/gemini-2.5-flash-lite

Score: 1/5 | Cost: $0.10/$0.40 per M

Actively refuses to use skill tools. Grocery: “The grocery comparison tool is not available.” Weather: “the tool seems to be unavailable right now.” Stock: two turns, no tool calls, an apology. Only the no-tool general test passes. Fast and cheap per token, but broken for any agent workflow that requires skill use.

Llama 4 Scout — meta-llama/llama-4-scout

Score: 1/5 | Cost: $0.08/$0.30 per M

Catastrophically broken output. Responses contain raw chat template markers (<|header_start|>assistant<|header_end|>) and tool calls rendered as plain text (memory_search(query="next rocket launch")). The one passing test barely counts: [[reply_to_current]] The capital of Denmark is Copenhagen. — the [[reply_to_current]] prefix is leaked internal syntax. This model is not correctly instruction-tuned for OpenRouter’s API format.

Qwen3 235B — qwen/qwen3-235b-a22b-2507

Score: 2/5 (3⚠️) | Avg: 51.8s | Cost: $0.07/$0.10 per M

Technically the most capable of the four: correct tool routing on every test, no hallucination, and real answers (ETSY at $52.20, -1.21%). But 25–92s per prompt and 74K–374K input tokens per query make it unusable for real-time chat. Weather alone took 74s across 10 turns reading 286K tokens. At $0.10/M output it’s cheap per token, but at 374K input tokens for a stock question the per-query cost balloons. An interesting model for batch or offline tasks, not for an interactive assistant.

Round 4 verdict: No model from this batch challenges MiniMax M2.7. MiniMax remains the best LIGHT tier option.

Round 5 Results (2026-04-02 / retested 2026-04-05)

Round 5 tested three models: Qwen 3.6 Plus (free), Gemma 4 31B, and Gemma 4 26B A4B. The two Gemma models initially appeared as complete failures on April 2 (the day they were released on OpenRouter) — 0 tokens, no responses, silent retries. Retesting on April 5 confirmed this was a launch-day provider instability, not a fundamental incompatibility. Both models use the standard OpenRouter tool-calling format correctly; the AkashML/Novita providers just weren’t stable at launch.

Round 5 Benchmark Results

Qwen 3.6 Plus — qwen/qwen3.6-plus:free 🆕 Free tier candidate

Score: 3/5 (2⚠️) | Avg: 14.6s | Cost: FREE | Tokens: ~504K in / 3K out

Strong free model with correct tool routing on every test. Reads SKILL.md, runs the right scripts, returns accurate real data — no hallucinations, no silent completions. The two WARNs are minor: grocery sent 2 messages instead of 1, and rocket had 1 tool error but still returned a correct single-message response with real launch data.

Natural prompt test (2026-04-02): Ran 3 open-ended questions via compare-models.py. Q1 (Christine’s flight): correctly checked FLIGHTS.md and reported no flight found — clean handling of a “not found” result. Q2 (run timing): pulled actual hourly forecast for Jupiter and gave a specific recommendation (6–7 AM, ~23°C, 66% humidity) with an evening fallback — comparable depth to Grok. Q3 (weekend events): found Man of La Mancha’s final weekend at Maltz Jupiter Theatre, Warriors home game, and date-night workshops in West Palm Beach. Total cost: $0.00 for 3 questions.

Compared to Arcee Trinity (the previous best free model, which timed out on grocery at 181s), Qwen 3.6 Plus finished grocery in 22s with correct data. Best free model tested to date.

Verdict: Viable fallback for the LIGHT tier when cost is the primary constraint. Not switching from MiniMax M2.7, but worth keeping as a named fallback option.

Gemma 4 26B A4B — google/gemma-4-26b-a4b-it 🆕

Score: 3/5 (2⚠️) | Avg: 28.8s | Cost: $0.13/$0.40 per M

MoE variant: 25.2B total parameters, 3.8B active per token. Grocery, rocket, and weather all pass cleanly and quickly (9–19s). The two WARNs are more concerning than Qwen’s minor issues: stock hit a tool error and fell back to web_search for supplemental data (right answer, wrong path), and the general knowledge test went completely off the rails — it fired memory_search 3 times (all errors), burned 86s and 37K tokens, then hallucinated an answer about a flight instead of answering “Copenhagen”. That’s not a chattiness ding; that’s context confusion.

Natural prompt test (2026-04-05): Q1 (Christine’s flight): correctly checked FLIGHTS.md and reported no flight found — good. Q2 (run timing): pulled real forecast, gave a specific 8 AM recommendation with temperature data — solid. Q3 (weekend events): gave some local Jupiter events but also injected Waterloo, ON info (for Carl, from session context), blurring boundaries between users. Total: $0.019 for 3 questions in 222s.

Verdict: Fast and cheap when it stays on track, but the context confusion on general knowledge and the cross-user bleed in Q3 are reliability concerns. Not competitive with MiniMax M2.7 for the LIGHT tier.

Gemma 4 31B — google/gemma-4-31b-it 🆕

Score: 3/5 (2❌) | Avg: 85.6s | Cost: $0.14/$0.40 per M

Dense 30.7B model. Tool calling works, but speed is the fundamental problem. Grocery took 124s (nearly hit the timeout), weather 49s, and both rocket and stock timed out at 123s — the model was still mid-run when the clock expired. The two timeouts aren’t flaky failures; the model is simply too slow for interactive chat. General knowledge at 9s shows it can be fast on trivial queries, but anything requiring multi-turn tool use pushes into territory where users would have already given up.

Natural prompt test (2026-04-05): Q1 (Christine’s flight): failed — 5 turns, 175s, no response. Q2 (run timing): correct and detailed, but took 399s across 9 turns — nearly 7 minutes for one question. Q3 (weekend events): responded in 84s but referenced “March 14” events, hallucinating stale data rather than checking current sources. Total: $0.036 for 3 questions in 658s (~11 minutes).

Verdict: Not viable for interactive use. The same density that likely gives it good reasoning also makes it too slow. The MoE sibling (26B A4B) is faster at the same price point.

Round 5 verdict: Qwen 3.6 Plus is the best free model tested. Gemma 4 26B A4B is faster than 31B but has reliability concerns. Neither displaces MiniMax M2.7.

Current Router Configuration

LIGHT  → minimax/minimax-m2.7         ($0.30/M in, $1.20/M out)
MEDIUM → x-ai/grok-4.1-fast          ($0.20/M in, $0.50/M out)
HEAVY  → anthropic/claude-sonnet-4.6  ($3.00/M in, $15.00/M out)

Key Findings

1. Per-token price is misleading. Devstral Small is the cheapest per token but the most expensive per test because it spirals into uncontrolled tool loops. Always evaluate total cost per task.

2. OAuth ≠ API. GPT-5.3-codex works through OpenRouter’s API but completely fails through ChatGPT Plus OAuth. The subscription model doesn’t support agent-style tool calling.

3. Local models can’t handle OpenClaw’s complexity. Models under 30B parameters pick generic tools (web_search) instead of skill-specific workflows. The ~19K token system prompt is too complex for small models to parse and prioritize.

4. Grok 4.1 Fast is the sweet spot. At 2 cents per 4-test run with 4/4 pass rate, it’s the best value by a wide margin. Gemini 3 Flash is a good backup at 5 cents.

5. Claude Sonnet justifies its premium only for HEAVY tasks. It’s the fastest and produces the best responses, but at 9x the cost of Grok, reserve it for complex multi-step reasoning.

6. Benchmark scores don’t predict real-world behavior. GPT-5.4-nano scores 4/5 on structured prompts but fails completely on open-ended personal assistant queries — it asks clarifying questions instead of using available tools. Always test with natural, open-ended prompts before deploying.

7. MiniMax M2.7 is the best LIGHT tier option. Clean 5/5 benchmark, 2.5x cheaper per query than Grok in natural prompt tests, and handles the full OpenClaw context (64k chars, 26 tools) without issue. A tool parameter naming mismatch (emojiName vs emoji) caused early friction but was fixed at the router level — the underlying capability was never in question.

OpenRouter Compliance Audit (2026-02-26)

Audited @mariozechner/pi-ai openai-completions adapter against OpenRouter docs.

Tool Calling: ✅ Fully Compliant — all 5 requirements met (tools in every request, tool result format, assistant tool_calls array, streaming accumulation, tool_choice support).

Interleaved Thinking: ⚠️ Two non-critical gaps — reasoning parameter uses OpenAI format (reasoning_effort) instead of OpenRouter unified format (reasoning: {effort}), and general reasoning_content isn’t preserved as reasoning_details across turns. Neither matters while thinkingDefault: "off".

No changes needed. If reasoning is enabled in the future, these would be upstream @mariozechner/pi-ai issues.