What these results establish
Test date: · Local client requests to a mock HTTP service
These existing checks used real client configuration or runtime paths, a dummy key and fixed mock responses. They establish which model IDs and reasoning fields the tested clients sent. They do not establish live supplier availability or reasoning quality.
Read the dated evidence summary (JSON) · Model capabilities
Agent versions and results
| Agent / version | Execution path | Observed result |
|---|---|---|
Claude Code2.1.309 | Native CLI with generated settings.json; one invocation per model and effort. | 15 explicit effort checks passed, plus six default-model requests. Two Off attempts sent high and failed the requested setting. |
Codex0.154.0 | Native app-server configuration parsing and model/list; model and effort switched within one thread. | 18 captured requests: 17 explicit effort settings and the MiniMax default selection. |
OpenCode1.19.4 | Native CLI; model and variant switched in one persistent session. | 18 captured requests: 17 explicit effort settings and the MiniMax default selection. |
Hermes0.0.0 | Managed source snapshot, not an identified release. Real configuration loading, model parsing and OpenAI SDK transport. | 18 captured requests: 17 explicit effort settings and the MiniMax default selection. |
DeepSeek Harness0.1.5-rc.1 | Real llm-pi-ai plugin configuration and streaming adapter; the plugin host service was a test double. | 18 captured requests: 17 explicit effort settings and the MiniMax default selection. |
Pi0.85.1 | Real ModelRuntime loaded models.json and called completeSimple for each model and effort. | 18 captured requests: 17 explicit effort settings and the MiniMax default selection. |
OpenClaw2026.10.6 | Real configuration validator, thinking defaults and HTTP runtime. The production Gateway and GUI were not started. | 18 captured requests: 17 explicit effort settings and the MiniMax default selection. |
Captured model IDs and reasoning fields
Values below are request fields observed in the saved captures, not a guarantee of model behavior. none is the wire value used for Off; omitted means no effort field was sent. Failed Claude Off attempts are excluded from this table and documented below.
| API model ID | Claude Code | Codex | OpenCode | Hermes | DeepSeek Harness | Pi | OpenClaw |
|---|---|---|---|---|---|---|---|
glm-5.3-flash | low, high, max | low, high, max | low, high, max | low, high, max | low, high, max | low, high, max | low, high, max |
deepseek-flash | low, high, max | none, low, high, max | none, low, high, max | none, low, high, max | none, low, high, max | none, low, high, max | none, low, high, max |
kimi-k3 | low, high, max | low, high, max | low, high, max | low, high, max | low, high, max | low, high, max | low, high, max |
glm-5.3 | low, high, max | low, high, max | low, high, max | low, high, max | low, high, max | low, high, max | low, high, max |
minimax-m3.1 | omitted | none | omitted | omitted | omitted | omitted | omitted |
qwen3.8-max | low, medium, xhigh | none, low, medium, xhigh | none, low, medium, xhigh | none, low, medium, xhigh | none, low, medium, xhigh | none, low, medium, xhigh | none, low, medium, xhigh |
Known limits
- Claude Code 2.1.309: Off attempts for deepseek-flash and qwen3.8-max still sent high. Off is not verified for this client version; use the documented positive effort levels.
- MiniMax M3.1: default reasoning only. Codex sent none; the other tested clients omitted effort. The local gateway fix removes implicit client reasoning fields for this model. These records do not verify deployment of that fix or a live M3.1 upstream route.
- Hermes was a managed source snapshot with package version 0.0.0, not a verified release. OpenClaw's host used Node 25, outside that release's supported range; use a supported Node version for deployment.
- These results do not certify Desktop interfaces, Windows/Linux installations or every later client version.
Response and tool coverage
The retained evidence focuses on configuration parsing and outgoing model/effort fields. Ordinary responses and streaming were not separately accepted against a production model in this audit. A mock reply or streaming adapter check does not establish live streaming reliability.
Function-tool execution, file access and complete tool-call round trips were not established by these captures. No production latency, success-rate or concurrency benchmark is claimed.
The accompanying local gateway regression used LiteLLM 1.92.0: 60 protocol conversion cases, with 84 tests and 53 subtests across the recorded gateway suite. This is protocol regression evidence, not an additional live client run.
Evidence and method
This page republishes the existing 2026-10-23 audit; it does not report a new test run. The JSON contains the retained model IDs, protocol paths, effort fields and selected Claude settings, plus original record filenames and SHA-256 digests. Keys, prompts, local paths and full payloads are not included.
To repeat the method, load the corresponding Agent configuration with a dummy key against a local mock endpoint, switch each model and effort, and compare captured request fields. Live response, streaming and tool execution require separate checks against the intended deployment.
Read the dated evidence summary (JSON) · Troubleshooting · Setup guides