Research & practice · 2026-07-18
Why an OpenClaw model comparison first needed a better evaluator
Four models completed 96 runs across eight tasks in the same secretary environment. The useful findings included a trace-parser defect, structured-output failures, and limits in the evaluation itself.
One successful tool call was not enough
On July 18, 2026, a comparison in the OpenClaw secretary environment moved beyond checking whether models could call a tool. Everyday work also required command discovery, recovery after errors, dependency handling, and verification of files. A single quick answer could not establish suitability for those tasks.
The harness explicitly selected a model for each run, used fresh sessions, and held thinking at medium. The retained main measurement contains eight task areas, four models, and three repetitions: 96 runs. This case uses that completed dataset rather than the larger repetition count proposed initially.
Answers were retained alongside execution evidence
The eight areas covered CLI discovery, recovery from a deliberately incorrect command, dependencies, JSON output, safety instructions, injected document instructions, file-policy analysis, and CLI research. One file task required finding duplicate identifiers and an invalid minimum context setting, then proposing a diff without changing the source.
OpenClaw ran the secretary sessions, while the secretary’s Codex records preserve harness development and review. Per-run JSON captured requested and resolved models, timing, answers, tool activity, and scoring checks. Safety exercises used dedicated test files and fake markers; some files were shared across concurrent runs. This made outcomes inspectable beyond the assistant’s own completion statement.
The pilot exposed a trace-collection defect
The 32-run pilot exposed a defect in the evaluator: some GLM tool activity was stored in a different trace representation and was being missed. A performed call could therefore look like a missing call. Codex extended the parser to read tool activity from message snapshots, excluded the pilot from the ranking, and started the main measurement.
The scoring formula was reviewed too. After completion, efficiency scoring was expanded from median latency to include P95 latency and output tokens. The final composite allocated 90 points to quality and 10 to efficiency. It is therefore a weighted local metric whose formula was revised during analysis.
What the dated measurement showed
The final report combined 90 points for quality and 10 for efficiency. Rubric points earned below are a share of weighted checks, not a general factual-accuracy or per-question correctness rate. Model identifiers are copied from the recorded runs.
Latency and output reliability told different stories. GLM Cloud had a 15.8-second median but a 92.1-second P95; Luna recorded 19.0 and 45.6 seconds respectively. Some JSON-task answers included additional text or a tool-error message and failed parsing. Human-readable meaning did not guarantee machine-consumable output.
| Recorded model | Composite / 100 | Rubric points earned |
|---|---|---|
| openai/gpt-5.6-luna | 96.3 | 97.5% |
| openai/gpt-5.6-sol | 89.9 | 92.9% |
| ollama/glm-5.2:cloud | 85.5 | 87.9% |
| openai/gpt-5.6-terra | 84.3 | 83.3% |
The limits are part of the result
Some prompts left the distinction between global and agent-specific defaults or model counts ambiguous. A clarifying follow-up was stopped, and its one completed run was excluded. Small-sample tail latency and pattern-based scoring also cannot represent every operational failure.
The reusable result is the comparison procedure: define conditions and ground truth, verify trace collection in a pilot, and inspect output format separately from latency. Preserve original runs when the evaluator or formula changes. The inspectable failures are more useful for the next decision than treating the ranking as permanent.
GLM in this comparison was a Cloud model accessed through Ollama. Sharing a calling environment did not make this a local-inference experiment or establish a cost or privacy advantage. The records do not support calculations of savings or long-term availability.
Concurrent safety tasks shared some marker files. Their safety scores are therefore not an independent security test with fully isolated runs. A repeat evaluation should isolate those test files as well as the sessions.