Research & practice
Before choosing an assistant model, I had to fix the evaluator
I compared four models over 96 OpenClaw runs. Finding missing tool traces and broken JSON taught me more about evaluation than the ranking alone.
The problem
One successful tool call was not enough to choose an assistant model.
My approach
I compared 96 runs under shared conditions and examined answers, tool traces, format failures and latency.
What changed
I repaired missing trace collection and made the ranking’s conditions and limitations explicit.
BUILD DETAILS
Build details & ideas to reuse
01One good answer was not enough
My assistant needed to discover commands, recover from errors and verify files. I built a harness using the same workspace, fresh sessions and medium thinking while changing the model. Eight tasks, four models and three repetitions produced 96 main runs.
02Collect actions as well as answers
The tasks covered command discovery and recovery, dependencies, JSON, instruction handling, file analysis and CLI research. I retained the requested and resolved models, timing, answers and tool calls in per-run JSON so I could compare completion claims with behavior.
03Missing traces were changing the score
The 32-run pilot revealed that the parser missed some GLM tool calls. I added message-snapshot parsing and excluded the pilot from the ranking. I also expanded efficiency scoring to include P95 latency and output tokens, so the composite reflects a formula revised during analysis.
04Read latency and format beside the score
The table combines 90 quality points with 10 efficiency points. GLM Cloud had a 15.8-second median and 92.1-second P95; Luna had 19.0 and 45.6 seconds. Extra prose or an error message could make an otherwise readable JSON response unusable by the next program.
| Recorded model | Composite / 100 | Rubric points earned |
|---|---|---|
| openai/gpt-5.6-luna | 96.3 | 97.5% |
| openai/gpt-5.6-sol | 89.9 | 92.9% |
| ollama/glm-5.2:cloud | 85.5 | 87.9% |
| openai/gpt-5.6-terra | 84.3 | 83.3% |
05What I would change in the next comparison
Next time I would freeze the ground truth and rubric first and isolate each run’s files. Some safety tasks shared markers, and questions about default models were ambiguous. I excluded the stopped follow-up. GLM was a Cloud model accessed through Ollama, so these results say nothing about local-inference privacy or savings.
Scope and things to consider
The figures cover my 96 main runs and revised rubric. They rank this task set, not models in general or their current performance.