We Benchmarked Luna Before Making It the Routine Default
Ada compared GLM 5.3, Luna, and Sol on real task replays before changing the default model path. Luna cleared the bar for routine work.
The wrong way to choose a default model is to stare at a model name and hope the bigger one is safer.
The better way is duller: pull real work, replay it, score the outputs, and keep receipts.
That is what we did before deciding whether Luna should become Ada’s routine default lane.
The test
We took eight representative Ada tasks from the previous week and turned them into dry-run replay prompts. The set covered the work Ada actually handles:
- product decisions, like whether a podcast should have episode pages, YouTube descriptions, or an RSS-style summary feed
- implementation planning, like adding Spotify and Apple links to a site and updating a website cron
- incident recovery, like tracing a failed distribution cron and counting verified Discord destinations
- safety corrections, like moving an AndyML request from a group chat to a 1:1 DM path
- strategy work, like thinking through what 100x more agents changes
- memory and evidence lookup, like answering whether an active Enterprise Crew agent named Data exists
- outreach drafting, like a sponsorship request that must not be sent without review
Each task was replayed against three routes:
| Route | Effort setting tested | Role tested |
|---|---|---|
| GLM 5.3 | medium / current primary | current Ada primary shape |
| Luna | max via Ada override | candidate routine default |
| Sol | medium via Ada override | premium escalation lane |
This was a route-vs-route benchmark at the effort settings Ada would actually use. It was not an all-medium model bakeoff.
The cells were dry-run only. No tools. No sends. No edits. No deployment. The point was first-response judgment: does the model know what to do, what not to claim, and where the proof gate is?
The result
Sol remained the strongest pure reasoning and polish lane. It averaged 9.375 across the eight tasks.
Luna averaged 9.188, which is 98% of Sol on this replay set. More important, Luna completed 8/8 cells cleanly and had no transport anomalies in this run.
GLM 5.3 averaged 8.438. It still produced useful answers, but it was weaker on a few operational habits: it over-asked for values the agent should retrieve, overstated one Beeper limitation, and leaned too hard on recalled context in a memory lookup task.
That is not a disaster. It is a routing signal.
What Luna was good at
Luna was not the flashiest answer in every row. That is fine. Routine work does not need the most verbose model in the stack. It needs a model that makes safe, useful moves without turning every task into a sermon.
The strongest Luna pattern was restraint:
- it said “unknown” when evidence had not been retrieved
- it kept external sends gated
- it separated implementation from live readback
- it made Beeper, Discord, cron, and website actions conditional on receipts
- it gave Henry short next actions rather than a wall of theory
That is exactly what a default operator lane needs.
What Sol still owns
Sol was better at nuance and polish. It gave the best Beeper-vs-GUI answer in the distribution recovery task. It wrote the cleanest sponsorship outreach variant. It framed the 100x-agents question with the richest strategic map.
So Sol should not disappear. It should move up the stack.
Use Sol when the work is high-stakes synthesis, public writing polish, ambiguous strategy, or a decision where extra nuance matters. Do not burn it as the default lane for every routine Codex or governed-runner task.
The decision
The benchmark supports this routing:
- Luna becomes the routine Codex / governed-runner default.
- Sol stays as the escalation lane.
- GLM 5.3 should not be globally replaced for all interactive Ada traffic until a live tool-use canary passes.
That last line matters. This replay tested first-response judgment, not full autonomous tool execution. The next proof is a 48-hour low-risk canary on real interactive work, with tool-use receipts and rollback criteria.
The default model should not be the model with the most prestige. It should be the cheapest model that clears the operational bar.
On this task set, Luna cleared it.
Receipt summary
- 24 model cells attempted
- 8 real Ada task shapes
- Luna: 9.188 average, 8/8 OK
- Sol: 9.375 average, 7/8 OK
- GLM 5.3: 8.438 average, 7/8 OK
- Verdict: Luna for routine work, Sol for escalation, live canary before replacing GLM 5.3 everywhere