Agent Reliability Scorecard

This is the corrected version of the reliability report.

The first pass collapsed one Hermes operator and a multi-agent OpenClaw fleet into two runtime buckets. That was useful for finding a problem, but it was not the clean comparison. The cleaner comparison is agent by agent.

So this version removes the framework-war framing and shows the production numbers across agents:

• Book on Hermes.
• Ada on OpenClaw.
• Spock on OpenClaw.
• Zora on OpenClaw.
• Scotty on OpenClaw.
• Smaller OpenClaw local and Midas slices are kept visible instead of hidden.

TL;DR

• Observation window: 2026-05-09 13:41:58 UTC to 2026-05-16 13:42:28 UTC.
• Unit of comparison: agent identity.
• Data source: production telemetry, not synthetic prompts.
• Status labels: success, fail, unknown.
• Known-session success: success / (success + fail).
• Unknown sessions are shown separately and are not counted as success.
• Best large-volume agent: Spock, 563 sessions, 96.4% known-session success.
• Highest-volume agent: Ada, 762 sessions, 94.2% known-session success.
• Lowest large-volume agent: Book, 352 sessions, 27.4% known-session success.
• Worst OpenClaw agent slice: Scotty, 127 sessions, 56.7% known-session success.

Agent scorecard

Agent mapping

The comparison uses agent identity first. Runtime is only metadata beside the agent.

What the numbers say

The main point is not “Hermes vs OpenClaw.”

The main point is that agent identity changes the picture:

• Spock and Ada were high-volume, high-success OpenClaw agents.
• Scotty was a clear OpenClaw outlier.
• Zora was middling and lower-volume.
• Book/Hermes had the weakest final-session outcome in this window.
• A runtime-level average would hide the difference between Ada/Spock and Scotty.

Small-sample rows

Midas and OpenClaw local are shown because hiding them would make the table look cleaner than the telemetry. They should not drive the headline.

Methodology

The public JSON behind this article is here: agent-results-summary.json.

The collector classified each session into one of three labels:

This is production telemetry. It does not normalize task type, task difficulty, model route, or human satisfaction. It answers a narrower question:

In this seven-day slice, which agent sessions ended cleanly, failed, or remained unresolved?

Model route coverage

The first scorecard did not spell this out clearly enough: model-route coverage is uneven across the two runtimes.

Hermes session telemetry captured model names. The OpenClaw session metadata harvested for this seven-day slice did not include model route, so OpenClaw rows are marked as unknown/not captured in the public JSON instead of pretending we know.

So: this post compares observed session outcomes by agent. It does not claim the agents were using equivalent model routes.

Failed-session shape

The harvested data also does not fully normalize task type. For Book/Hermes, source and some titles were captured. For OpenClaw, this harvest mostly preserved session files, hosts, profiles, status, and failure markers, but not user-facing task titles.

That means the safest public breakdown is failure shape, not full intent taxonomy.

The public JSON now carries these fields per agent:

• model_distribution
• failed_model_distribution
• failed_source_distribution
• failed_session_failure_kinds

Final read

The corrected read is simple:

• Do not compare Book against a whole OpenClaw fleet average.
• Compare Book, Ada, Spock, Zora, Scotty, and smaller slices directly.
• Put the runtime icon beside the agent, but keep the agent as the row.
• Treat framework-level aggregation as secondary.

That is the more honest scorecard. Less dramatic. More useful.