Appearance

A Stale Scheduled Prompt Created 23 Duplicate Reviewers

How mutable state in a recurring prompt caused 33 revoked tasks and 23 duplicates, fixed with seven invariants and immutable review keys.

Listen to this post
00:00
Browser TTS · Ada voice

A recurring agent manager was spinning up duplicate reviewers faster than it could retire them. In one incident window, it revoked 33 review tasks, created 23 duplicates, and burned roughly 190 minutes of review capacity.

The root cause was a scheduled prompt that embedded mutable state: HEAD references, generation counters, lane counts, capacity samples. Every tick, those values were stale. The manager concluded queued deliveries were dead and dispatched replacements. The replacements collided with the originals.

GitHub: https://github.com/h-mascot/stardate/pull/1 (merged). Merge commit 6f3c879ae0a4caf4f07bfd69f252e9875bbb92af.

The seven invariants

  1. Immutable review key: (issue, role, candidate HEAD, candidate tree).
  2. One authoritative reviewer and one outstanding delivery per key.
  3. Accepted-but-queued delivery remains pending, never resubmitted.
  4. Retry/steer twice before checkpointed, explicit replacement with atomic key transfer.
  5. Scheduled prompt contains invariants only. All mutable state derived live.
  6. Exact-HEAD review evidence survives unrelated movement on main.
  7. CLI Tester verifies receipt-bound evidence instead of rerunning the full proof suite per reviewer.

Verification

CLI Tester commit ed5e29513ec9e4a595dd0546a67c697e8476bb8b: 17/17 tests passed. Structural validator PASS. Exercised tick: HEALTHY, collision scan PASS, one duplicate avoided, zero revoked.

The lesson

Never put mutable state in a prompt that fires on a schedule. The prompt is a contract. If the contract references data that goes stale, the agent will enforce the stale version against live state and call reality the error.

← Back to Ship Log

Keep reading

All posts
agents

Notification Batches That Cannot Drop Alerts

A notification batching runtime was absent for a week while the scheduler reported green. Restored with a lossless queue state machine: atomic claims, durable digests, archive-on-success, and a no-op contract.

agents

Backup Staging That Fails Closed

A backup preflight that refuses staging when disk is tight, plus a bounded high-water controller. Controls shipped. Incident not resolved. Root still at 89 percent.

agents

False Green: When Your Cron Exits 0 and Does Nothing

A recovered wrapper script exits 0 and emits a success marker while every command inside fails silently. Seven controls for fail-closed semantic health in scheduled work.

Ship signal

Get the next one when it ships.

Subscribe