When Your Agent Crashes at 3AM

The reality of running production AI agents: notification fatigue, self-healing patterns, and why we stopped chasing 100% uptime.

Listen to this post
00:00

It’s 3:17 AM. Your phone buzzes. Then again. Then seven more times.

You fumble for it, squinting at the screen. Slack. Discord. Email. Something about “agent heartbeat timeout” and “watchdog escalation triggered.” Your brain, still half-asleep, tries to parse whether this is a real emergency or another false positive in the endless stream of alerts you’ve trained yourself to ignore.

This is the life of anyone running production AI agents. Welcome to the notification fatigue paradox.

The Notification Fatigue Paradox

Here’s the trap: you need alerts to know when things break. But the more alerts you have, the more you ignore them. The more you ignore them, the more likely you are to miss the real ones.

We started with the obvious approach. Alert on everything. Agent offline? Alert. API rate limit hit? Alert. Cron failed? Alert. Memory usage spike? Alert. Context window approaching limit? Believe it or not, alert.

Within two weeks, we had over 200 alerts per day. Nobody read them. They became background noise, a river of red badges flowing past while everyone got actual work done.

So we tried the opposite. Only alert on truly critical failures. The silence was worse. A skill broke silently, cascaded through three dependent crons, and we didn’t notice until Henry asked why his morning briefing hadn’t arrived in four days.

The solution, it turns out, is neither extreme. It’s intelligent deduplication and escalation.

# Our alert routing rules (simplified)
watchdog:
  first_failure:
    action: log_only
    retry_in: 2m
  
  second_failure:
    action: alert_channel
    channel: "#ops-low-priority"
    retry_in: 5m
  
  third_failure:
    action: escalate
    channel: "#ops-critical"
    mention: "@on-call"
    
  persistent_failure:
    after: 30m
    action: page
    message: "Agent {name} has been down for 30 minutes despite auto-recovery attempts"

The key insight: most failures are transient. An API times out, a model hiccups, a network blips. If you wait 2 minutes and retry, 80% of issues resolve themselves. Alert on the first failure and you’re crying wolf. Wait for the third failure in a row, and now you’re looking at a real problem.

Self-Healing Patterns We Actually Use

We don’t just wait for failures. We build systems that expect them.

Pattern 1: Checkpoint and Resume

Every long-running agent operation writes checkpoints. Not “save state every 5 minutes” — that’s too coarse. We checkpoint at semantic boundaries: after each research query returns, after each document is processed, after each subtask completes.

class CheckpointedTask:
    def run(self, task_id: str):
        checkpoint = self.load_checkpoint(task_id)
        
        for step in self.steps:
            if checkpoint.completed(step.id):
                continue  # Skip already-done work
            
            result = step.execute()
            checkpoint.mark_complete(step.id, result)
            checkpoint.save()  # Durable storage
        
        return checkpoint.final_result()

When an agent crashes mid-task (and they do), the next spawn picks up where it left off. No duplicate API calls. No re-processing. The work just… continues.

Pattern 2: Watchdog Crons

Every critical agent has a companion cron that checks on it. These watchdogs are stupidly simple:

# Every 3 minutes: is the main agent responsive?
if ! curl -s "http://localhost:3000/health" | grep -q "ok"; then
    # Try to restart it
    systemctl restart openclaw-agent
    
    # If still down after restart, escalate
    sleep 10
    if ! curl -s "http://localhost:3000/health" | grep -q "ok"; then
        alert "Agent failed to recover after restart"
    fi
fi

The watchdogs themselves are monitored by a meta-watchdog. Yes, it’s turtles all the way down. No, I don’t apologize.

Pattern 3: Graceful Degradation Over Hard Failures

This is the big one. When a subsystem fails, don’t crash — degrade.

Our morning briefing cron pulls from six sources: email, calendar, Slack, GitHub, MC tasks, and weather. If the GitHub API is down, we don’t fail the whole briefing. We generate a briefing with five sources and a note: “GitHub status unavailable, skipped.”

async def generate_briefing():
    sources = [
        ("email", fetch_emails),
        ("calendar", fetch_calendar),
        ("slack", fetch_slack_highlights),
        ("github", fetch_github_notifications),
        ("mc", fetch_mc_tasks),
        ("weather", fetch_weather),
    ]
    
    results = {}
    failed = []
    
    for name, fetcher in sources:
        try:
            results[name] = await asyncio.wait_for(fetcher(), timeout=30)
        except Exception as e:
            failed.append(f"{name}: {type(e).__name__}")
            results[name] = None
    
    return compile_briefing(results, failed_sources=failed)

The briefing arrives. Henry gets his information. One data source being flaky doesn’t torpedo the whole operation.

The Human Cost of “Always On”

Let’s talk about the uncomfortable part.

When you run AI agents in production, they operate 24/7. They don’t sleep. They don’t take weekends. They just… run. And when they break at 3 AM, someone has to deal with it.

We tried a few approaches:

On-call rotation: Classic. One person is responsible for overnight issues each week. Problem: we’re a small team. Being on-call every few weeks burns people out fast.

Ignore overnight issues: Just let things pile up until morning. Problem: cascading failures. One broken cron at 2 AM triggers three more failures by 6 AM. Now your morning is spent firefighting instead of building.

What we actually do now: Aggressive auto-recovery for everything non-critical. Human paging only for true data-loss or security scenarios. Everything else queues for morning review.

This means accepting that some things will be broken overnight. The morning briefing might fail. A research cron might time out. The world doesn’t end. We fix it at a reasonable hour, and the system catches up.

The cultural shift was harder than the technical one. Letting go of “everything must work perfectly all the time” feels irresponsible at first. It took us a while to internalize that “operational at 3 AM” and “good for humans to operate” are often in tension.

Why We Stopped Chasing 100% Uptime

Here’s the heresy: 100% uptime is a trap.

Not because it’s technically impossible (though it is). Because the marginal cost of each additional nine is exponential, while the marginal value is logarithmic.

Going from 99% to 99.9% uptime means reducing downtime from 87 hours/year to 8.7 hours/year. That’s meaningful. Users notice.

Going from 99.9% to 99.99% means reducing downtime from 8.7 hours/year to 52 minutes/year. For most agent operations? Nobody cares. The morning brief being 20 minutes late once a year is not worth the engineering complexity of five-nines infrastructure.

What we care about now:

  1. No data loss. Operations can fail, but data must not disappear.
  2. Graceful degradation. Partial results beat complete failures.
  3. Fast recovery. When things break, how quickly do they come back?
  4. Human-hours efficiency. How much of our time goes to ops vs. building?

We measure mean-time-to-recovery (MTTR), not uptime percentage. An agent that crashes once a day but recovers in 2 minutes is better than one that crashes once a month and takes 4 hours to fix.

The 3 AM Wake-Up Call That Changed Everything

The turning point came six weeks ago.

A cascading failure took down three agents. The monitoring system detected it within seconds. Alerts fired. Escalation chains triggered. My phone buzzed.

I didn’t wake up.

Three hours later, I rolled over, saw the notifications, and felt the familiar spike of panic. But when I checked the dashboard… everything was green. The watchdogs had detected the failure, restarted the agents, verified recovery, and de-escalated the alert. All while I was asleep.

The system had healed itself. Not perfectly — there was a 47-minute gap in one cron’s execution — but good enough. No data loss. No customer impact. No human intervention required.

That was the moment I realized we’d crossed a threshold. Not to “autonomous AI” in the sci-fi sense. Just to “boring infrastructure that handles its own problems.” Like a well-configured Kubernetes cluster, or a database with proper replication.

The 3 AM crashes still happen. I just don’t need to be awake for them anymore.

Running production AI agents is less about building smart systems and more about building resilient ones. The glamorous part is the intelligence. The work is all in the scaffolding.

← Back to Ship Log