Get early access
Features · Alerting & Incidents · Auto-Recovery
Alerting & Incidents · Auto-Recovery

Most services don't need a human.
They need a restart.

When a service crashes at 03:42, the playbook is usually three steps and you're back. Auto-Recovery runs those three steps in seconds — bounded by limits, audited, and only where you've allowed it.

Without Auto-Recovery

The classic 03:42

  • 03:42:11 php-fpm on api-prod exited with signal 11.
  • 03:42:14 Upstream 5xx rate climbing. Probe failures across two regions.
  • 03:43:02 PagerDuty fires. Phone vibrates.
  • 03:43:48 On-call grabs laptop. SSH. systemctl restart php-fpm.
  • 03:45:14 Service back. Latency normalises.
~3 minutes downtime · 1 page · 1 lost-sleep human. All for a fix the playbook already knew.
With Auto-Recovery

Same incident, no page

  • 03:42:11 php-fpm on api-prod exited with signal 11.
  • 03:42:14 Recovery rule for php-fpm matches. Policy: auto_limited · max 3 / 1h.
  • 03:42:15 Agent runs systemctl restart php-fpm.
  • 03:42:17 Service back. Probes recovering. Cooldown 10 min.
  • 03:42:18 Event recorded in audit log + incident timeline.
6 seconds downtime · 0 pages · 0 woken humans. The page only fires if the loop trips the limit.
— Policies

Three knobs, no surprises.

You decide which services restart themselves and how much rope they get. The default for new rules is conservative.

Policy
Behaviour
Typical use
manual
Recovery is recommended, never executed. The on-call sees a one-click suggestion in the alert.
primary databases
auto_once
One restart attempt within the cooldown window. If the second crash arrives before the cooldown, a human gets paged.
queue workers, edge nginx
auto_limited
Up to N restarts per window. Exceeds the budget → page fires with the full restart history attached.
php-fpm, redis, app workers
— Coverage

Services we know how to nudge.

Each service has a per-type recovery rule that knows the right systemctl unit, the right journalctl filter, and the right grace period. Adding a custom service is a one-form change.

Nnginxrestart
Aapache2graceful
Ccaddyreload
Mmysqlmanual
Ppostgresqlmanual
Rredisrestart
Fphp-fpmrestart
Ddockerrestart
Ssupervisorrestart
+customyour unit

Stop being the restart human.

Bounded, audited, opt-in per service. Six seconds, not three minutes.

Get early access All features