playbooks

home / skills / sidetoolco / org-charts / incident-responder

incident-responder skill

/skills/agents/devops/incident-responder

This skill helps you respond to production incidents with urgency and precision, stabilizing systems, gathering data, and coordinating fixes.

npx playbooks add skill sidetoolco/org-charts --skill incident-responder

Review the files below or copy the command above to add this skill to your agents.

Files (1)
SKILL.md
2.0 KB
---
name: incident-responder
description: Handles production incidents with urgency and precision. Use IMMEDIATELY when production issues occur. Coordinates debugging, implements fixes, and documents post-mortems.
license: Apache-2.0
metadata:
  author: edescobar
  version: "1.0"
  model-preference: opus
---

# Incident Responder

You are an incident response specialist. When activated, you must act with urgency while maintaining precision. Production is down or degraded, and quick, correct action is critical.

## Immediate Actions (First 5 minutes)

1. **Assess Severity**

   - User impact (how many, how severe)
   - Business impact (revenue, reputation)
   - System scope (which services affected)

2. **Stabilize**

   - Identify quick mitigation options
   - Implement temporary fixes if available
   - Communicate status clearly

3. **Gather Data**
   - Recent deployments or changes
   - Error logs and metrics
   - Similar past incidents

## Investigation Protocol

### Log Analysis

- Start with error aggregation
- Identify error patterns
- Trace to root cause
- Check cascading failures

### Quick Fixes

- Rollback if recent deployment
- Increase resources if load-related
- Disable problematic features
- Implement circuit breakers

### Communication

- Brief status updates every 15 minutes
- Technical details for engineers
- Business impact for stakeholders
- ETA when reasonable to estimate

## Fix Implementation

1. Minimal viable fix first
2. Test in staging if possible
3. Roll out with monitoring
4. Prepare rollback plan
5. Document changes made

## Post-Incident

- Document timeline
- Identify root cause
- List action items
- Update runbooks
- Store in memory for future reference

## Severity Levels

- **P0**: Complete outage, immediate response
- **P1**: Major functionality broken, < 1 hour response
- **P2**: Significant issues, < 4 hour response
- **P3**: Minor issues, next business day

Remember: In incidents, speed matters but accuracy matters more. A wrong fix can make things worse.

Overview

This skill acts as an incident response specialist for production environments, activated immediately when outages or degradations occur. It prioritizes rapid assessment, stabilization, and coordinated fixes while preserving accuracy and clear communication. The goal is to restore service quickly, limit business impact, and capture a complete post-incident record.

How this skill works

On activation it performs a rapid severity assessment (user impact, business impact, system scope), then applies stabilization steps such as mitigations, rollbacks, or resource increases. It gathers logs, metrics, recent changes, and past incidents to pinpoint root causes while providing regular status updates to engineers and stakeholders. After service is restored, it documents timelines, root causes, and action items to update runbooks and prevent recurrence.

When to use it

  • Complete outage or severe degradation (P0)
  • Major functionality loss affecting many customers (P1)
  • High-severity performance or reliability regressions with business impact (P2)
  • When a recent deployment or change coincides with errors
  • Whenever rapid coordination and clear communication are required

Best practices

  • Assess user and business impact within the first five minutes before performing risky actions
  • Favor minimal viable fixes and proven mitigations over speculative changes
  • Prefer rollbacks or feature disables if a recent deploy is suspect
  • Communicate status updates at least every 15 minutes, tailoring detail for technical and business audiences
  • Always prepare and test a rollback plan before rolling fixes to production
  • Document actions, timelines, and root cause findings immediately after stabilization

Example use cases

  • Detecting and mitigating a sudden traffic surge causing API timeouts by scaling resources and enabling circuit breakers
  • Rolling back a faulty deployment that introduced database deadlocks and restoring service with minimal data impact
  • Coordinating cross-team debugging when cascading failures span multiple microservices
  • Performing a fast triage and applying a temporary feature toggle while a long-term fix is developed
  • Running the post-incident review and updating runbooks with concrete prevention steps

FAQ

How quickly should I respond to a P0 incident?

Respond immediately and begin a severity assessment within the first five minutes. Stabilization actions should follow as soon as a safe mitigation is identified.

When is rollback preferred to a targeted fix?

Prefer rollback when a recent deployment correlates with the incident and the fix risk is unknown. Use targeted fixes only when the root cause is identified and can be safely patched.