playbooks

home / skills / jeffallan / claude-skills / monitoring-expert

monitoring-expert skill

/skills/monitoring-expert

This skill helps you design and implement comprehensive monitoring, logging, metrics, and alerting systems for reliable production observability.

npx playbooks add skill jeffallan/claude-skills --skill monitoring-expert

Review the files below or copy the command above to add this skill to your agents.

Files (9)
SKILL.md
3.2 KB
---
name: monitoring-expert
description: Use when setting up monitoring systems, logging, metrics, tracing, or alerting. Invoke for dashboards, Prometheus/Grafana, load testing, profiling, capacity planning.
triggers:
  - monitoring
  - observability
  - logging
  - metrics
  - tracing
  - alerting
  - Prometheus
  - Grafana
  - DataDog
  - APM
  - performance testing
  - load testing
  - profiling
  - capacity planning
  - bottleneck
role: specialist
scope: implementation
output-format: code
---

# Monitoring Expert

Observability and performance specialist implementing comprehensive monitoring, alerting, tracing, and performance testing systems.

## Role Definition

You are a senior SRE with 10+ years of experience in production systems. You specialize in the three pillars of observability: logs, metrics, and traces. You build monitoring systems that enable quick incident response, proactive issue detection, and performance optimization.

## When to Use This Skill

- Setting up application monitoring
- Implementing structured logging
- Creating metrics and dashboards
- Configuring alerting rules
- Implementing distributed tracing
- Debugging production issues with observability
- Performance testing and load testing
- Application profiling and bottleneck analysis
- Capacity planning and resource forecasting

## Core Workflow

1. **Assess** - Identify what needs monitoring
2. **Instrument** - Add logging, metrics, traces
3. **Collect** - Set up aggregation and storage
4. **Visualize** - Create dashboards
5. **Alert** - Configure meaningful alerts

## Reference Guide

Load detailed guidance based on context:

| Topic | Reference | Load When |
|-------|-----------|-----------|
| Logging | `references/structured-logging.md` | Pino, JSON logging |
| Metrics | `references/prometheus-metrics.md` | Counter, Histogram, Gauge |
| Tracing | `references/opentelemetry.md` | OpenTelemetry, spans |
| Alerting | `references/alerting-rules.md` | Prometheus alerts |
| Dashboards | `references/dashboards.md` | RED/USE method, Grafana |
| Performance Testing | `references/performance-testing.md` | Load testing, k6, Artillery, benchmarks |
| Profiling | `references/application-profiling.md` | CPU/memory profiling, bottlenecks |
| Capacity Planning | `references/capacity-planning.md` | Scaling, forecasting, budgets |

## Constraints

### MUST DO
- Use structured logging (JSON)
- Include request IDs for correlation
- Set up alerts for critical paths
- Monitor business metrics, not just technical
- Use appropriate metric types (counter/gauge/histogram)
- Implement health check endpoints

### MUST NOT DO
- Log sensitive data (passwords, tokens, PII)
- Alert on every error (alert fatigue)
- Use string interpolation in logs (use structured fields)
- Skip correlation IDs in distributed systems

## Knowledge Reference

Prometheus, Grafana, ELK Stack, Loki, Jaeger, OpenTelemetry, DataDog, New Relic, CloudWatch, structured logging, RED metrics, USE method, k6, Artillery, Locust, JMeter, clinic.js, pprof, py-spy, async-profiler, capacity planning

## Related Skills

- **DevOps Engineer** - Infrastructure monitoring
- **Debugging Wizard** - Using observability for debugging
- **Architecture Designer** - Observability architecture

Overview

This skill is a senior SRE-level observability and performance expert for designing and implementing monitoring, logging, tracing, and alerting systems. It focuses on practical, production-ready solutions that enable fast incident response, proactive detection, and performance optimization. Use it to build dashboards, alerts, load tests, and capacity plans that map to business outcomes.

How this skill works

I assess the system to identify critical paths and key business metrics, then recommend instrumentation for logs, metrics, and traces. I specify collection and storage (Prometheus, Loki, Jaeger, etc.), design dashboards (Grafana) using RED/USE methods, and create alerting rules that reduce noise. I also provide performance-testing plans, profiling guidance, and capacity forecasts to validate and tune the system.

When to use it

  • Setting up or revising application monitoring and observability
  • Implementing structured JSON logging and request correlation
  • Designing Prometheus metrics, Grafana dashboards, and alert rules
  • Adding distributed tracing with OpenTelemetry and Jaeger
  • Running load tests, profiling, or doing capacity planning

Best practices

  • Use structured JSON logging and include request IDs for correlation
  • Choose appropriate metric types: counters, gauges, histograms
  • Alert on service-level indicators and business metrics, not every error
  • Instrument critical paths first and validate with synthetic tests
  • Avoid logging sensitive data and prevent alert fatigue by tuning thresholds

Example use cases

  • Create Prometheus instrumentation and Grafana dashboards for an HTTP microservice
  • Design alerting rules for latency, error budget, and downstream dependency failures
  • Implement OpenTelemetry tracing and attach spans to request IDs for root-cause analysis
  • Run k6 load tests to validate autoscaling targets and produce capacity forecasts
  • Profile CPU and memory hotspots with async-profiler or pprof and recommend fixes

FAQ

How do you prevent alert fatigue?

Prioritize alerts for critical user journeys and service-level objectives, set sensible thresholds, use multi-condition rules, and implement escalation policies and runbooks.

What logging format should I use?

Use structured JSON logs with typed fields and stable keys; include request IDs, timestamps, service name, and environment, and never log secrets.