home / skills / amnadtaowsoam / cerebraskills / drift-detection

drift-detection skill

safe

/77-mlops-data-engineering/drift-detection

This skill helps teams monitor data, concept, and performance drift and automate alerting and retraining actions.

npx playbooks add skill amnadtaowsoam/cerebraskills --skill drift-detection

Review the files below or copy the command above to add this skill to your agents.

Files (1)

SKILL.md

1.2 KB

---
name: Drift Detection (Alias)
description: Alias skill path for drift detection; points to canonical drift detection + AI observability skills.
---

# Drift Detection

## Overview

This is an **alias skill** so docs can reference `77-mlops-data-engineering/drift-detection`. In this repo, drift guidance is covered by:
- `77-mlops-data-engineering/drift-detection-retraining` (triggering retraining)
- `06-ai-ml-production/ai-observability` (monitoring + alerts)

## Best Practices

- Monitor **data drift** (input distribution), **concept drift** (relationship changes), and **performance drift** (metric degradation).
- Use alert thresholds with rate limiting to avoid paging storms.
- Tie drift signals to actions: investigate → rollback → retrain → redeploy.

## Code Examples

```text
# Canonical references:
77-mlops-data-engineering/drift-detection-retraining/SKILL.md
06-ai-ml-production/ai-observability/SKILL.md
```

## Checklist

- [ ] Choose drift metrics per feature group (KS/PSI/JS, embedding drift, etc.)
- [ ] Define retraining triggers and human approval gates (if needed)
- [ ] Add dashboards + runbooks for drift alerts

## References

- Canonical skill: `77-mlops-data-engineering/drift-detection-retraining/SKILL.md`

Overview

This skill provides an alias path for drift detection guidance and points to the canonical retraining and AI observability resources. It consolidates best practices for detecting data, concept, and performance drift and links the detection signals to operational actions. Use it as a quick reference to the canonical drift retraining and monitoring workflows.

How this skill works

The alias routes users to two canonical capabilities: retraining triggers and AI observability for monitoring and alerts. It highlights what to measure (input distributions, embeddings, model performance) and how to connect alerts to investigation, rollback, retraining, and redeployment. It does not replace the canonical guides but acts as a single entry point to them.

When to use it

You need a single reference that maps drift detection to retraining and monitoring workflows.
Setting up production monitoring where input, concept, or performance drift could affect outcomes.
Designing retraining triggers and human approval gates for models in production.
Configuring alerting thresholds and rate limits to avoid alert storms.
Creating dashboards and runbooks for operational response to drift.

Best practices

Monitor three types of drift: data (input distribution), concept (input→label relationship), and performance (metric degradation).
Select appropriate drift metrics per feature group (KS, PSI, JS, embedding distances) and validate them periodically.
Define clear retraining triggers and include human approval gates when needed to avoid unnecessary model churn.
Use alert thresholds with rate limiting and escalation policies to prevent paging storms.
Tie drift signals to a concrete action plan: investigate → rollback (if needed) → retrain → redeploy, and document each step in runbooks.

Example use cases

Detecting input distribution shifts after a marketing campaign that changes user behavior.
Triggering automated retraining when performance metrics fall below an SLA during nightly evaluations.
Using embedding drift detection to identify changes in user intent for a recommender system.
Configuring observability dashboards and alerts to notify ops teams of rising PSI or JS values.
Establishing a retraining pipeline with human approval gates for high-risk models.

FAQ

Is this skill the canonical source for drift detection?

No. This alias points to the canonical retraining and observability resources that contain the detailed guidance and code examples.

Which types of drift should I prioritize?

Prioritize all three: data drift (inputs), concept drift (relationship changes), and performance drift (metrics). The relative priority depends on your risk profile and model use case.