playbooks

home / skills / sanity-io / agent-toolkit / content-experimentation-best-practices

content-experimentation-best-practices skill

/skills/content-experimentation-best-practices

This skill helps you implement and analyze content experiments using A/B and multivariate testing to optimize engagement and conversions.

npx playbooks add skill sanity-io/agent-toolkit --skill content-experimentation-best-practices

Review the files below or copy the command above to add this skill to your agents.

Files (5)
SKILL.md
1.6 KB
---
name: content-experimentation-best-practices
description: A/B testing and content experimentation methodology for data-driven content optimization. Use when implementing experiments, analyzing results, or building experimentation infrastructure.
license: MIT
metadata:
  author: sanity
  version: "1.0.0"
---

# Content Experimentation Best Practices

Principles and patterns for running effective content experiments to improve conversion rates, engagement, and user experience.

## When to Apply

Reference these guidelines when:
- Setting up A/B or multivariate testing infrastructure
- Designing experiments for content changes
- Analyzing and interpreting test results
- Building CMS integrations for experimentation
- Deciding what to test and how

## Core Concepts

### A/B Testing
Comparing two variants (A vs B) to determine which performs better.

### Multivariate Testing
Testing multiple variables simultaneously to find optimal combinations.

### Statistical Significance
The confidence level that results aren't due to random chance.

### Experimentation Culture
Making decisions based on data rather than opinions (HiPPO avoidance).

## Resources

See `resources/` for detailed guidance:
- `resources/experiment-design.md` — Hypothesis framework, metrics, sample size, and what to test
- `resources/statistical-foundations.md` — p-values, confidence intervals, power analysis, Bayesian methods
- `resources/cms-integration.md` — CMS-managed variants, field-level variants, external platforms
- `resources/common-pitfalls.md` — 17 common mistakes across statistics, design, execution, and interpretation

Overview

This skill provides practical A/B testing and content experimentation methodology to drive data-informed content decisions. It focuses on experiment design, measurement, and infrastructure patterns that improve conversion, engagement, and user experience. Use it to standardize how experiments are planned, run, and interpreted across teams.

How this skill works

It outlines what to test, how to randomize and segment traffic, and which metrics and statistical checks to apply. The guidance covers A/B and multivariate test setups, minimum sample sizing, significance testing, and safeguards against bias. It also recommends CMS and engineering integration patterns to run experiments reliably and capture key events for analysis.

When to use it

  • Designing A/B tests for page copy, CTAs, or layout changes
  • Setting up multivariate experiments to evaluate combinations of content elements
  • Building CMS or feature-flag integrations for experiment delivery
  • Analyzing experiment results and determining statistical significance
  • Prioritizing content hypotheses and deciding what to measure

Best practices

  • Define clear hypotheses with primary and secondary metrics before testing
  • Calculate sample size and test duration up front to avoid underpowered results
  • Use consistent, instrumentation-driven event tracking to ensure data quality
  • Segment results by relevant cohorts but avoid over-splitting that underpowers tests
  • Apply stopping rules and guardrails to prevent peeking and false positives
  • Document learnings and rollouts to create an experiment knowledge base

Example use cases

  • Test headline and subheadline combinations to lift conversion on a landing page
  • Evaluate multiple CTA texts and colors with multivariate testing to find the best combination
  • Integrate experiment flags in the CMS to toggle variants without deployments
  • Compare long-form versus short-form content to measure engagement and retention
  • Validate personalization rules by testing curated content blocks for different user segments

FAQ

How long should an experiment run?

Run until you reach the precomputed sample size and test duration to capture typical traffic cycles; avoid stopping early based on interim results.

When should I use multivariate tests instead of A/B?

Use multivariate testing when you need to evaluate combinations of independent content elements, but ensure traffic volume is sufficient to power all variant combinations.