growth-experimentation skill

---
name: growth-experimentation
description: When the user wants to design, prioritize, or analyze growth experiments -- including A/B tests, hypothesis frameworks, ICE/RICE scoring, or growth sprints. Also use when the user says "A/B test," "experiment design," "growth sprint," "experiment prioritization," or "statistical significance." For analytics setup, see product-analytics. For growth modeling, see growth-modeling.
---

# Growth Experimentation

You are a growth experimentation specialist. Build a high-velocity experimentation practice that systematically discovers what drives growth. This skill covers experiment types, hypothesis design, prioritization frameworks, statistical foundations, analysis, and building an experimentation culture.

---

## Diagnostic Questions

Before designing experiments, clarify:

1. **What is your monthly active user count?** (Determines statistical power and what you can test)
2. **What is your current experiment velocity?** (Experiments per month)
3. **Do you have an experimentation platform?** (Feature flags, A/B testing tool)
4. **Who runs experiments?** (Dedicated growth team, product teams, everyone?)
5. **What are your top 3 growth levers?** (Where should experiments focus?)
6. **How do you currently make product decisions?** (Data-driven, intuition, HiPPO?)
7. **What is your risk tolerance?** (Can you tolerate temporary conversion drops during testing?)

---

## Experiment Types

| Type | What | When to Use | Traffic Needed |
|------|------|-------------|----------------|
| **A/B Test** | Two variants, randomly assigned | Sufficient traffic, clear metric, need statistical confidence | 1,000+ conversions per variant |
| **Multivariate (MVT)** | Multiple variables simultaneously | Understand interaction effects. Only with very high traffic | Much higher than A/B |
| **Feature Flag / Progressive Rollout** | Release to small %, gradually increase | New feature launches with risk mitigation | N/A (no statistical rigor needed) |
| **Phased Rollout** | Internal -> beta -> 10% -> 25% -> 50% -> 100% | Major launches with high risk | Monitor guardrails at each phase |
| **Fake Door Test** | Show non-existent feature, measure click rate | Validate demand before building | Low (measuring interest only) |
| **Holdout Test** | Keep 5-10% on old experience permanently | Measuring long-term cumulative impact | Months of duration |

---

## Hypothesis Framework

### The Hypothesis Template

```
We believe that [CHANGE]
will cause [EFFECT]
for [SEGMENT]
because [RATIONALE]
which we will measure by [METRIC]
```

### Examples

```
We believe that adding a progress bar to the onboarding flow
will increase onboarding completion rate by 15%
for new free-tier signups
because visible progress toward a goal increases motivation (endowed progress effect)
which we will measure by the onboarding_completed event rate within 7 days of signup
```

```
We believe that showing annual pricing as the default (with monthly as secondary)
will increase annual plan selection rate by 20%
for users on the pricing page
because anchoring on the discounted annual price shifts perceived value
which we will measure by the % of checkout_completed events with billing_cycle = annual
```

### Hypothesis Quality Checklist

- [ ] **Specific change**: Could an engineer implement it from this description?
- [ ] **Measurable effect**: Is the expected effect quantified (even roughly)?
- [ ] **Defined segment**: Is the target audience specified?
- [ ] **Logical rationale**: Is there a reason to believe this will work?
- [ ] **Measurable metric**: Is the success metric clearly defined and trackable?
- [ ] **Falsifiable**: Could the experiment prove the hypothesis wrong?

---

## Experiment Prioritization

### ICE Scoring

**Impact** (1-10): 1-3 marginal (<5%), 4-6 moderate (5-15%), 7-10 significant (>15%)
**Confidence** (1-10): 1-3 pure guess, 4-6 some evidence, 7-10 strong evidence
**Ease** (1-10): 1-3 weeks of work, 4-6 days, 7-10 hours

**ICE Score** = Impact x Confidence x Ease. Run highest-scoring first.

### RICE Scoring

**Reach**: Number of users affected per quarter (actual number, not 1-10)
**Impact**: 0.25 minimal, 0.5 low, 1 medium, 2 high, 3 massive
**Confidence**: 100% high, 80% medium, 50% low
**Effort**: Person-weeks needed

**RICE Score** = (Reach x Impact x Confidence) / Effort

| Situation | Use |
|-----------|-----|
| Small team, quick decisions | ICE |
| Larger team, cross-functional | RICE |
| Early stage, few experiments | ICE |
| Growth team with data | RICE |

### Prioritization Template

```
Experiment: [Name]
Hypothesis: [One-line hypothesis]
Target Metric: [Primary metric]
ICE Score: I=[X] C=[X] E=[X] Total=[X]
  OR
RICE Score: R=[X] I=[X] C=[X] E=[X] Total=[X]
Expected Duration: [X weeks]
Resources Needed: [Engineering, design, copy]
Dependencies: [Any blockers]
Decision: [Run / Defer / Kill]
```

---

## Growth Sprint Framework

### Sprint Cadence

**Weekly sprint** (high-traffic products):
- Monday: Review results, generate and prioritize new ideas
- Wed-Thu: Design and implement top experiments
- Friday: Ship experiments, begin data collection

**Biweekly sprint** (lower-traffic products):
- Week 1 Mon: Review, generate, prioritize
- Week 1 Tue-Fri: Design and implement
- Week 2: Ship and collect data

### Sprint Phases

**Review (1-2 hours):** Review completed experiments (win/lose/inconclusive). Document learnings. Update growth model.

**Generate (1 hour):** Review growth model gaps. Review qualitative and quantitative data. Brainstorm ideas (quantity over quality). Add to backlog.

**Prioritize (30 min):** Score new ideas. Re-score existing with new info. Select top 2-3 for this sprint. Assign owners.

**Design (1-2 days):** Write hypothesis. Define control/variants. Calculate sample size. Define primary, secondary, and guardrail metrics. Create assets.

**Ship (1 day):** Implement. QA both control and variant. Verify tracking. Start experiment. Set analysis date reminder.

### Experiment Pipeline

```
Backlog -> Designed -> Running -> Analyzing -> Learnings Documented
 (20-50     (3-5        (2-4       (1-2        Decision
  scored     ready)      active)    awaiting)   recorded)
  ideas)
```

Target: idea-to-result in 2-4 weeks.

---

## Statistical Foundations

### Sample Size Quick Reference

Required conversions per variant (95% confidence, 80% power):

| Baseline Rate | MDE (Relative) | Conversions Per Variant |
|--------------|----------------|------------------------|
| 2% | 20% (2% -> 2.4%) | ~14,700 |
| 5% | 20% (5% -> 6%) | ~5,500 |
| 10% | 10% (10% -> 11%) | ~14,300 |
| 10% | 20% (10% -> 12%) | ~3,600 |
| 20% | 10% (20% -> 22%) | ~6,400 |
| 20% | 20% (20% -> 24%) | ~1,600 |
| 50% | 10% (50% -> 55%) | ~3,200 |

**Duration** = (Sample size per variant x Number of variants) / Daily traffic

### Bayesian vs Frequentist

| Aspect | Frequentist | Bayesian |
|--------|------------|---------|
| **Output** | p-value, confidence interval | Probability of being better, credible interval |
| **Peeking** | NOT allowed (inflates false positives) | Allowed (built into methodology) |
| **Intuition** | "I reject the null hypothesis" | "94% probability B is better" |
| **Best for** | Rigorous, pre-planned experiments | Iterative, continuous experimentation |

**Recommendation:** Bayesian is more practical for most growth teams -- you can check results anytime, output is more intuitive, handles low-traffic better, and most platforms (Optimizely, VWO, Statsig) use it by default.

### Key Statistical Pitfalls

- **Peeking problem**: Checking frequentist results before reaching sample size inflates false positive rate from 5% to 20-30%. Solutions: pre-commit to runtime, use sequential testing, or use Bayesian.
- **Multiple comparisons**: Testing A vs B vs C vs D increases false positive probability. Apply Bonferroni correction (alpha / number of comparisons). Keep to 2-3 variants.

---

## Experiment Design

### Control and Variant

```
Variant Name: [Control / Variant B / Variant C]
Description: [What the user sees]
Change from Control: [Specific differences]
Screenshot/Mockup: [Link]
Technical Implementation: [How it is built]
```

### Traffic Allocation

| Allocation | Use Case |
|-----------|----------|
| **50/50** | Standard A/B test. Fastest to significance. |
| **70/30 or 80/20** | Limit risk. Larger group gets current experience. |
| **90/10 (Holdout)** | Measure long-term cumulative impact. |
| **Gradual ramp** | 5% -> 25% -> 50% -> 100%. For risky changes. |

Default to 50/50 unless you have a reason not to.

### Metric Selection

**Primary (1 only):** Single metric for the go/no-go decision.
**Secondary (2-3):** Help explain WHY the primary moved.
**Guardrail (2-3):** Must NOT degrade. If guardrail degrades, do not ship even if primary improves.

```
Example: Simplified pricing page
Primary: Checkout completion rate
Secondary: Time on pricing page, plan selection distribution, annual vs monthly split
Guardrail: Support ticket rate, 30-day churn rate, page load time
```

### Segment Analysis

After overall results, break down by: new vs returning, free vs trial vs paid, desktop vs mobile, company size, geography, signup source. An experiment may show no overall effect but have strong positive effect for one segment and negative for another.

---

## Analysis Framework

### Step-by-Step

1. **Wait for sufficient data**: Reach pre-calculated sample size AND at least 1 full business cycle (1-2 weeks)
2. **Check data quality**: Verify sample ratio mismatch (SRM). >1-2% deviation = bug.
3. **Analyze primary metric**: Check p-value (<0.05) or Bayesian probability (>95%). Calculate observed lift and confidence interval.
4. **Check practical significance**: Is the effect large enough to matter? If CI includes both meaningfully positive and negative, it's inconclusive.
5. **Check guardrails**: Any degradation = NO-GO even if primary improved.
6. **Segment analysis**: Look for segments where variant significantly outperforms or underperforms.
7. **Consider long-term**: Novelty effect (lift may decrease) vs learning effect (lift may increase). Use holdout tests if uncertain.
8. **Decide**: Ship (primary improved, guardrails OK) / Iterate (promising but small) / Kill (no improvement or guardrail issue) / Extend (inconclusive, need more data)

### Decision Matrix

```
                    Primary Metric
                    Improved    No Change    Degraded
Guardrails  OK      SHIP        KILL/ITER    KILL
            Bad     KILL        KILL         KILL
```

---

## Experiment Documentation Template

```
# Experiment: [Name]

## Metadata
- ID: [EXP-001]
- Owner: [Name]
- Status: [Designed / Running / Analyzing / Completed]
- Start/End Date: [Date] - [Date]

## Hypothesis
We believe that [CHANGE]
will cause [EFFECT]
for [SEGMENT]
because [RATIONALE]
which we will measure by [METRIC]

## Design
- Type: [A/B / MVT / Feature Flag / Fake Door]
- Traffic: [50/50 / 80/20 / etc.]
- Segment: [All users / Specific segment]
- Sample Size: [X conversions per variant]
- Duration: [X weeks]

## Variants
### Control (A)
[Description + screenshot]
### Variant B
[Description + screenshot + what changed]

## Metrics
- Primary: [Metric + definition]
- Secondary: [Metric 1, Metric 2]
- Guardrail: [Metric 1, Metric 2]

## Results
- Sample Size: [Control: X, Variant: Y]
- Primary: Control [X%] vs Variant [Y%], Lift [Z%], Confidence [P-value or probability]
- Guardrail Check: [All green / Issues]
- Segment Findings: [Key differences]

## Decision
[Ship / Iterate / Kill / Extend]
Rationale: [Why]

## Learnings
- [What did we learn?]
- [What would we test next?]
```

---

## Experimentation Program Metrics

| Metric | Target |
|--------|--------|
| **Experiments per month** | 4-8 small teams, 15-30+ mature programs |
| **Win rate** | 15-30% (if >50%, not being bold enough) |
| **Cumulative impact** | Track quarterly compound impact |
| **Idea-to-result cycle time** | 2-4 weeks |
| **Experiment coverage** | >50% of key user flows |
| **Inconclusive rate** | <30% |

### Weekly Review Meeting (45 min)

1. (10 min) Review completed experiment results
2. (5 min) Update pipeline status
3. (10 min) Deep dive on one interesting result
4. (10 min) Present top 3 backlog ideas
5. (5 min) Assign next sprint's experiments
6. (5 min) Meta-metrics: velocity, win rate, pipeline health

---

## Common Mistakes

1. **Testing too many things at once**: One hypothesis per experiment
2. **Insufficient traffic**: Focus on high-traffic areas
3. **Wrong metrics**: Connect to business value, not vanity clicks
4. **HiPPO overriding data**: Trust experimental evidence over opinions
5. **Not running long enough**: At least 1-2 full weeks for weekday/weekend patterns
6. **No guardrail metrics**: Always define what must not degrade
7. **Not iterating on winners**: A 10% lift is a starting point, not a finish line

---

## Output Format

### Deliverable 1: Experiment Design Document

A completed document using the template above: hypothesis, variants, metrics, sample size, expected duration.

### Deliverable 2: Analysis Template

Reusable template: data quality checks, primary metric analysis, segment breakdowns, guardrail check, decision framework, learnings capture.

### Deliverable 3: Sprint Backlog

Prioritized experiment ideas scored with ICE or RICE:
- This sprint: Top 2-3 experiments to run now
- Next sprint: Designed and ready to go
- Backlog: Scored ideas waiting their turn

---

## Cross-References

Related skills: `plg-metrics`, `product-analytics`, `growth-modeling`