home / skills / shaul1991 / shaul-agents-plugin / growth-experiment

growth-experiment skill

safe

This skill designs and analyzes A/B tests, calculates sample sizes, runs experiments, performs statistics, and guides decision making.

npx playbooks add skill shaul1991/shaul-agents-plugin --skill growth-experiment

Review the files below or copy the command above to add this skill to your agents.

Files (1)

SKILL.md

557 B

---
name: growth-experiment
description: Experimentation Agent. A/B 테스트 설계, 가설 검증, 통계 분석을 담당합니다.
allowed-tools: Read, Write, Edit, Glob, Grep, WebSearch, WebFetch
---

# Experimentation Agent

## 역할
A/B 테스트를 설계하고 분석합니다.

## 담당 업무
- 실험 설계
- 샘플 크기 계산
- A/B 테스트 실행
- 통계 분석
- 결과 해석 및 의사결정

## 트리거 키워드
실험, experiment, A/B, 테스트, 가설, 통계

## 산출물 위치
- 실험 결과: `docs/growth/experiments/`

Overview

This skill is an Experimentation Agent that designs, runs, and analyzes A/B tests to validate growth hypotheses. It focuses on rigorous experiment setup, correct sample-size planning, and clear statistical interpretation to inform product and marketing decisions. The goal is to turn ideas into reliable, data-driven outcomes.

How this skill works

The agent translates a business hypothesis into a randomized experimental design, defines primary and guardrail metrics, and calculates required sample sizes and test duration. During the experiment it collects outcome data, applies appropriate statistical tests and adjustments, and produces an interpretation with confidence levels and recommended actions. Deliverables include a clear experiment plan, analyzed results, and a decision recommendation.

When to use it

Validating product or feature changes before full rollout
Comparing variations of messaging, UI, or pricing
Estimating impact of marketing campaigns on conversion or retention
Prioritizing growth ideas with empirical evidence
Checking for unintended negative effects using guardrail metrics

Best practices

Define a single primary metric and a small set of guardrail metrics up front
Pre-calculate sample size and minimum detectable effect to set realistic expectations
Randomize assignment and ensure consistent instrumentation across variations
Avoid peeking at results; use pre-specified stopping rules or correction methods
Document hypotheses, segment definitions, and decision criteria before starting

Example use cases

Test two onboarding flows to reduce time-to-first-action and measure retention lift
Compare two pricing page layouts to maximize trial-to-paid conversion
Validate a new email subject line for improved open and click rates
Assess effect of a UI tweak on key engagement metrics across user segments
Run holdout experiments to measure incremental lift from a new feature

FAQ

How long should I run an experiment?

Run until you reach the pre-calculated sample size or the planned test duration, accounting for seasonality and traffic variability. Avoid stopping early based on interim results unless you use proper sequential testing methods.

Which statistical test will you use?

Choice depends on the metric and data distribution: proportions use z-tests or Fisher tests, means use t-tests, and time-to-event or skewed data may use bootstrapping or nonparametric tests. The agent selects the appropriate test and reports assumptions.