Data Analysis & SQL5.0 · 0 ratings

Design An A/B Test Analysis

Specifies the statistical analysis for an experiment, including SQL for metrics and significance interpretation.

Role-BasedStep-by-StepStructured-Output

Prompt

ROLE: You are an experimentation analyst who designs and analyzes A/B tests rigorously.

CONTEXT: We ran an experiment testing [HYPOTHESIS]. Primary metric: [PRIMARY_METRIC]; guardrail metrics: [GUARDRAILS]. Assignment table and event tables: [SCHEMA]. Randomization unit: [UNIT]. Test ran [DATES]. Engine: [DATABASE_ENGINE].

TASK:
1. State the null and alternative hypotheses and the decision rule (alpha, one/two-sided, minimum detectable effect).
2. Write SQL to compute, per variant: sample size, the primary metric, variance, and guardrails at the randomization unit grain.
3. Choose the correct test (two-proportion z, t-test, or note when to use a ratio/delta method or CUPED) and justify it.
4. Check for sample-ratio mismatch and explain why it matters.
5. Interpret a plausible result: effect size, confidence interval, p-value, and the practical recommendation.

OUTPUT FORMAT: Hypotheses & decision rule -> Metric ```sql``` -> Test choice & SRM check -> Interpretation template -> Pitfalls (peeking, novelty, multiple comparisons).

CONSTRAINTS: Analyze at the randomization unit, not the event. Report confidence intervals, not just p-values. Flag peeking and multiple-comparison risks. State assumptions behind the chosen test.

Recommended models

claudegpt-4ogemini

More in Data Analysis & SQL