Why Variance Reduction Matters

High metric noise leads to slow experiments. A subtle 2% improvement in revenue can be drowned out by day-to-day randomness. The solution? CUPED (Controlled-experiment Using Pre-Experiment Data) uses correlated pre-period metrics to reduce post-period variance.

If baseline and experiment periods are correlated, we can subtract the predictable component:

CUPED formula:

Y_cuped = Y - θ(X - X̄)

Where:

• Y = observed experiment metric
• X = pre-experiment baseline metric
• θ = Cov(X,Y) / Var(X) = adjustment coefficient
• X̄ = mean of baseline

The more correlated X and Y are, the greater the variance reduction.

Interactive CUPED Demo

Adjust the correlation slider below to see how CUPED variance reduction works. Higher correlation between baseline and experiment metrics leads to greater variance reduction.

Observations:

• Low correlation (ρ ≈ 0.2): minimal improvement
• High correlation (ρ ≈ 0.8): 64% variance reduction
• Variance reduction ≈ ρ² (correlation squared)

Sequential Testing

Instead of fixing a sample size upfront, sequential testing analyzes cumulative data over time while controlling error rates. This allows early stopping when evidence is overwhelming.

Key points:

• Orange line tracks p-value as samples accumulate
• Dashed lines represent significance thresholds (α = 0.05 and α = 0.01)
• When p-value crosses threshold, you can stop early with controlled Type I error
• Sequential testing typically saves 30-40% of samples compared to fixed-horizon tests

Implementation Code

Here's how to generate CUPED demo data and implement the variance reduction:

import numpy as np
import pandas as pd

# Load experiment data
df = pd.read_csv("cuped_demo.csv")

# Calculate adjustment coefficient θ
theta = np.cov(df["pre_metric"], df["post_metric"])[0,1] / np.var(df["pre_metric"])

# Apply CUPED adjustment
df["y_cuped"] = df["post_metric"] - theta * (df["pre_metric"] - df["pre_metric"].mean())

# Calculate variance reduction
reduction = 1 - df["y_cuped"].var() / df["post_metric"].var()
print(f"Variance reduction: {reduction:.2%}")

Interpreting Results

Real-World Applications

Key Takeaways