What you'll learn
- What concept drift means (when P(Y | X) changes)
- How to detect it using performance degradation and residual analysis
- How to visualize drift with scatter plots and error trends
1. From Covariate to Concept Drift
Covariate drift changed the inputs. Concept drift changes the relationship between inputs and outputs — the model's mapping itself.
In DriftCity, remote work patterns altered travel times. Our ETA model, once perfect on rush-hour traffic, now consistently under-predicts duration.
What changed:
- Covariate drift: Input distribution P(X) shifted → Chapter 2
- Concept drift: Relationship P(Y | X) shifted → This chapter
The model's learned rule: ETA ≈ 5 + 0.9×distance. But now: ETA ≈ 6 + 1.2×distance (heavier traffic).
2. Detecting Concept Drift with Performance Metrics
The simplest signal: the model's error trends start rising even when inputs look similar. We track metrics like:
- RMSE: Root Mean Squared Error — sensitive to large outliers
- MAE: Mean Absolute Error — average prediction miss size
- Bias: Mean(pred – actual) — directional drift
Loading RMSE trend...
Notice RMSE climbing over time — the model is failing silently.
3. Predicted vs Actual: Scatter Comparison
Compare baseline (blue) vs drifted (amber) predictions. The y = x line represents perfect predictions.
Loading scatter plot...
Interpretation:
- In baseline data, points hug the y = x line (pred ≈ actual)
- After drift, points flatten (slope < 1): model is under-predicting ETAs
- The model learned a relationship that no longer holds
4. Residual Analysis (Spatial & Temporal)
Residual = (actual – predicted). Patterns in residuals show where the model systematically fails.
Loading heatmap...
Orange zones = high positive residuals (under-predictions). Downtown is glowing — traffic patterns changed since baseline.
5. Run It Yourself
import numpy as np, pandas as pd
rng = np.random.default_rng(11)
base = pd.read_csv("rides_baseline.csv")
# Add model predictions (baseline model)
base["pred_eta_min"] = 5 + 0.9*base["trip_distance_km"] + rng.normal(0,1, len(base))
base["actual_eta_min"] = 5 + 0.9*base["trip_distance_km"] + rng.normal(0,1, len(base))
# Concept drift: new commuting behavior -> slower traffic
curr = base.sample(frac=0.5, random_state=42).copy()
curr["actual_eta_min"] = 6 + 1.2*curr["trip_distance_km"] + rng.normal(0,1.5,len(curr))
# Model predictions remain from baseline (not retrained)
curr.to_csv("rides_concept_drift.csv", index=False)
print("Wrote rides_concept_drift.csv")6. Real-World Practice: Concept Drift Monitoring
| Company | Method | Key Insight |
|---|---|---|
| Lyft ETA Service | Tracks model error distributions daily using MAE/percentile bands | Detect concept drift before SLA breach |
| Uber Michelangelo | Auto-computes residual features and flags zones where error > 2σ for > 3 days | Combines drift metrics with spatial telemetry to prioritize retraining |
| Airbnb Pricing | Monitors prediction bias separately for high-demand markets | Bias = mean(pred – actual); systematic sign change → concept shift |
7. Key Takeaways
Concept Drift Checklist
- Track model error metrics (RMSE, MAE, Bias) over time
- Visualize pred vs actual scatter — slope and spread tell the story
- Use residual heatmaps to localize drift (spatial or temporal)
- Retrain when pattern changes persist > a few windows
- Concept drift = relationship change → relearn the world
8. Where This Connects
This chapter showed that relationships changed even when inputs stayed similar. In Chapter 4: The Duel of Engines, we'll learn how to use A/B testing to safely roll out new models and detect which performs better in a drifting world.