1. Promise

One-sentence promise

After this lecture, you can distinguish data drift from performance drift, set thresholds and alerts that are operationally meaningful, and define retraining triggers with a simple governance workflow.

End tag line

Monitoring is your early warning system.

2. Learning objectives

After reading this lecture, you will be able to:

  • Define data drift and performance drift and explain why they differ.
  • Choose monitoring metrics that match the decision and horizon.
  • Set thresholds and alerts with escalation actions (warn / restrict / stop / retrain).
  • Specify retraining triggers and a lightweight governance loop that a risk function accepts.
  • Produce a minimal monitoring “spec” that can be implemented by engineering.

3. Conceptual core

3.1 Data drift vs performance drift

Data drift (input drift)

Data drift means the distribution of inputs changes: Xt∼Pt(X),Pt≠Pt−k. X_t \sim P_t(X), \quad P_t \neq P_{t-k}.

Examples: - volatility regime changes, - liquidity/spread changes, - feature missingness increases, - new client behaviour changes volumes.

Data drift can happen even if the model still performs well. It is a leading indicator.

Performance drift (output drift)

Performance drift means realised error worsens: Errort=ℓ(yt,ŷt) \text{Error}_t = \ell(y_t, \hat{y}_t) and its distribution changes.

Examples: - RMSE increases, - calibration deteriorates, - hit-rate drops, - drawdowns increase.

Performance drift is a lagging indicator because you need outcomes.

Key distinction

  • Data drift tells you the world changed.
  • Performance drift tells you the model is failing under the new world.

Validator sentence

Monitor inputs to prevent surprises, and monitor outcomes to confirm safety.

3.2 What to monitor (minimal set)

(A) Data integrity checks (hard fails)

  • monotonic timestamps,
  • no duplicates,
  • expected frequency,
  • missingness per feature below threshold,
  • stale values detection (flat-lining).

These are “stop the line” checks.

(B) Data drift metrics (soft alerts)

Simple, defensible choices:

  • rolling mean and std for key features,
  • z-score shift relative to training: zt=μt,window−μtrainσtrain z_t = \frac{\mu_{t,\text{window}} - \mu_{\text{train}}}{\sigma_{\text{train}}}
  • quantile shift (p10/p50/p90),
  • PSI (Population Stability Index) if you want a standard risk metric.

You do not need complex metrics. You need stable ones.

(C) Performance metrics (decision-aligned)

Forecasting: - RMSE/MAE in rolling windows, - directional accuracy (if sign matters), - worst-window metric.

Classification: - AUC stability, - calibration error, - population default rate shift.

Trading decision: - realised Sharpe-like, - drawdown, - turnover and cost.

Rule

Monitor the metric that the business actually cares about.

3.3 Thresholds and alerts

A threshold must have:

  • metric definition,
  • window definition,
  • baseline level (from training/validation),
  • escalation action.

Three-tier scheme (practical)

  • Green: within expected range → continue.
  • Amber: drift warning → restrict usage / increase review.
  • Red: unsafe → disable model / fallback / escalate.

Example (forecast RMSE):

  • Green: RMSE ≤ baseline_mean + 1σ
  • Amber: baseline_mean + 1σ < RMSE ≤ baseline_mean + 2σ
  • Red: RMSE > baseline_mean + 2σ for 3 consecutive windows

Validator sentence

A threshold without an action is not a control.

3.4 Retraining triggers and governance

Retraining is not “whenever performance drops”. Retraining is a controlled change.

Typical retraining triggers

  • performance drift is red for kk windows,
  • sustained data drift (amber for long enough),
  • new data coverage (new instruments / new regime),
  • feature changes or vendor changes,
  • periodic refresh (e.g., monthly) if stable.

Governance loop (lightweight but real)

  1. Detect
  • monitoring alert triggers.
  1. Triage
  • confirm it is not a data pipeline bug.
  1. Decide
  • restrict, rollback, retrain, or accept.
  1. Rebuild
  • new run ID, new dataset snapshot, new config.
  1. Validate
  • same evaluation protocol, plus stability vs previous model.
  1. Approve and deploy
  • sign-off by owner + risk/control function if required.
  1. Document
  • update model card, assumptions, change log.

Validator sentence

Retraining without governance is model drift by another name.

4. Monitoring spec (deliverable format)

A monitoring spec should include:

  • metrics to compute,
  • windows (daily, weekly),
  • thresholds and tiers,
  • alert channels and owners,
  • actions per tier,
  • retraining trigger rules,
  • fallback behaviour.

This is engineering-implementable.

5. Control lens

Monitoring is state estimation. Alerts are constraint violations. Retraining is controller re-identification.

You do not run MPC without: - state estimation, - constraint checks, - fallback mode.

You should not run models without them either.

6. Python lab (simple data drift + performance drift monitoring)

This lab simulates:

  • a model with stable initial regime,
  • then a regime shift that causes drift.

It produces:

  • rolling feature mean shift (data drift),
  • rolling RMSE (performance drift),
  • a simple amber/red alert logic.
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

rng = np.random.default_rng(0)

# ---------------------------------------------------------
# 1) Simulate data with a regime shift
# ---------------------------------------------------------
T = 700
idx = pd.date_range("2024-01-01", periods=T, freq="B")

t = np.arange(T)
x = rng.normal(0, 1, T)
# Drift in feature after midpoint
x[T//2:] = x[T//2:] + 1.0

# Target depends on x with changing relationship
beta = np.ones(T) * 0.6
beta[T//2:] = 0.2
noise = rng.normal(0, 1.0, T)
y = beta * x + noise

df = pd.DataFrame({"x": x, "y": y}, index=idx)

# "Model" is trained on first half: uses beta_hat from train
split = T // 2
train = df.iloc[:split]
test = df.iloc[split:]

beta_hat = float(np.sum(train["x"] * train["y"]) / (np.sum(train["x"]**2) + 1e-12))

# Predictions
df["yhat"] = beta_hat * df["x"]

# ---------------------------------------------------------
# 2) Data drift monitor: rolling mean z-score vs training stats
# ---------------------------------------------------------
mu_train = float(train["x"].mean())
sd_train = float(train["x"].std(ddof=0) + 1e-12)

W = 40
df["x_mean_roll"] = df["x"].rolling(W).mean()
df["x_mean_z"] = (df["x_mean_roll"] - mu_train) / sd_train

# ---------------------------------------------------------
# 3) Performance drift monitor: rolling RMSE
# ---------------------------------------------------------
df["err2"] = (df["y"] - df["yhat"])**2
df["rmse_roll"] = df["err2"].rolling(W).mean().apply(np.sqrt)

# Baseline RMSE distribution from training window
rmse_train_roll = df.iloc[:split]["rmse_roll"].dropna()
rmse_base = float(rmse_train_roll.mean())
rmse_std = float(rmse_train_roll.std(ddof=0) + 1e-12)

# ---------------------------------------------------------
# 4) Threshold logic (simple)
# ---------------------------------------------------------
# Data drift tiers (z-score of rolling mean)
df["data_alert"] = "green"
df.loc[df["x_mean_z"].abs() > 2.0, "data_alert"] = "amber"
df.loc[df["x_mean_z"].abs() > 3.0, "data_alert"] = "red"

# Performance tiers (rolling RMSE)
df["perf_alert"] = "green"
df.loc[df["rmse_roll"] > (rmse_base + 1.0*rmse_std), "perf_alert"] = "amber"
df.loc[df["rmse_roll"] > (rmse_base + 2.0*rmse_std), "perf_alert"] = "red"

# ---------------------------------------------------------
# 5) Plots
# ---------------------------------------------------------
plt.figure(figsize=(8, 3.8))
plt.plot(df.index, df["x_mean_z"], label="data drift: rolling mean z-score")
plt.axhline(2.0, linestyle="--")
plt.axhline(-2.0, linestyle="--")
plt.axhline(3.0, linestyle=":")
plt.axhline(-3.0, linestyle=":")
plt.axvline(df.index[split], linestyle="--", label="regime change")
plt.title("Data drift monitoring")
plt.xlabel("date")
plt.ylabel("z-score")
plt.legend()
plt.tight_layout()
plt.show()

plt.figure(figsize=(8, 3.8))
plt.plot(df.index, df["rmse_roll"], label="performance drift: rolling RMSE")
plt.axhline(rmse_base + 1.0*rmse_std, linestyle="--", label="amber threshold")
plt.axhline(rmse_base + 2.0*rmse_std, linestyle=":", label="red threshold")
plt.axvline(df.index[split], linestyle="--", label="regime change")
plt.title("Performance drift monitoring")
plt.xlabel("date")
plt.ylabel("RMSE (rolling)")
plt.legend()
plt.tight_layout()
plt.show()

# ---------------------------------------------------------
# 6) Minimal alert summary (last 10 days)
# ---------------------------------------------------------
tail = df.tail(10)[["x_mean_z", "rmse_roll", "data_alert", "perf_alert"]]
print("=== Last 10 observations (alerts) ===")
print(tail)

print("\nBaseline rolling RMSE (train):", rmse_base, "std:", rmse_std)
print("Estimated beta_hat from train:", beta_hat)

7. What to say after the lab (interview-grade)

  • “We separate data drift from performance drift.”
  • “Data drift is a leading indicator; performance drift confirms harm.”
  • “We define thresholds relative to training behaviour and use tiers.”
  • “Alerts have actions: warn, restrict, disable, retrain.”
  • “Retraining is governed: new snapshot, new run ID, same protocol, documented change.”

8. Summary

  • Monitoring turns models into controlled systems.
  • Data drift and performance drift answer different questions.
  • Thresholds must map to actions and owners.
  • Retraining triggers require governance and documentation.
  • Simple monitoring beats no monitoring.