Module 104 — Evaluation discipline for quantitative models — Quant Model Playbook

1. Promise

One-sentence promise

After this lecture, you can evaluate models with time-series cross-validation, test stability across subperiods and regimes, and perform error analysis that explains where the model fails and why.

End tag line

Average performance is not evidence. Stability is.

2. Learning objectives

After reading this lecture, you will be able to:

Implement rolling and expanding time-series cross-validation.
Add purge/embargo logic when labels overlap.
Measure stability across subperiods and regime slices.
Diagnose model errors by time, by volatility, and by feature conditions.
Convert “it fails” into a testable hypothesis about why it fails.

3. Prerequisites and setup

Required maths and prior knowledge

Mean and variance.
Basic understanding of train/validation/test by time.
Familiarity with RMSE/MAE/AUC and simple strategies.

Python environment assumptions

Python 3.9+ in a notebook.

Packages used

numpy, pandas, matplotlib

No external data is required.

4. Conceptual core

4.1 The key idea

Evaluation is a protocol, not a number.

You must show:

Performance under a time-consistent CV scheme.
Stability across time and regimes.
Error anatomy: where it fails and why.

A single global metric can hide catastrophic failure modes.

4.2 Time-series cross-validation (rolling / expanding)

Setup

Data: $\{(X_t, Y_{t+h})\}_{t=1}^{T-h}$ .

We evaluate a model family $f_\theta$ by repeated fits on a past window and tests on a future window.

Rolling window CV

Let training length be $L$ and test block length be $B$ .

Fold $k$ uses:

Train: $[t_k, t_k + L - 1]$
Test: $[t_k + L, t_k + L + B - 1]$

Then advance $t_{k+1} = t_k + B$ .

This keeps training size fixed and adapts to changing regimes.

Expanding window CV

Fold $k$ uses:

Train: $[1, t_k]$
Test: $[t_k+1, t_k+B]$

This mimics “learning over time” with accumulating history.

Purge and embargo (if needed)

If labels overlap due to horizon $h$ or windowed targets:

Purge training samples whose label window overlaps the test window.
Embargo a gap of $e$ observations after the test start.

Minimal rule of thumb:

If $Y_{t+h}$ uses information from $[t+1, t+h]$ , purge at least $h$ points near boundaries.

Validator sentence

If you cannot describe your CV split in indices, you do not have an evaluation.

4.3 Stability checks (subperiods, regimes)

A stable model should not rely on one lucky window.

You test stability by slicing performance:

(1) Subperiod slices

Split time into $S$ contiguous blocks (e.g., years, quarters).

Compute metric per block: $m_s = \mathrm{Metric}\big(\{(y_t,\hat{y}_t)\}_{t\in \text{block } s}\big).$

Then report: - mean of $m_s$ , - dispersion (std, IQR), - worst block.

A robust model has acceptable worst-block behaviour.

(2) Regime slices

Define a regime indicator $R_t$ (e.g., volatility quartiles).

Example: $R_t = \begin{cases} \text{low vol} & \sigma_t \le q_{25} \\ \text{mid vol} & q_{25}<\sigma_t\le q_{75} \\ \text{high vol} & \sigma_t > q_{75} \end{cases}$

Evaluate metrics conditional on regime: $m(r)=\mathbb{E}[\ell(Y,\hat{Y}) \mid R=r].$

Finance-relevant regimes: - high vs low volatility, - trend vs mean-reversion periods, - crisis vs calm windows.

Validator sentence

If the model fails in one regime that matters commercially, it is not deployable.

4.4 Error analysis (where it fails, why)

Error analysis converts residuals into insight.

Residuals

For regression: $e_t = y_t - \hat{y}_t.$

You ask:

When are $|e_t|$ large?
Are errors asymmetric? (systematic under/over prediction)
Are errors clustered? (regime dependence)
Are errors linked to certain feature ranges? (non-linearity, missing interactions)

Useful decompositions

Error by time:

Plot $|e_t|$ over time.
Mark crisis windows.

Error by volatility:

Bucket by realised $\sigma_t$ .
Compare MAE/RMSE by bucket.

Error by sign:

Separate $y_t>0$ vs $y_t<0$ or large moves vs small moves.

Error by feature quantile:

For a feature $x$ , compare MAE for bottom 20% vs top 20%.
This reveals monotonicity violations and missing interactions.

Root-cause hypotheses (typical)

The mapping $X\to Y$ changes across regimes.
The model is too smooth (misses jumps) or too reactive (fits noise).
Key feature availability shifts (staleness, missing values).
Execution assumptions differ from reality (for trading decisions).

5. Control lens

5.1 Analogue and failure mode

Control analogue

You do not validate a controller on one trajectory.

You validate across disturbances, operating points, and parameter variations.

Finance analogue

A model must be evaluated across regimes and stress windows, not only on average.

6. Worked example (evaluation narrative)

Use case:

Forecast next-day return sign for a simple directional rule.

Evaluation protocol:

Time-series CV with rolling folds.
Compute fold metrics.
Slice performance by volatility regime.
Inspect errors in high-vol windows.

Interpretation:

A model that only works in calm markets is a conditional strategy, not a general model.
If the business needs crisis robustness, you must either redesign the model or constrain usage.

7. Python lab (rolling CV + stability + error anatomy)

This lab creates a synthetic series with a regime change. It then compares a baseline (last value) vs a simple linear lag model.

You get three figures:

Fold-by-fold RMSE across time.
Metric by volatility regime.
Absolute error over time (shows failure window).

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

rng = np.random.default_rng(0)

# -----------------------------
# Synthetic data with regime change
# -----------------------------
T = 700
eps = rng.normal(0, 1.0, T)

y = np.zeros(T)

# Regime 1: mild persistence, low noise
for t in range(1, 350):
    y[t] = 0.25 * y[t-1] + 0.6 * eps[t]

# Regime 2: weaker persistence, higher noise (stress regime)
for t in range(350, T):
    y[t] = 0.05 * y[t-1] + 1.4 * eps[t]

df = pd.DataFrame({"y": y})
df["lag1"] = df["y"].shift(1)
df["lag2"] = df["y"].shift(2)
df["lag3"] = df["y"].shift(3)

# Rolling realised volatility proxy
Lvol = 20
df["vol"] = df["y"].diff().rolling(Lvol).std()

df = df.dropna().reset_index(drop=True)

def rmse(y_true, y_hat):
    y_true = np.asarray(y_true); y_hat = np.asarray(y_hat)
    return float(np.sqrt(np.mean((y_true - y_hat) ** 2)))

def fit_ols(X, y):
    # closed-form via lstsq
    beta = np.linalg.lstsq(X, y, rcond=None)[0]
    return beta

def predict_ols(X, beta):
    return X @ beta

# -----------------------------
# Rolling CV parameters
# -----------------------------
L = 220   # train length
B = 60    # test block length

fold_starts = list(range(0, len(df) - (L + B) + 1, B))

fold_rmse_last = []
fold_rmse_lin = []
fold_midpoints = []

all_errors = []  # collect per-time errors for later plotting
all_times = []

for s in fold_starts:
    train = df.iloc[s:s+L]
    test  = df.iloc[s+L:s+L+B]

    # Baseline: last value
    pred_last = test["lag1"].values

    # Linear lags model
    X_tr = np.column_stack([
        np.ones(len(train)),
        train["lag1"].values,
        train["lag2"].values,
        train["lag3"].values,
    ])
    y_tr = train["y"].values
    beta = fit_ols(X_tr, y_tr)

    X_te = np.column_stack([
        np.ones(len(test)),
        test["lag1"].values,
        test["lag2"].values,
        test["lag3"].values,
    ])
    pred_lin = predict_ols(X_te, beta)

    fold_rmse_last.append(rmse(test["y"].values, pred_last))
    fold_rmse_lin.append(rmse(test["y"].values, pred_lin))
    fold_midpoints.append(s + L + B/2)

    # store abs errors for anatomy
    abs_e_last = np.abs(test["y"].values - pred_last)
    abs_e_lin  = np.abs(test["y"].values - pred_lin)
    all_errors.append(pd.DataFrame({
        "abs_e_last": abs_e_last,
        "abs_e_lin": abs_e_lin,
        "vol": test["vol"].values
    }))
    all_times.append(test.index.values)

err_df = pd.concat(all_errors, ignore_index=True)

# -----------------------------
# Figure 1: Fold metrics over time
# -----------------------------
plt.figure(figsize=(8, 4))
plt.plot(fold_midpoints, fold_rmse_last, marker="o", label="Last-value baseline RMSE")
plt.plot(fold_midpoints, fold_rmse_lin,  marker="o", label="Linear-lags RMSE")
plt.axvline(x=350, linestyle="--", label="Regime change (approx)")
plt.title("Rolling CV: fold-by-fold RMSE shows stability (or lack of it)")
plt.xlabel("Time index (approx)")
plt.ylabel("RMSE (per fold)")
plt.legend()
plt.tight_layout()
plt.show()

# -----------------------------
# Figure 2: Metric by volatility regime
# -----------------------------
# Define regimes by volatility quantiles
q1, q3 = np.nanquantile(err_df["vol"], [0.25, 0.75])
regime = pd.cut(
    err_df["vol"],
    bins=[-np.inf, q1, q3, np.inf],
    labels=["low vol", "mid vol", "high vol"]
)
err_df["regime"] = regime

reg_summary = err_df.groupby("regime")[["abs_e_last", "abs_e_lin"]].mean()

plt.figure(figsize=(7, 3.6))
plt.plot(reg_summary.index, reg_summary["abs_e_last"].values, marker="o", label="Last-value MAE (by regime)")
plt.plot(reg_summary.index, reg_summary["abs_e_lin"].values,  marker="o", label="Linear-lags MAE (by regime)")
plt.title("Stability check: average absolute error by volatility regime")
plt.xlabel("Regime (volatility buckets)")
plt.ylabel("Mean absolute error")
plt.legend()
plt.tight_layout()
plt.show()

print("=== Error by regime (mean abs error) ===")
print(reg_summary)

# -----------------------------
# Figure 3: Error anatomy over time (one long test stream)
# -----------------------------
# Rebuild one contiguous test stream for plotting
# (Use the last fold's test window as a simple, clean view)
last_s = fold_starts[-1]
train = df.iloc[last_s:last_s+L]
test  = df.iloc[last_s+L:last_s+L+B]

pred_last = test["lag1"].values

X_tr = np.column_stack([np.ones(len(train)), train["lag1"], train["lag2"], train["lag3"]])
beta = fit_ols(X_tr, train["y"].values)

X_te = np.column_stack([np.ones(len(test)), test["lag1"], test["lag2"], test["lag3"]])
pred_lin = predict_ols(X_te, beta)

abs_e_last = np.abs(test["y"].values - pred_last)
abs_e_lin  = np.abs(test["y"].values - pred_lin)

plt.figure(figsize=(8, 4))
plt.plot(abs_e_last, label="|error| last-value")
plt.plot(abs_e_lin,  label="|error| linear-lags")
plt.title("Error anatomy: where does it fail in the test window?")
plt.xlabel("Test time index (within fold)")
plt.ylabel("Absolute error")
plt.legend()
plt.tight_layout()
plt.show()

8. Diagnostics and failure modes (evaluation checklist)

8.1 CV discipline

Are folds strictly time-ordered?
Are validation folds used for model selection only?
Is test kept for a final report?
Do labels overlap across fold boundaries (need purge/embargo)?

8.2 Stability discipline

Do you report metric per subperiod and worst block?
Do you report performance in stress-like regimes?
Do you have a “red zone” regime that must meet a minimum standard?

8.3 Error analysis discipline

Do you plot residuals vs time and vs volatility?
Do you check asymmetry (systematic bias)?
Do you check errors by feature quantile (nonlinearities)?
Do you link failure windows to market conditions or data quality?

9. Mini-exercises

Exercise 1

Why are random CV splits wrong for most financial time series?

Solution They leak temporal structure and regimes. They mix future conditions into training.

Exercise 2

Your model performs well on average but fails in high-volatility regimes. What are two valid responses?

Solution Restrict model usage to regimes where it is validated. Or redesign and retrain with regime robustness as a requirement.

Exercise 3 (trap)

You used rolling CV. Therefore the model is robust. Agree or disagree.

Solution Disagree. Rolling CV is necessary, not sufficient. You still need subperiod/regime stability checks and error anatomy.

10. Summary

Use rolling or expanding CV for time series.
Evaluate stability across subperiods and regimes.
Perform error analysis to identify failure modes.
A model that fails where it matters is not deployable, regardless of its average score.

11. Out of scope

Full statistical significance testing (SPA tests, reality checks).
Advanced dependence-aware bootstrap methods.
Full transaction cost and execution modelling.