1. Promise

One-sentence promise

After this lecture, you can assess robustness via parameter sensitivity, subperiod stability, and simple stress tests, and you can convert results into safe-usage and monitoring language.

End tag line

Robust models survive small shocks.

2. Learning objectives

After reading this lecture, you will be able to:

  • Run sensitivity analysis on key parameters (horizon, window sizes, thresholds).
  • Evaluate stability across subperiods and regimes, not only on average.
  • Perform simple stress tests and scenario analysis in a defensible way.
  • Summarise robustness evidence in a format a validator or stakeholder accepts.

3. Conceptual core

3.1 Sensitivity to parameters

What it means

Your pipeline has parameters that are not “learned” but chosen:

  • horizon hh,
  • feature windows (e.g., rolling vol window),
  • thresholds (e.g., trade if score > 0),
  • universe filters (liquidity cutoff),
  • regularisation strength λ\lambda.

Robustness asks:

If you change these within a reasonable range, does performance:

  • stay similar,
  • degrade smoothly,
  • or collapse unpredictably?

Interpretation

  • Smooth degradation is acceptable.
  • Performance cliffs indicate fragility and overfitting to a choice.

Validator sentence

If one “magic number” makes the result, you do not have a result.

3.2 Stability across subperiods

Why it matters

Markets are not stationary. You must show that performance is not concentrated in one lucky interval.

Subperiod views:

  • by calendar (years/quarters),
  • by volatility regime (low/high),
  • by liquidity regime,
  • by event buckets (earnings weeks, crisis periods).

Robustness evidence

  • fold metrics distribution,
  • worst-window performance,
  • rolling error,
  • performance by regime.

Validator sentence

Average performance is a weak claim. Worst-window is a real claim.

3.3 Stress tests / scenario analysis (simple is fine)

Stress testing is not “predict the crisis perfectly”. Stress testing is:

“How does the model behave under adverse but plausible conditions?”

Simple defensible stress tests:

  • Volatility shock: multiply returns noise by 2–3.
  • Liquidity shock proxy: add a spread/slippage penalty.
  • Missing data shock: drop or mask a block of observations.
  • Correlation breakdown: scramble relationships between features and target.

The deliverable is:

  • a table: baseline vs stressed metrics,
  • a statement: “model is disabled or exposure reduced under these conditions”.

Validator sentence

Stress tests exist to define safe fallback behaviour.

4. Robustness evidence pack (what you deliver)

A robustness section should include:

  1. Parameter sensitivity table (grid).
  2. Subperiod stability plot (rolling metric).
  3. Stress test table (baseline vs stress).
  4. A short conclusion:
    • safe usage envelope,
    • triggers and actions.

5. Control lens

Robust control asks: - “What if the plant parameters change?” - “What if noise increases?”

Robust modelling is the same: - parameter drift, - volatility regime shifts, - missing signals.

Finance just adds incentives and execution.

6. Python lab (sensitivity grid + subperiod stability + stress tests)

This lab is fast and produces:

  • sensitivity results for horizon hh and vol window WW,
  • subperiod RMSE and rolling RMSE,
  • three stress tests.

We use a simple baseline model so the focus is on robustness protocol.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

rng = np.random.default_rng(0)

# ---------------------------------------------------------
# 1) Generate a time series with a regime change
# ---------------------------------------------------------
T = 700
t = np.arange(T)
idx = pd.date_range("2024-01-01", periods=T, freq="B")

season = 0.3 * np.sin(2*np.pi*t/5)
sigma = np.ones(T) * 0.7
sigma[T//2:] = 1.5  # volatility shock regime

y = np.zeros(T)
for k in range(1, T):
    y[k] = 0.25*y[k-1] + season[k] + sigma[k]*rng.normal()

df0 = pd.DataFrame({"y": y}, index=idx)

# ---------------------------------------------------------
# 2) Feature builder and evaluation (simple OLS baseline)
# ---------------------------------------------------------
def build(df, h=1, W=20):
    d = df.copy()
    d["r1"] = d["y"].diff()
    d["y_target"] = d["r1"].shift(-h)         # predict future change
    d["lag1"] = d["r1"].shift(1)
    d["lag2"] = d["r1"].shift(2)
    d[f"vol_{W}"] = d["r1"].rolling(W).std()
    d = d.dropna()
    return d

def rmse(y_true, y_hat):
    return float(np.sqrt(np.mean((y_true - y_hat)**2)))

def fit_predict_ols(train, test, feat_cols):
    Xtr = np.column_stack([np.ones(len(train))] + [train[c].values for c in feat_cols])
    ytr = train["y_target"].values
    beta = np.linalg.lstsq(Xtr, ytr, rcond=None)[0]

    Xte = np.column_stack([np.ones(len(test))] + [test[c].values for c in feat_cols])
    yhat = Xte @ beta
    return yhat

def eval_split(d, split_idx):
    train = d.iloc[:split_idx]
    test = d.iloc[split_idx:]
    feat_cols = [c for c in d.columns if c.startswith("lag") or c.startswith("vol_")]
    yhat = fit_predict_ols(train, test, feat_cols)
    return rmse(test["y_target"].values, yhat), test.index, test["y_target"].values, yhat

split_idx = len(df0) // 2  # train on first half, test on second half

# ---------------------------------------------------------
# 3) (A) Sensitivity grid: horizon h and vol window W
# ---------------------------------------------------------
Hs = [1, 3, 5]
Ws = [10, 20, 40]

rows = []
for h in Hs:
    for W in Ws:
        d = build(df0, h=h, W=W)
        # adjust split for dropped rows
        split = len(d) // 2
        score, _, _, _ = eval_split(d, split)
        rows.append({"horizon_h": h, "vol_window_W": W, "test_rmse": score})

sens_df = pd.DataFrame(rows).sort_values(["horizon_h", "vol_window_W"])
print("=== Sensitivity grid (test RMSE) ===")
print(sens_df.to_string(index=False))

# ---------------------------------------------------------
# 4) (B) Stability across subperiods: RMSE by quarter + rolling RMSE
# ---------------------------------------------------------
d = build(df0, h=1, W=20)
split = len(d) // 2
score, test_idx, ytrue, yhat = eval_split(d, split)

test = d.iloc[split:].copy()
test["err2"] = (test["y_target"].values - yhat)**2
test["rmse_roll_40"] = test["err2"].rolling(40).mean().apply(np.sqrt)

rmse_by_q = test.groupby(pd.Grouper(freq="Q"))["err2"].mean().apply(np.sqrt)

plt.figure(figsize=(8, 3.8))
plt.plot(test.index, test["rmse_roll_40"], label="rolling RMSE (40)")
plt.title("Stability check: rolling RMSE on test period")
plt.xlabel("date")
plt.ylabel("RMSE")
plt.legend()
plt.tight_layout()
plt.show()

plt.figure(figsize=(7.5, 3.5))
plt.plot(rmse_by_q.index.astype(str), rmse_by_q.values, marker="o")
plt.title("Subperiod stability: RMSE by quarter (test)")
plt.xlabel("quarter")
plt.ylabel("RMSE")
plt.xticks(rotation=45, ha="right")
plt.tight_layout()
plt.show()

print("\nRMSE by quarter (test):")
print(rmse_by_q)

# ---------------------------------------------------------
# 5) (C) Stress tests (simple and defensible)
#     Baseline vs stressed metrics
# ---------------------------------------------------------
def stress_volatility(df, factor=2.0):
    d = df.copy()
    eps = rng.normal(0, 1, len(d))
    d["y_stress"] = d["y"].values + (factor - 1.0) * eps  # add extra noise
    return d.rename(columns={"y_stress": "y"})[["y"]]

def stress_missing_block(df, start=450, end=520):
    d = df.copy()
    d.iloc[start:end] = np.nan
    d = d.ffill()  # policy must be explicit
    return d

def stress_feature_break(df):
    # Break relationship by permuting y (destroys dependence structure)
    d = df.copy()
    d["y"] = rng.permutation(d["y"].values)
    return d

def run_one(df_in, label):
    d = build(df_in, h=1, W=20)
    split = len(d) // 2
    score, _, _, _ = eval_split(d, split)
    return {"scenario": label, "test_rmse": score}

base = run_one(df0[["y"]], "baseline")
vols = run_one(stress_volatility(df0[["y"]], factor=2.0), "volatility_shock_x2")
miss = run_one(stress_missing_block(df0[["y"]], start=450, end=520), "missing_block_ffill")
perm = run_one(stress_feature_break(df0[["y"]]), "dependence_break_permute")

stress_df = pd.DataFrame([base, vols, miss, perm])
print("\n=== Stress test table (test RMSE) ===")
print(stress_df.to_string(index=False))

7. How to explain robustness results (stakeholder language)

Parameter sensitivity

  • “We tested horizon and window choices in a reasonable range.”
  • “Performance changes smoothly; no single magic value drives the result.”
  • “If a cliff exists, we treat it as fragility and tighten usage scope.”

Subperiod stability

  • “We report performance by quarter and rolling windows.”
  • “We focus on the worst period, not the average.”
  • “If failures cluster in high-vol periods, we restrict usage or adjust controls.”

Stress tests

  • “Under volatility shock, error increases as expected.”
  • “Under missing data, we define explicit fallback behaviour.”
  • “Under dependence break, performance collapses, which confirms the model relies on that structure.”

Validator sentence

Robustness is evidence that your result is not an accident.

8. Summary

  • Robustness is sensitivity + stability + stress.
  • Sensitivity checks prevent “magic parameter” results.
  • Subperiod stability prevents lucky window narratives.
  • Stress tests define safe usage and fallback behaviour.
  • Simple protocols beat complex stories.