Module 103 — Baselines and iteration for quantitative models — Quant Model Playbook

1. Promise

One-sentence promise

After this lecture, you can build naive baselines, define a defensible feature set v0, and apply a model selection logic where simplicity is the default and complexity must be justified by decision-level evidence.

End tag line

If you cannot beat a baseline, you do not have a model.

2. Learning objectives

After reading this lecture, you will be able to:

Construct naive baselines (last value, seasonal naive, simple linear model) for forecasting and classification.
Define a minimal feature set v0 that is time-correct and defensible.
Compare models using pre-defined metrics and a consistent split protocol.
Apply a model selection logic: simplest model that meets requirements wins.
Explain when added complexity is justified and how to validate it.

3. Prerequisites and setup

Required maths and prior knowledge

Mean, variance, covariance.
Linear regression intuition.
Familiarity with the idea of train/validation/test by time.

Python environment assumptions

Python 3.9+ in a notebook.

Packages used

numpy, pandas, matplotlib

No external data is required.

4. Conceptual core

4.1 Why baselines are non-negotiable

A baseline is a simple reference model $f_0$ that any real model must beat.

Baselines serve three roles:

Reality check: detect leakage and overfitting.
Value check: quantify marginal value of complexity.
Communication: explain performance in human terms.

Validator sentence

If you do not beat a naive baseline out-of-sample, the correct conclusion is: “no evidence of signal”.

4.2 Naive baselines (forecasting)

Let $y_t$ be a time series.

Baseline 1: Last value (random-walk / persistence)

$\hat{y}_{t+1} = y_t.$

This is surprisingly hard to beat for prices and spreads.

Baseline 2: Seasonal naive

With seasonality period $s$ (e.g., 5 for weekdays):

$\hat{y}_{t+1} = y_{t+1-s}.$

Use when the series has strong day-of-week or hour-of-day patterns.

Baseline 3: Linear model on simple lags

Define lag features $x_t = [1, y_t, y_{t-1}, \dots, y_{t-k}]$ .

Fit: $\hat{y}_{t+1} = \beta^\top x_t,$ with $\beta$ estimated on training data.

This is the first “learned” baseline and often a strong benchmark.

Practical note

For returns, “last value” is often near zero signal. For volatility, “last value” and “rolling mean” are strong baselines.

4.3 Naive baselines (classification)

For labels $y\in\{0,1\}$ with class rate $\pi=\mathbb{E}[y]$ :

Constant probability baseline: $\hat{p}=\pi_{\text{train}}$ .
Constant class baseline: predict majority class.

Calibration baseline

A model that cannot beat the constant probability baseline in Brier score is not useful.

4.4 Feature set v0 (simple, defensible)

Feature set v0 is the minimal set of features that:

are available at decision time $t$ ,
have clear economic meaning,
are robust across regimes,
can be explained to governance.

Generic v0 template (time series / market data)

For an asset $i$ :

Lagged returns: $r_{t}, r_{t-1}, \dots$
Rolling volatility: $\sigma_t = \sqrt{\frac{1}{L}\sum_{j=0}^{L-1} r_{t-j}^2}$
Rolling mean / momentum: $\bar{r}_t = \frac{1}{L}\sum_{j=0}^{L-1} r_{t-j}$
Volume / turnover lags (if relevant): $v_t, \Delta v_t$
Calendar features: day-of-week, month, hour bucket (one-hot or integer)

Credit / risk v0 template

Age of account, utilisation, payment history summaries
Debt-to-income proxies
Simple monotone transforms and buckets

Validator lens

v0 must be auditable: each feature has a definition, timestamp, and missing-data policy.
v0 should be robust to small data quality issues.

Slide-ready sentence

v0 is the smallest feature set you can defend in a model committee.

4.5 Model selection logic (simplicity first)

Model selection is not “pick the best validation metric”.

It is a decision under constraints.

Default rule

Choose the simplest model that meets:

performance threshold on validation,
stability across periods,
constraint requirements.

A formalised logic (practical)

Let models be ordered by complexity: $M_0 \prec M_1 \prec \dots$ .

Pick the smallest $M_k$ such that:

$ (M_k) (M_0) + _{} $ on validation and test.
Performance is stable across sub-periods.
Latency, interpretability, and governance constraints are satisfied.
Improvement persists after adding realistic costs and slippage (if trading).

Complexity must be justified

Complexity is justified only if it buys at least one of:

materially better decision-level outcomes,
materially lower tail risk / constraint violations,
materially better robustness under regime shifts.

Validator sentence

A black-box uplift without stability and explanation is not an uplift. It is a liability.

5. Control lens

5.1 Analogue and failure mode

Control analogue

In control, you start with a baseline controller (e.g., PID). You only move to MPC when PID fails to meet constraints or performance.

Failure mode

Designing MPC to track noise looks impressive in simulation. It fails on the plant due to unmodelled dynamics and constraints.

Finance analogue: complex ML looks good in backtests, fails out-of-sample.

6. Worked example (single asset forecasting)

Use case:

You forecast a daily series $y_t$ .

Protocol:

Walk-forward split.
Compare last-value baseline, seasonal naive, and linear lag model.
Plot errors and show which baseline wins.

Interpretation:

If linear cannot beat last-value, you likely do not have incremental signal.
If seasonal naive wins, the “signal” may be calendar structure, not alpha.

7. Python lab (baselines + plots, fast)

This lab builds three baselines on a synthetic series with weak trend + seasonality.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

rng = np.random.default_rng(0)

# Synthetic series: trend + weekly seasonality + noise
T = 260
t = np.arange(T)
season = 0.8 * np.sin(2 * np.pi * t / 5)      # weekday pattern (period=5)
trend = 0.01 * t
y = trend + season + rng.normal(0, 0.8, T)

df = pd.DataFrame({"y": y})
df["y_lag1"] = df["y"].shift(1)
df["y_lag2"] = df["y"].shift(2)
df["y_lag3"] = df["y"].shift(3)
df = df.dropna().reset_index(drop=True)

# Time split
split = int(0.7 * len(df))
train = df.iloc[:split].copy()
test = df.iloc[split:].copy()

def rmse(y_true, y_hat):
    y_true = np.asarray(y_true); y_hat = np.asarray(y_hat)
    return float(np.sqrt(np.mean((y_true - y_hat) ** 2)))

def mae(y_true, y_hat):
    y_true = np.asarray(y_true); y_hat = np.asarray(y_hat)
    return float(np.mean(np.abs(y_true - y_hat)))

# Baseline 1: last value
test["pred_last"] = test["y_lag1"]

# Baseline 2: seasonal naive (period s=5)
s = 5
df["y_lag_season"] = df["y"].shift(s)
df2 = df.dropna().reset_index(drop=True)
split2 = int(0.7 * len(df2))
test2 = df2.iloc[split2:].copy()
test2["pred_seasonal"] = test2["y_lag_season"]

# Baseline 3: linear model on lags (closed-form OLS)
# y_t = beta0 + beta1*y_{t-1} + beta2*y_{t-2} + beta3*y_{t-3} + eps
X_tr = np.column_stack([
    np.ones(len(train)),
    train["y_lag1"].values,
    train["y_lag2"].values,
    train["y_lag3"].values,
])
y_tr = train["y"].values

beta = np.linalg.lstsq(X_tr, y_tr, rcond=None)[0]

X_te = np.column_stack([
    np.ones(len(test)),
    test["y_lag1"].values,
    test["y_lag2"].values,
    test["y_lag3"].values,
])
test["pred_lin"] = X_te @ beta

# Metrics
print("=== Test metrics (same split) ===")
print("Last value   RMSE:", rmse(test["y"], test["pred_last"]), " MAE:", mae(test["y"], test["pred_last"]))
print("Linear lags  RMSE:", rmse(test["y"], test["pred_lin"]),  " MAE:", mae(test["y"], test["pred_lin"]))

print("\n=== Seasonal naive metrics (slightly different due to lag s) ===")
print("Seasonal     RMSE:", rmse(test2["y"], test2["pred_seasonal"]), " MAE:", mae(test2["y"], test2["pred_seasonal"]))

# Plot: last vs linear on test
plt.figure(figsize=(8, 4))
plt.plot(test["y"].values[:80], label="y (test)")
plt.plot(test["pred_last"].values[:80], label="last-value")
plt.plot(test["pred_lin"].values[:80], label="linear-lags")
plt.title("Baselines on test window")
plt.xlabel("Time index (test)")
plt.ylabel("Value")
plt.legend()
plt.tight_layout()
plt.show()

# Plot: error distributions
e_last = test["y"].values - test["pred_last"].values
e_lin  = test["y"].values - test["pred_lin"].values

plt.figure(figsize=(8, 4))
plt.hist(e_last, bins=40, alpha=0.6, label="last-value errors")
plt.hist(e_lin,  bins=40, alpha=0.6, label="linear-lags errors")
plt.title("Error distributions: baseline comparison")
plt.xlabel("Error")
plt.ylabel("Count")
plt.legend()
plt.tight_layout()
plt.show()

8. Diagnostics and iteration protocol (v0 → v1 → …)

8.1 A practical iteration loop

Build baselines $M_0$ : last value, seasonal naive, simple linear.
Define feature set v0 and document it.
Fit simplest learnable model (linear or logistic).
Evaluate on validation and test using pre-registered metrics.
Only then consider complexity (trees, boosting, deep learning).

8.2 What justifies complexity

Clear uplift $\ge \Delta_{\min}$ on test, not only validation.
Stability across sub-periods.
No new leakage pathways.
Meets latency and interpretability constraints.
Decision-level metric improves after costs.

9. Mini-exercises

Exercise 1

You forecast daily volatility. Which baseline is usually strongest?

Solution Last-value or rolling average volatility. Volatility is persistent.

Exercise 2

Your complex model beats linear on validation but not on test. What is the correct conclusion?

Solution No evidence of improvement. Complexity is not justified.

Exercise 3 (trap)

Your model beats last-value baseline in RMSE but produces worse trading Sharpe. Is it an improvement?

Solution No. If the decision objective is risk-adjusted performance, decision-level metric dominates.

10. Summary

Baselines are mandatory and often strong in finance.
Feature set v0 must be time-correct, interpretable, and auditable.
Model selection defaults to simplicity.
Complexity must be justified by stable, decision-level evidence under constraints.

11. Out of scope

Full hyperparameter tuning pipelines.
Advanced ensembling and stacking.
Detailed transaction cost modelling (covered in execution lectures).