Module 204 — Validation for forecasting in time series — Quant Model Playbook

1. Promise

One-sentence promise

After this lecture, you can validate forecasts using rolling and expanding windows, choose between direct and recursive multi-step forecasting, and pick a horizon that matches the business decision.

End tag line

Horizon and validation define what “good” means.

2. Learning objectives

After reading this lecture, you will be able to:

Explain rolling-window vs expanding-window validation and when to use each.
Implement both schemes and report fold-by-fold metrics.
Distinguish multi-step forecasting approaches:
- direct (one model per horizon),
- recursive (iterate a 1-step model forward).
Understand horizon alignment:
- forecast horizon $h$ must match the timing of the decision and evaluation.
Avoid the main leakage pathways in forecasting validation.

3. Prerequisites and setup

You understand returns vs prices, and time splits.
Python 3.9+ with numpy, pandas, matplotlib.

No external data is required.

4. Conceptual core

4.1 Rolling window vs expanding window

Rolling window (fixed training length)

You train on a moving window of length $L$ .

Train: $[t, t+L-1]$
Test: $[t+L, t+L+B-1]$

Then move forward by $B$ .

Why use rolling - Adapts to regime shifts. - Less contaminated by ancient history. - Mimics “recent-only” learning.

Failure mode - Too small $L$ increases variance and instability.

Expanding window (growing training length)

You train on everything available so far.

Train: $[1, t]$
Test: $[t+1, t+B]$

Why use expanding - Uses all data, reduces estimator variance. - Mimics accumulation of evidence.

Failure mode - Can be slow to adapt if the data generating process changes.

Validator sentence Choose rolling when the world changes. Choose expanding when the world is stable and data is scarce.

4.2 Multi-step horizon forecasting (direct vs recursive)

You want forecasts for $h > 1$ .

Direct forecasting

Train a separate model for each horizon $h$ :

$\hat{y}_{t+h} = f_h(X_t).$

Pros: - Avoids error accumulation. - Each horizon can use its own optimal mapping.

Cons: - More models to train and maintain. - Needs more data to fit each $f_h$ well.

Recursive forecasting

Train a 1-step model:

$\hat{y}_{t+1} = f(X_t)$

Then iterate forward: $\hat{y}_{t+2} = f(\hat{X}_{t+1}),\ \dots,\ \hat{y}_{t+h}.$

Pros: - One model to maintain. - Works naturally for some state-space style approaches.

Cons: - Errors compound with each step. - Requires careful feature updating for the “future” steps.

Practical rule - If the decision horizon is short (1–5 steps) and the system is stable, recursive can be acceptable. - If the horizon is longer or you care about stability, direct is often safer.

4.3 How to pick forecast horizon aligned with business decision

Horizon must match three things:

Decision cadence

How often do you act?
Example: daily portfolio rebalance → $h=1$ day is natural.

Outcome evaluation

When is performance judged?
Example: credit default over 12 months → horizon 12 months.

System dynamics

How fast does the world change?
Faster change → shorter horizons, more frequent updates.

Typical finance mappings

Intraday quoting: seconds to minutes.
Execution scheduling: minutes to hours.
Daily allocation: 1 day.
Risk (VaR horizon): 1–10 days depending on framework.
Credit default: 12 months (regulatory often drives it).

Validator sentence If your horizon does not match the decision, your metric is irrelevant.

5. Control lens

Rolling vs expanding mirrors: - rolling identification vs batch identification.

Direct vs recursive mirrors: - open-loop multi-step prediction vs closed-loop rollout of a one-step model.

Horizon selection mirrors: - controller sampling time and prediction horizon selection.

6. Python lab (rolling vs expanding, direct vs recursive, horizon effect)

This lab is designed to be fast and clear.

We simulate a series with: - weak mean dynamics, - seasonality, - a regime change.

We compare:

Rolling vs expanding validation.
Direct vs recursive multi-step forecasts (for $h=5$ ).
How fold metrics change when you move from $h=1$ to $h=5$ .

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

rng = np.random.default_rng(0)

# ---------------------------------------------------------
# 1) Synthetic series with seasonality + regime change
# ---------------------------------------------------------
T = 700
t = np.arange(T)

season = 0.4 * np.sin(2*np.pi*t/5)  # weekly-ish
eps = rng.normal(0, 1, T)

# Regime change: increase noise variance in second half
sigma = np.ones(T) * 0.7
sigma[350:] = 1.5

y = np.zeros(T)
phi = 0.25  # some persistence in the level-like series
for k in range(1, T):
    y[k] = phi * y[k-1] + season[k] + sigma[k] * eps[k]

df = pd.DataFrame({"y": y})
for lag in [1,2,3,5]:
    df[f"lag{lag}"] = df["y"].shift(lag)
df = df.dropna().reset_index(drop=True)

# ---------------------------------------------------------
# 2) Helpers: OLS model and metrics
# ---------------------------------------------------------
def rmse(y_true, y_hat):
    y_true = np.asarray(y_true); y_hat = np.asarray(y_hat)
    return float(np.sqrt(np.mean((y_true - y_hat)**2)))

def fit_ols(X, y):
    beta = np.linalg.lstsq(X, y, rcond=None)[0]
    return beta

def predict_ols(X, beta):
    return X @ beta

def make_design(d, cols):
    X = np.column_stack([np.ones(len(d))] + [d[c].values for c in cols])
    return X

# Features for 1-step model
feat_cols = ["lag1", "lag2", "lag3", "lag5"]

# ---------------------------------------------------------
# 3) Validation schemes: rolling vs expanding
# ---------------------------------------------------------
def rolling_folds(n, L_train, B_test):
    starts = list(range(0, n - (L_train + B_test) + 1, B_test))
    return [(s, s+L_train, s+L_train+B_test) for s in starts]

def expanding_folds(n, first_train_end, B_test, step):
    # train = [0:train_end), test = [train_end:train_end+B)
    folds = []
    train_end = first_train_end
    while train_end + B_test <= n:
        folds.append((0, train_end, train_end+B_test))
        train_end += step
    return folds

# ---------------------------------------------------------
# 4) Multi-step forecasting: direct vs recursive (h=5)
# ---------------------------------------------------------
def add_target_horizon(df_in, h):
    d = df_in.copy()
    d[f"y_h{h}"] = d["y"].shift(-h)
    d = d.dropna().reset_index(drop=True)
    return d

def direct_forecast(train, test, feat_cols, h):
    # Fit model to predict y_{t+h} directly from features at time t
    X_tr = make_design(train, feat_cols)
    y_tr = train[f"y_h{h}"].values
    beta = fit_ols(X_tr, y_tr)

    X_te = make_design(test, feat_cols)
    pred = predict_ols(X_te, beta)
    return pred

def recursive_forecast(train, test, feat_cols, h):
    # Fit 1-step model, then roll forward h steps for each test point
    # For simplicity, we use only lag features and simulate forward using predictions.
    X_tr = make_design(train, feat_cols)
    y_tr = train["y"].values  # 1-step target is y_t itself with lagged features already aligned
    # But we need proper 1-step setup: predict y_t from lags at t (already true in df)
    beta = fit_ols(X_tr, y_tr)

    # Recursive rollout for each test row
    preds_h = []
    for i in range(len(test)):
        # Start from known lags for this test row
        l1 = float(test.iloc[i]["lag1"])
        l2 = float(test.iloc[i]["lag2"])
        l3 = float(test.iloc[i]["lag3"])
        l5 = float(test.iloc[i]["lag5"])

        # We'll maintain a small lag buffer for recursion
        # For h up to 5, we only need to shift lags forward each step
        # Here we approximate lag5 update by shifting from the buffer as we roll
        buffer = [l1, l2, l3, float(test.iloc[i]["y"])]  # include current y as anchor (not perfect, but instructive)

        y_next = None
        for step in range(h):
            # Construct feature vector using most recent lags
            # We approximate lag5 using older buffer element if available; else fall back to l3
            lag1 = buffer[0]
            lag2 = buffer[1]
            lag3 = buffer[2]
            lag5 = buffer[-1] if len(buffer) >= 5 else lag3

            X = np.array([1.0, lag1, lag2, lag3, lag5])
            y_next = float(X @ beta)

            # Update buffer: new y becomes lag1 next step
            buffer = [y_next] + buffer[:4]

        preds_h.append(y_next)

    return np.array(preds_h)

# ---------------------------------------------------------
# 5) Run experiments
# ---------------------------------------------------------
# Build dataset for h=1 and h=5 direct forecasting
df_h1 = add_target_horizon(df, 1)
df_h5 = add_target_horizon(df, 5)

n1 = len(df_h1)
n5 = len(df_h5)

# Use common fold design parameters
L_train = 240
B_test  = 60

roll_folds_1 = rolling_folds(n1, L_train, B_test)
roll_folds_5 = rolling_folds(n5, L_train, B_test)

exp_folds_1 = expanding_folds(n1, first_train_end=L_train, B_test=B_test, step=B_test)
exp_folds_5 = expanding_folds(n5, first_train_end=L_train, B_test=B_test, step=B_test)

def eval_scheme(df_h, folds, h, scheme_name, do_recursive=False):
    fold_scores = []
    fold_pos = []
    for (a, b, c) in folds:
        tr = df_h.iloc[a:b].copy()
        te = df_h.iloc[b:c].copy()

        if do_recursive:
            # recursion uses a 1-step model on df_h1 structure, so only meaningful for h>1 with aligned inputs
            pred = recursive_forecast(tr, te, feat_cols, h=h)
            y_true = te[f"y_h{h}"].values  # compare to y_{t+h}
        else:
            pred = direct_forecast(tr, te, feat_cols, h=h)
            y_true = te[f"y_h{h}"].values

        fold_scores.append(rmse(y_true, pred))
        fold_pos.append((b + c) / 2)

    return np.array(fold_pos), np.array(fold_scores)

# Rolling vs expanding, h=1 direct
pos_r1, score_r1 = eval_scheme(df_h1, roll_folds_1, h=1, scheme_name="rolling", do_recursive=False)
pos_e1, score_e1 = eval_scheme(df_h1, exp_folds_1,  h=1, scheme_name="expanding", do_recursive=False)

# Rolling vs expanding, h=5 direct
pos_r5, score_r5 = eval_scheme(df_h5, roll_folds_5, h=5, scheme_name="rolling", do_recursive=False)
pos_e5, score_e5 = eval_scheme(df_h5, exp_folds_5,  h=5, scheme_name="expanding", do_recursive=False)

# Direct vs recursive for h=5 (use rolling folds on df_h5)
pos_rd5, score_rd5 = pos_r5, score_r5
pos_rr5, score_rr5 = eval_scheme(df_h5, roll_folds_5, h=5, scheme_name="rolling", do_recursive=True)

# ---------------------------------------------------------
# 6) Plots
# ---------------------------------------------------------
plt.figure(figsize=(8, 4))
plt.plot(pos_r1, score_r1, marker="o", label="Rolling CV (h=1, direct)")
plt.plot(pos_e1, score_e1, marker="o", label="Expanding CV (h=1, direct)")
plt.axvline(x=len(df)/2, linestyle="--", label="Regime change (approx)")
plt.title("Rolling vs expanding validation (h=1)")
plt.xlabel("Time index (approx)")
plt.ylabel("Fold RMSE")
plt.legend()
plt.tight_layout()
plt.show()

plt.figure(figsize=(8, 4))
plt.plot(pos_r5, score_r5, marker="o", label="Rolling CV (h=5, direct)")
plt.plot(pos_e5, score_e5, marker="o", label="Expanding CV (h=5, direct)")
plt.axvline(x=len(df)/2, linestyle="--", label="Regime change (approx)")
plt.title("Rolling vs expanding validation (h=5)")
plt.xlabel("Time index (approx)")
plt.ylabel("Fold RMSE")
plt.legend()
plt.tight_layout()
plt.show()

plt.figure(figsize=(8, 4))
plt.plot(pos_rd5, score_rd5, marker="o", label="Direct (h=5)")
plt.plot(pos_rr5, score_rr5, marker="o", label="Recursive (h=5)")
plt.axvline(x=len(df)/2, linestyle="--", label="Regime change (approx)")
plt.title("Direct vs recursive multi-step forecasting (rolling CV, h=5)")
plt.xlabel("Time index (approx)")
plt.ylabel("Fold RMSE")
plt.legend()
plt.tight_layout()
plt.show()

print("=== Summary (average fold RMSE) ===")
print("h=1  rolling direct :", float(np.mean(score_r1)))
print("h=1  expanding direct:", float(np.mean(score_e1)))
print("h=5  rolling direct :", float(np.mean(score_r5)))
print("h=5  expanding direct:", float(np.mean(score_e5)))
print("h=5  rolling direct :", float(np.mean(score_rd5)))
print("h=5  rolling recursive:", float(np.mean(score_rr5)))

7.1 Results interpretation (what the student should learn)

Rolling vs expanding is a choice about regime change

Rolling typically adapts faster to changes.
Expanding uses more history but can be slow to forget.

If the world changes, expanding can look “stable” but be wrong.

Horizon makes the task harder

RMSE usually increases when you go from $h=1$ to $h=5$ . This is normal. The correct response is not “try a bigger model”. The correct response is: “is $h=5$ the right business horizon?”

Direct vs recursive is a bias–variance trade-off

Recursive uses one model but can accumulate error.
Direct fits the horizon directly but needs more data and care.

A good interviewer sentence:

“Recursive is attractive operationally, but I only use it when error accumulation is controlled and stability holds across folds.”

8. Horizon alignment (how to speak it)

Ask three questions:

When do we act?
When do we measure success?
How fast does the environment change?

Then choose $h$ .

Examples: - Daily allocation → $h = 1$ day. - Weekly capacity planning → $h = 5$ business days. - Credit default → $h = 12$ months.

Validator sentence

A forecast horizon is a contract with the business. It must match decision timing and evaluation timing.