1. Promise
One-sentence promise
After this lecture, you can choose between AR/MA/ARIMA, state-space/Kalman, and ML feature-based models based on the data structure, the decision horizon, and the constraints, and you can explain when deep learning is real value versus theatre.
End tag line
Model choice is a business decision under uncertainty.
2. Learning objectives
After reading this lecture, you will be able to:
- Describe AR/MA/ARIMA as linear dynamics models for stationary transformed series.
- Explain state-space and Kalman filtering as latent-signal tracking and smoothing.
- Choose feature-based ML models (regularised regression, trees, boosting) for engineered features and non-linearities.
- State clear criteria for when deep learning helps and when it is theatre.
- Communicate model choice in a validator tone: constraints, stability, evidence.
3. Prerequisites and setup
- You know returns vs prices, differencing, seasonality, volatility clustering.
- You know evaluation discipline: time splits, stability, baselines.
4. Conceptual core
4.1 AR / MA / ARIMA intuition (linear dynamics)
What problem they solve
AR/MA/ARIMA are designed for linear dependence in a single series after you have made it reasonably stationary.
They answer: > Can I predict the next value using a linear combination of past values and past shocks?
The mental model (no formulas required)
AR (autoregressive) The series has “inertia”. Yesterday influences today directly.
MA (moving average) Shocks echo for a short time. You cannot observe shocks directly, but their effects persist.
ARIMA The “I” stands for integration. You difference the series first to remove trend, then apply ARMA to the differenced series.
In practice: - ARIMA is often strong for demand, rates, macro indicators. - For daily equity returns, linear predictability is often weak. - For volatility proxies, AR-type dynamics are often strong.
When AR/MA/ARIMA is the right choice
Use it when:
- You have a clear single time series forecasting task.
- You believe dynamics are approximately linear in transformed space.
- You need a strong baseline with interpretability.
- Data volume is moderate and you want fast, stable estimation.
When it is not the right choice
Avoid if:
- The series has strong non-linearities you cannot remove by transforms.
- Exogenous drivers dominate (then you need ARIMAX or a different setup).
- The decision is cross-sectional ranking across many assets.
- Regimes change abruptly (you might need time-varying parameters or state-space).
Slide-ready sentence
ARIMA is a disciplined baseline for linear dynamics after stationarity work.
4.2 State-space / Kalman intuition (latent signals, smoothing)
The core idea
Many finance signals are not directly observable. You see noisy observations. You want to infer a latent state.
State-space modelling treats the world as:
- A hidden state that evolves over time.
- An observation that is a noisy view of .
Two equations define the story:
- State evolution: how the latent signal changes.
- Observation model: how measurements relate to the state.
The Kalman filter is the classic solution for linear-Gaussian cases.
What it does well
- Smoothing: reduce noise and recover a stable signal.
- Tracking: adapt to changes gradually.
- Handling missing data: natural in the filtering framework.
- Time-varying parameters: treat coefficients as states.
Finance examples (typical uses)
- Estimating a “fair price” from noisy quotes.
- Tracking a latent trend component in macro time series.
- Dynamic hedge ratio estimation.
- Separating signal and noise in spread trading.
When to choose state-space / Kalman
Choose it when:
- You believe there is a latent process you are trying to track.
- Noise levels change and you want explicit modelling of uncertainty.
- You need online updates, not batch refits.
- You value interpretability and structured uncertainty.
Avoid it when:
- The system is strongly non-linear and you cannot justify approximations.
- You have little idea how to specify state and observation meaningfully.
Slide-ready sentence
Kalman filtering is the language of latent signal extraction under uncertainty.
4.3 ML for time series (feature-based)
Econometrics models dynamics explicitly. Feature-based ML models relationships in engineered features.
The workflow
- Transform the series (returns, differences).
- Engineer features:
- lags, rolling means, rolling vol,
- calendar features,
- cross-asset features if relevant.
- Fit a supervised model.
- Validate with time-series CV and stability slices.
Regularised regression (Lasso / Ridge / Elastic Net)
Use when:
- You have many correlated features.
- You want interpretability and stability.
- You want a strong baseline beyond ARIMA.
Why it works: - Regularisation prevents overfitting. - Coefficients are easy to explain. - You can enforce sign or monotonic constraints in some settings.
Trees and boosting (RF, XGBoost/LightGBM-like intuition)
Use when:
- Relationships are non-linear or involve interactions.
- You suspect “if–then” structure (e.g., effect changes by regime).
- You can tolerate lower interpretability but still want strong performance.
Strengths: - Capture non-linearities and interactions without manual specification. - Often strong on tabular time-series features.
Risks: - Easy to overfit on time structure. - Sensitive to leakage in feature engineering. - Can learn regime artefacts that do not persist.
Slide-ready sentence
Feature-based ML wins when your signal is in engineered covariates and interactions, not in pure linear dynamics.
4.4 Deep learning: when it helps vs when it is theatre
Deep learning is powerful when it can exploit:
- large data,
- stable patterns,
- complex structure that shallow models cannot capture.
In finance, those conditions are often not met.
When deep learning helps (real value)
Deep learning tends to help when you have at least one of:
- Very large datasets
- millions of sequences,
- many assets,
- many markets,
- long history.
- Rich inputs
- high-frequency sequences,
- limit order book tensors,
- text/news,
- alternative data with complex structure.
- Long context
- the decision depends on patterns over long windows,
- hierarchical temporal structure exists.
- Representation learning
- you need embeddings (e.g., instruments, clients, event types).
- Strong non-linear temporal interactions
- patterns that are not captured by rolling features.
When deep learning is theatre
Deep learning is usually theatre when:
- You have only a few thousand daily observations.
- The signal-to-noise is extremely low.
- A linear model or boosting already matches performance.
- The decision requires interpretability and stable drivers.
- You cannot reproduce performance across regimes.
- The model cannot be monitored and governed.
A good rule:
If you cannot beat: - last value, - seasonal naive, - linear regularised, - boosted trees, under strict time-series CV and regime checks, then deep learning is not justified.
How to speak about it professionally
- “We can test deep learning as a challenger.”
- “It must justify complexity via out-of-sample stability and operational feasibility.”
- “Otherwise it increases model risk without improving decisions.”
Slide-ready sentence
Deep learning is justified by data scale, structure richness, and stable uplift. Otherwise it is theatre.
5. Decision guide (a simple selection map)
5.1 If the problem is a single series forecast
- Start: naive baselines.
- If linear dependence is clear: ARIMA.
- If latent signal and noise dominate: state-space/Kalman.
- If you have exogenous drivers and non-linearities: feature-based ML.
5.2 If the problem is decision-level (trading / ranking)
- Start: features + regularised regression.
- Then: boosting if interactions matter.
- Deep learning only if you have scale + rich inputs + stability.
6. Validator checklist (what you must be able to say)
- What is the baseline and why is it hard to beat?
- What structure in the data motivates this model class?
- What is the evaluation protocol (rolling CV, regimes)?
- What controls exist (constraints, monitoring, fallback)?
- What limitations are known?
7. Typical pitfalls (and how to avoid them)
- ARIMA on non-stationary levels
- Transform first (log returns, differencing).
- Kalman filter with meaningless states
- State variables must have interpretation and units.
- Boosting that learns leakage
- Make feature timing explicit and use strict time splits.
- Deep learning as a status symbol
- Require decision-level uplift and regime robustness.
8. Mini-exercises
Exercise 1
You forecast weekly demand with clear seasonality and mild trend. Which model class is a natural starting point?
Solution Seasonal baselines and ARIMA on differenced series. If you need latent smoothing, state-space.
Exercise 2
You want to estimate a latent “fair value” from noisy quotes and occasional missing data. Which model class is most appropriate?
Solution State-space / Kalman filtering.
Exercise 3 (trap)
You have 3,000 daily observations and 50 features. Deep learning is the best choice. Agree or disagree.
Solution Disagree. Start with regularised regression and boosting. Deep learning must be justified by stable uplift and richer structure.
9. Summary
- AR/MA/ARIMA are linear dynamics tools after stationarity transforms.
- State-space/Kalman is for latent signal tracking and smoothing under uncertainty.
- Feature-based ML models non-linearities and interactions in engineered covariates.
- Deep learning helps with scale, rich inputs, and stable uplift. Otherwise it is theatre.
10. Out of scope (for this sub-lecture)
- Full ARIMA estimation and information criteria.
- Kalman derivations and filtering equations.
- Specific deep architectures (LSTM, Transformer) and tuning workflows.
Python lab
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
rng = np.random.default_rng(0)
# ---------------------------------------------------------
# 1) Synthetic series with structure:
# - latent trend-like signal (slow)
# - observed noise (higher)
# - volatility regime shift in second half
# ---------------------------------------------------------
T = 600
t = np.arange(T)
# Latent state (slowly varying)
x_true = np.zeros(T)
for k in range(1, T):
x_true[k] = 0.98 * x_true[k-1] + rng.normal(0, 0.15)
# Observation noise with a regime change
obs_noise = rng.normal(0, 0.6, T)
obs_noise[T//2:] = rng.normal(0, 1.2, T - T//2)
y = x_true + obs_noise # observed series
df = pd.DataFrame({"y": y, "x_true": x_true})
for lag in [1, 2, 3, 5, 10]:
df[f"lag{lag}"] = df["y"].shift(lag)
# Rolling volatility proxy (for regime awareness)
df["vol20"] = df["y"].diff().rolling(20).std()
df = df.dropna().reset_index(drop=True)
# ---------------------------------------------------------
# Helpers
# ---------------------------------------------------------
def rmse(y_true, y_hat):
y_true = np.asarray(y_true); y_hat = np.asarray(y_hat)
return float(np.sqrt(np.mean((y_true - y_hat) ** 2)))
def mae(y_true, y_hat):
y_true = np.asarray(y_true); y_hat = np.asarray(y_hat)
return float(np.mean(np.abs(y_true - y_hat)))
def fit_ar_ols(train_df, lags):
# AR(p): y_t = b0 + sum b_i * y_{t-i}
X = np.column_stack([np.ones(len(train_df))] + [train_df[f"lag{L}"].values for L in lags])
y = train_df["y"].values
beta = np.linalg.lstsq(X, y, rcond=None)[0]
return beta
def predict_ar_ols(test_df, lags, beta):
X = np.column_stack([np.ones(len(test_df))] + [test_df[f"lag{L}"].values for L in lags])
return X @ beta
def kalman_1d_filter(y_obs, q=0.02, r=0.6**2, x0=0.0, p0=1.0):
"""
Simple 1D Kalman filter:
x_t = x_{t-1} + w_t, w_t ~ N(0, q)
y_t = x_t + v_t, v_t ~ N(0, r)
Returns filtered state estimates x_hat.
"""
x_hat = np.zeros_like(y_obs, dtype=float)
x = x0
p = p0
for i, y_i in enumerate(y_obs):
# predict
p = p + q
# update
k = p / (p + r) # Kalman gain
x = x + k * (y_i - x)
p = (1 - k) * p
x_hat[i] = x
return x_hat
# ---------------------------------------------------------
# 2) Rolling CV to compare:
# - Baseline: last value
# - AR(3) via OLS (econometrics linear dynamics intuition)
# - Kalman smoothing as a "latent signal" extractor (state-space intuition)
# - Ridge on features (ML feature-based)
# ---------------------------------------------------------
L_train = 220
B_test = 60
starts = list(range(0, len(df) - (L_train + B_test) + 1, B_test))
fold_mid = []
fold_rmse = {"last": [], "ar3": [], "kalman": [], "ridge": []}
# Attempt to use sklearn Ridge; if unavailable, fall back to OLS on same feature set
try:
from sklearn.linear_model import Ridge
SKLEARN_OK = True
except Exception:
SKLEARN_OK = False
feature_cols = ["lag1", "lag2", "lag3", "lag5", "lag10", "vol20"]
for s in starts:
tr = df.iloc[s:s+L_train].copy()
te = df.iloc[s+L_train:s+L_train+B_test].copy()
y_te = te["y"].values
# Baseline: last value
pred_last = te["lag1"].values
# AR(3) via OLS
beta_ar3 = fit_ar_ols(tr, lags=[1, 2, 3])
pred_ar3 = predict_ar_ols(te, lags=[1, 2, 3], beta=beta_ar3)
# Kalman: fit "by judgement" with fixed q,r; use as a smoother then predict next as current filtered
# For one-step-ahead forecast: use filtered state at time t as prediction for y_{t+1} (random-walk state)
y_tr = tr["y"].values
xhat_tr = kalman_1d_filter(y_tr, q=0.03, r=float(np.var(y_tr - np.mean(y_tr))))
# Forecast on test: continue filtering sequentially; prediction at each step is filtered state before update.
# Simpler: filter on concatenated series and take state estimate as prediction of observation level.
y_concat = np.concatenate([y_tr, y_te])
xhat_concat = kalman_1d_filter(y_concat, q=0.03, r=float(np.var(y_tr - np.mean(y_tr))))
pred_kalman = xhat_concat[len(y_tr):] # smoothed level estimate during test
# ML: Ridge on engineered features
X_tr = tr[feature_cols].values
X_te = te[feature_cols].values
y_tr = tr["y"].values
if SKLEARN_OK:
model = Ridge(alpha=1.0)
model.fit(X_tr, y_tr)
pred_ridge = model.predict(X_te)
else:
# OLS fallback
X_tr2 = np.column_stack([np.ones(len(X_tr)), X_tr])
beta = np.linalg.lstsq(X_tr2, y_tr, rcond=None)[0]
X_te2 = np.column_stack([np.ones(len(X_te)), X_te])
pred_ridge = X_te2 @ beta
fold_mid.append(s + L_train + B_test/2)
fold_rmse["last"].append(rmse(y_te, pred_last))
fold_rmse["ar3"].append(rmse(y_te, pred_ar3))
fold_rmse["kalman"].append(rmse(y_te, pred_kalman))
fold_rmse["ridge"].append(rmse(y_te, pred_ridge))
# ---------------------------------------------------------
# 3) Plots: fold-by-fold RMSE (stability) + one window overlay
# ---------------------------------------------------------
plt.figure(figsize=(8, 4))
plt.plot(fold_mid, fold_rmse["last"], marker="o", label="Baseline: last value")
plt.plot(fold_mid, fold_rmse["ar3"], marker="o", label="AR(3) OLS")
plt.plot(fold_mid, fold_rmse["kalman"], marker="o", label="Kalman (latent smoothing)")
plt.plot(fold_mid, fold_rmse["ridge"], marker="o", label="Ridge on features")
plt.axvline(x=len(df)/2, linestyle="--", label="Regime change (approx)")
plt.title("Rolling CV: fold RMSE shows stability and regime sensitivity")
plt.xlabel("Time index (approx)")
plt.ylabel("RMSE per fold")
plt.legend()
plt.tight_layout()
plt.show()
print("=== Average RMSE across folds ===")
for k in fold_rmse:
print(f"{k:7s}: {np.mean(fold_rmse[k]):.4f}")
# One illustrative fold overlay (last fold)
s = starts[-1]
tr = df.iloc[s:s+L_train].copy()
te = df.iloc[s+L_train:s+L_train+B_test].copy()
pred_last = te["lag1"].values
beta_ar3 = fit_ar_ols(tr, lags=[1,2,3])
pred_ar3 = predict_ar_ols(te, lags=[1,2,3], beta_ar3)
y_tr = tr["y"].values
y_te = te["y"].values
y_concat = np.concatenate([y_tr, y_te])
xhat_concat = kalman_1d_filter(y_concat, q=0.03, r=float(np.var(y_tr - np.mean(y_tr))))
pred_kalman = xhat_concat[len(y_tr):]
X_tr = tr[feature_cols].values
X_te = te[feature_cols].values
if SKLEARN_OK:
from sklearn.linear_model import Ridge
model = Ridge(alpha=1.0)
model.fit(X_tr, tr["y"].values)
pred_ridge = model.predict(X_te)
else:
X_tr2 = np.column_stack([np.ones(len(X_tr)), X_tr])
beta = np.linalg.lstsq(X_tr2, tr["y"].values, rcond=None)[0]
X_te2 = np.column_stack([np.ones(len(X_te)), X_te])
pred_ridge = X_te2 @ beta
plt.figure(figsize=(8, 4))
plt.plot(y_te[:80], label="y (test)")
plt.plot(pred_last[:80], label="last value")
plt.plot(pred_ar3[:80], label="AR(3)")
plt.plot(pred_kalman[:80], label="Kalman")
plt.plot(pred_ridge[:80], label="Ridge")
plt.title("One fold: prediction overlay (first 80 test points)")
plt.xlabel("Test index (within fold)")
plt.ylabel("Value")
plt.legend()
plt.tight_layout()
plt.show()