1. Promise

One-sentence promise

After this lecture, you can frame a quantitative modelling task by choosing a target, horizon, decision type, success metric, and constraints that are consistent with the business decision and the validation standard.

End tag line

Frame the problem before you touch the data.

2. Learning objectives

After reading this lecture, you will be able to:

  • Define a target variable YY that matches the business outcome.
  • Choose a prediction horizon hh that matches the decision cadence.
  • Distinguish forecast vs classification vs ranking problems and write each in notation.
  • Select a success metric (RMSE, MAE, AUC, Sharpe-like) and justify it mathematically.
  • Specify operational constraints (latency, interpretability, client acceptance) as testable requirements.
  • Detect and explain misalignment between target, horizon, metric, and constraints.

3. Prerequisites and setup

Required maths and prior knowledge

  • Random variables and expectation: 𝔼[X]\mathbb{E}[X].
  • Variance and covariance: Var(X)\mathrm{Var}(X), Cov(X,Y)\mathrm{Cov}(X,Y).
  • Basic regression and classification concepts.

Python environment assumptions

  • Python 3.9 or newer.
  • Jupyter notebook or similar.

Packages used

  • numpy
  • pandas
  • matplotlib

No external data is required. We use small synthetic datasets.

4. Conceptual core

4.1 The key idea

Problem framing is the act of defining:

  • Target Yt+hY_{t+h}: what you predict.
  • Horizon hh: for when you predict it.
  • Decision type: how the output will be used (forecast, classification, ranking).
  • Success metric: how you measure “good”.
  • Constraints: what must hold in production.

A model is well framed if these elements are mutually consistent.

A model is poorly framed if these elements conflict, even if some metric looks strong.

4.2 Defining the target

Let Xt∈ℝdX_t\in\mathbb{R}^d be features available at time tt.

Let Yt+hY_{t+h} be the outcome observed at time t+ht+h.

A model is a function fθf_\theta producing Ŷt+h=fθ(Xt)\hat{Y}_{t+h} = f_\theta(X_t).

Target types

  • Continuous: Yt+h∈ℝY_{t+h}\in\mathbb{R} (returns, spreads, load).
  • Binary: Yt+h∈{0,1}Y_{t+h}\in\{0,1\} (default, fraud).
  • Ordinal/categorical: Yt+h∈{1,…,K}Y_{t+h}\in\{1,\dots,K\} (risk bands).

Micro-example 1

  • Use case: intraday market making.
  • Target: mid-price change Δmt+h=mt+h−mt\Delta m_{t+h}=m_{t+h}-m_t.
  • Horizon: h=5h=5 seconds.

Micro-example 2

  • Use case: fraud screening.
  • Target: fraud flag Y∈{0,1}Y\in\{0,1\}.

Common confusion

You must distinguish “what you care about” from “what you observe”.

You may care about “true creditworthiness”, but you observe “default in 12 months”.

4.3 Choosing the horizon

The horizon hh is the gap between information time and evaluation time.

You decide:

  • Decision cadence: how often actions are taken.
  • Reaction speed: how quickly the environment changes.
  • Operational delay: when features are available and when actions take effect.

Micro-example 3

  • Use case: daily risk limits.
  • Horizon: h=1h=1 trading day.

Micro-example 4

  • Use case: retail mortgage credit risk.
  • Horizon: h=12h=12 months.
  • Reason: regulatory default definitions often use fixed observation windows.

Common confusion

Using the horizon that is convenient in the dataset rather than the horizon that matches the decision.

4.4 Decision type: forecast vs classification vs ranking

After choosing (Y,h)(Y,h), you decide how output becomes an action.

Forecast (regression)

  • Output: Ŷt+h∈ℝ\hat{Y}_{t+h}\in\mathbb{R}.
  • Use: pricing, sizing, forecasting.

Classification

  • Output: probability p̂t=ℙ(Yt+h=1∣Xt)\hat{p}_t=\mathbb{P}(Y_{t+h}=1\mid X_t).
  • Decision: p̂t>τ⇒\hat{p}_t>\tau \Rightarrow take action.

Ranking

  • Output: score st=fθ(Xt)s_t=f_\theta(X_t).
  • Decision: pick top KK items by score.
  • Only the ordering matters, not the absolute value.

Common confusion

Optimising a forecast error metric in a problem where only ranking matters.

4.5 Success metrics: RMSE, MAE, AUC, Sharpe-like

A success metric is a formal loss or utility function.

It encodes what errors are costly.

Regression losses

Residual ei=yi−ŷie_i = y_i-\hat{y}_i.

  • MAE: MAE=1n∑i=1n|ei|\mathrm{MAE}=\frac{1}{n}\sum_{i=1}^n |e_i|.
  • RMSE: RMSE=1n∑i=1nei2\mathrm{RMSE}=\sqrt{\frac{1}{n}\sum_{i=1}^n e_i^2}.

Key mathematical fact

  • Minimising squared loss yields the conditional mean: ŷ(x)=arg⁡minc𝔼[(Y−c)2∣X=x]=𝔼[Y∣X=x]. \hat{y}(x)=\arg\min_{c}\mathbb{E}[(Y-c)^2\mid X=x]=\mathbb{E}[Y\mid X=x].
  • Minimising absolute loss yields the conditional median: ŷ(x)=arg⁡minc𝔼[|Y−c|∣X=x]=median(Y∣X=x). \hat{y}(x)=\arg\min_{c}\mathbb{E}[|Y-c|\mid X=x]=\mathrm{median}(Y\mid X=x).

So metric choice changes what “optimal prediction” means under heavy tails.

Classification metrics

Given labels yi∈{0,1}y_i\in\{0,1\} and predicted probabilities p̂i∈[0,1]\hat{p}_i\in[0,1]:

  • Brier score: Brier=1n∑i=1n(yi−p̂i)2\mathrm{Brier}=\frac{1}{n}\sum_{i=1}^n (y_i-\hat{p}_i)^2.

AUC is a ranking metric. It measures how often a random positive scores above a random negative.

AUC does not guarantee calibrated probabilities.

Sharpe-like (decision-level)

For a strategy return series RtR_t:

Sharpe=𝔼[Rt]Var(Rt)+ε. \mathrm{Sharpe}=\frac{\mathbb{E}[R_t]}{\sqrt{\mathrm{Var}(R_t)}+\varepsilon}.

A Sharpe-like metric is meaningful only after you specify:

  • the strategy rule,
  • costs,
  • risk scaling,
  • and the evaluation period.

Common confusions

  • Using AUC when calibrated probabilities are required for pricing or provisioning.
  • Using RMSE when the decision only depends on sign or rank.

4.6 Defining constraints

Constraints are requirements that must be met even if predictive quality is high.

State them as testable conditions.

Latency

  • Requirement: end-to-end inference time ≤Lmax\le L_{\max}.
  • Include feature computation and post-processing.

Interpretability

  • Requirement: explainability artefact per decision, or stable global behaviour.
  • Examples: monotonicity, reason codes, model cards.

Client acceptance

  • Requirement: decisions must be defensible and stable.
  • Examples: avoid unexplained flips, provide consistent drivers.

Common confusion

Assuming “better metric” implies “better model”. A slightly weaker model that meets constraints is often the only deployable one.

5. Control lens

5.1 Analogue and failure mode

Control analogue

In control, you define reference, horizon, cost, and constraints before designing the controller.

Failure mode

A controller tuned on the wrong cost or horizon may look excellent in simulation and fail on the real plant.

The finance analogue is a model that “scores well” but produces bad decisions.

5.2 What the control engineer already knows

  • Horizon is a design choice.
  • Loss defines trade-offs.
  • Constraints dominate feasibility.
  • Sensitivity and robustness matter more than in-sample fit.

5.3 What finance changes

  • The “plant” is stochastic and non-stationary.
  • The cost function mixes profit, risk, and behaviour.
  • Constraints include regulation, governance, and client impact.
  • Framing must be revisited as regimes change.

6. Worked examples

6.1 Example 1: Daily return → decision metric dominates

Use case:

  • You choose exposure daily based on a signal.

Target:

  • Next-day excess return rt+1∈ℝr_{t+1}\in\mathbb{R}.

Horizon:

  • h=1h=1 trading day.

Decision:

  • Go long if r̂t+1>0\hat{r}_{t+1}>0, else flat.

Metric:

  • Primary: Sharpe-like of realised strategy returns.
  • Secondary: MAE/RMSE of predicted returns.

Constraints:

  • Latency: minutes to hours.
  • Interpretability: useful but not always mandatory.
  • Risk limits: drawdown bounds.

6.2 Example 2: Mortgage default → discrimination + calibration + constraints

Use case:

  • Accept/reject loans and price risk.

Target:

  • Y=1Y=1 if default within 12 months, else 00.

Horizon:

  • 12 months.

Decision:

  • Use p̂\hat{p} for thresholds and pricing.

Metric:

  • Primary: AUC for discrimination.
  • Secondary: Brier and calibration plots for probability quality.

Constraints:

  • Latency: milliseconds.
  • Interpretability: mandatory for governance and regulation.
  • Client acceptance: reason codes and consistency.

7. Python lab (plots included, no external data)

This lab shows two core lessons:

  1. MAE vs RMSE prefer different models under outliers.
  2. AUC (ranking) can look good even when calibration is imperfect.
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

rng = np.random.default_rng(42)

# -----------------------------
# Part A: Regression with outliers (MAE vs RMSE)
# -----------------------------
n = 2000
y = rng.normal(0.0, 1.0, n)

noise = rng.normal(0.0, 0.6, n)
out_idx = rng.choice(n, size=40, replace=False)
noise[out_idx] += rng.normal(0.0, 8.0, size=40)

pred_a = y + noise
pred_b = np.clip(pred_a, -2.5, 2.5)

def mae(y_true, y_hat):
    return float(np.mean(np.abs(y_true - y_hat)))

def rmse(y_true, y_hat):
    return float(np.sqrt(np.mean((y_true - y_hat) ** 2)))

e_a = y - pred_a
e_b = y - pred_b

print("=== Regression metrics ===")
print(f"Model A MAE={mae(y, pred_a):.4f}, RMSE={rmse(y, pred_a):.4f}")
print(f"Model B MAE={mae(y, pred_b):.4f}, RMSE={rmse(y, pred_b):.4f}")

plt.figure(figsize=(7, 3.6))
plt.hist(e_a, bins=60, alpha=0.6, label="Residuals A")
plt.hist(e_b, bins=60, alpha=0.6, label="Residuals B")
plt.title("Residual distribution: RMSE is dominated by tails")
plt.xlabel("Residual e = y - ŷ")
plt.ylabel("Count")
plt.legend()
plt.tight_layout()
plt.show()

se_a = e_a**2
se_b = e_b**2
top = 40
plt.figure(figsize=(7, 3.6))
plt.plot(np.sort(se_a)[::-1][:top], marker="o", linestyle="", label="A top e²")
plt.plot(np.sort(se_b)[::-1][:top], marker="o", linestyle="", label="B top e²")
plt.title("Largest squared errors dominate RMSE")
plt.xlabel("Rank among largest errors")
plt.ylabel("Squared error e²")
plt.legend()
plt.tight_layout()
plt.show()

# -----------------------------
# Part B: Classification (AUC + Brier + reliability plot)
# -----------------------------
n_obs = 3000
score = rng.normal(0, 1, n_obs)

# True PD via logistic link
true_pd = 1.0 / (1.0 + np.exp(-(score - 0.4)))
y_cls = rng.binomial(1, true_pd)

# Two models: one slightly biased, one noisier
score_c = score + 0.25
score_d = score + rng.normal(0, 0.7, n_obs)

def logistic(z):
    return 1.0 / (1.0 + np.exp(-z))

p_c = logistic(score_c)
p_d = logistic(score_d)

def brier(y_true, p_hat):
    return float(np.mean((y_true - p_hat) ** 2))

def auc(y_true, scores):
    # AUC = P(score_pos > score_neg) + 0.5*P(ties)
    y_true = np.asarray(y_true)
    scores = np.asarray(scores)
    pos = scores[y_true == 1]
    neg = scores[y_true == 0]
    # Efficient pairwise via sorting ranks
    # Compute ranks with average for ties
    order = np.argsort(scores)
    ranks = np.empty_like(order, dtype=float)
    ranks[order] = np.arange(len(scores), dtype=float)
    # Handle ties: average rank per tie group
    sorted_scores = scores[order]
    i = 0
    while i < len(scores):
        j = i
        while j + 1 < len(scores) and sorted_scores[j + 1] == sorted_scores[i]:
            j += 1
        if j > i:
            avg = 0.5 * (i + j)
            ranks[order[i:j+1]] = avg
        i = j + 1
    # Mann–Whitney U statistic
    n_pos = np.sum(y_true == 1)
    n_neg = np.sum(y_true == 0)
    sum_ranks_pos = np.sum(ranks[y_true == 1])
    U = sum_ranks_pos - n_pos * (n_pos - 1) / 2.0
    return float(U / (n_pos * n_neg))

print("\n=== Classification metrics ===")
print(f"Model C AUC={auc(y_cls, score_c):.3f}, Brier={brier(y_cls, p_c):.3f}")
print(f"Model D AUC={auc(y_cls, score_d):.3f}, Brier={brier(y_cls, p_d):.3f}")

# Reliability plot for Model C
bins = np.linspace(0, 1, 11)
bin_id = np.digitize(p_c, bins) - 1
bin_centres = 0.5 * (bins[:-1] + bins[1:])

obs = []
cnt = []
for k in range(len(bins) - 1):
    mask = bin_id == k
    cnt.append(int(mask.sum()))
    obs.append(float(y_cls[mask].mean()) if mask.sum() > 0 else np.nan)

plt.figure(figsize=(4.2, 4.2))
plt.plot([0, 1], [0, 1], "--", label="Perfect calibration")
plt.plot(bin_centres, obs, "o-", label="Model C")
plt.title("Reliability plot (calibration check)")
plt.xlabel("Predicted PD (bin centre)")
plt.ylabel("Observed default rate")
plt.legend()
plt.tight_layout()
plt.show()

# Quick sanity checks
assert len(y) == n
assert len(y_cls) == n_obs

8. Diagnostics and failure modes

  1. Target mismatch
    • Symptom: prediction does not map to the business decision.
    • Fix: restate YY as the quantity that the decision truly depends on.
  2. Horizon mismatch
    • Symptom: train on h=1h=1 but decision acts on h=10h=10.
    • Fix: rebuild labels at the correct horizon or formalise a mapping.
  3. Wrong decision type
    • Symptom: optimising RMSE when only top-KK ordering matters.
    • Fix: switch to ranking metrics and decision-level evaluation.
  4. Metric shopping
    • Symptom: primary metric changes after results are known.
    • Fix: pre-register primary and secondary metrics.
  5. Calibration ignored
    • Symptom: high AUC but poor pricing or provisions.
    • Fix: add Brier and reliability plots; calibrate if needed.
  6. Constraints ignored
    • Symptom: model cannot run in time or cannot be approved.
    • Fix: convert constraints into tests and restrict model class.
  7. Leakage
    • Symptom: performance collapses in production.
    • Fix: audit feature availability at time tt and label construction at t+ht+h.

9. Mini-exercises

Exercise 1

A desk rebalances every 5 minutes. They ask for “a return forecast”.

  • Define a target and horizon.

Solution Target: 5-minute excess return rt+5minr_{t+5\text{min}}. Horizon: h=5h=5 minutes.

Exercise 2

A credit model has AUC =0.90=0.90. Can you use p̂\hat{p} directly for pricing?

Solution No. AUC measures ranking, not calibration. Pricing needs calibrated probabilities. Check Brier and reliability plots.

Exercise 3

When are you more likely to prefer RMSE over MAE?

Solution When large errors are disproportionately costly. RMSE penalises tails through e2e^2.

10. Summary

  • Framing defines YY, hh, decision type, metric, and constraints before modelling.
  • RMSE and MAE correspond to different optimal predictors (mean vs median).
  • AUC measures discrimination, not calibration.
  • Sharpe-like metrics evaluate decisions, not pure predictions.
  • Constraints often determine deployability more than model score.

11. Out of scope

  • Full model training pipelines and hyperparameter tuning.
  • Advanced risk measures (CVaR, drawdown optimisation).
  • Execution microstructure and detailed cost models.

These belong in later lectures once framing is stable.