Module 201 — Time series data transformations for quantitative models

1. Promise

One-sentence promise

After this lecture, you can transform time series data into returns, differences, trends, and scaled features in a time-consistent way, and you can explain why each transformation exists.

End tag line

Transformations are part of the model.

2. Learning objectives

After reading this lecture, you will be able to:

Explain why prices are often non-stationary and why returns are used instead.
Compute simple returns and log returns and interpret them.
Apply differencing and de-trending and state what problem they solve.
Choose a scaling method and fit it on training data only.
Detect “peeking” leakage created by normalisation on the full sample.
Communicate transformations in econometrics language: stationarity, unit roots, drift, volatility.

3. Prerequisites and setup

Required maths

Logarithms: $\log(ab)=\log a+\log b$ .
Mean and variance.

Python

Python 3.9+.
numpy, pandas, matplotlib.

4. Conceptual core

4.1 Prices vs returns

4.1.1 Why prices are tricky

Many asset prices behave approximately like a random walk with drift: $P_t = P_{t-1}\exp(\mu + \sigma \varepsilon_t), \quad \varepsilon_t \sim \mathcal{N}(0,1).$

Then the level $P_t$ is typically non-stationary: - mean changes with time, - variance grows with time.

Models trained on non-stationary levels often learn “time” rather than “signal”.

4.1.2 Simple returns

Simple return over one step: $R_t = \frac{P_t - P_{t-1}}{P_{t-1}}.$

Interpretation: - $R_t = 0.01$ means +1%.

Useful properties: - Intuitive for P&L approximations. - Not additive over time.

4.1.3 Log returns

Log return: $r_t = \log\left(\frac{P_t}{P_{t-1}}\right) = \log P_t - \log P_{t-1}.$

Why log returns are popular: - Additive across time: $\sum_{k=1}^n r_{t-k+1} = \log\left(\frac{P_t}{P_{t-n}}\right).$ - For small returns: $r_t \approx R_t$ . - Often closer to stationary than price levels.

Practical rule - Use returns for prediction tasks. - Use prices when the decision is explicitly about levels (e.g., barrier options), but then you must model non-stationarity explicitly.

Validator sentence If you fit on prices, you must justify why the level is the right object.

4.2 Differencing

Differencing is the operation: $\Delta x_t = x_t - x_{t-1}.$

It removes a unit root component in many cases.

4.2.1 Differencing prices and logs

Difference of prices: $\Delta P_t = P_t - P_{t-1}$ .
Difference of log prices: $\Delta(\log P_t) = \log P_t - \log P_{t-1} = r_t,$ which is exactly the log return.

So “log returns” are “first differences of log prices”.

4.2.2 When differencing is useful

To remove drift/trend from a series such as:
- load demand,
- interest rate levels,
- spreads that trend.

But differencing can remove long-run information. It can also amplify noise.

Validator sentence Differencing is a bias–variance trade-off. It reduces non-stationarity but can increase noise.

4.4 Scaling (normalisation) — fit scaler on train only

Scaling transforms a feature $x$ to a normalised version $z$ .

A common scaling is standardisation: $z = \frac{x - \mu}{\sigma}.$

Critical rule Compute $\mu$ and $\sigma$ on the training set only: $\mu_{\text{train}} = \frac{1}{n_{\text{train}}}\sum_{t\in \text{train}} x_t,\quad \sigma_{\text{train}}^2 = \frac{1}{n_{\text{train}}}\sum_{t\in \text{train}}(x_t-\mu_{\text{train}})^2.$

Then apply those parameters unchanged to validation/test.

If you compute $\mu$ and $\sigma$ on the full sample, you leak future distribution information into training.

This is “peeking via normalisation”.

Validator sentence A scaler is a model component. It must be trained like a model.

5. Control lens

5.1 Analogue and failure mode

In control, you normalise signals based on what the controller can know online.

If you scale using future statistics, you are designing a controller with privileged information. It will fail on the real plant.

Finance is the same.

6. Worked examples (micro narratives)

6.1 Equity price series

Prices trend upward.
Returns fluctuate around a small mean.
A model trained on prices may learn time and drift.
A model trained on returns targets the incremental move.

6.2 Load demand series

Load has daily and weekly seasonality and long-term trend.
Use de-trending and seasonal adjustment.
Use differencing if you target short-run changes.

6.3 Scaling risk features

Volatility regime shifts change feature distribution.
Fit scaler on train only.
Monitor drift in production.

7. Practical checklist (what to say in interviews)

What object do you model: price, return, spread, volatility?
Is it stationary enough for your method?
Do you need differencing or de-trending?
Are your transforms causal (only use past information)?
Is scaling fit on training only?
Did you document every transformation and its motivation?

8. Common mistakes (and how to correct them)

Training on price levels without justification
- Correction: use returns or include explicit trend terms and regime tests.
Differencing everything mechanically
- Correction: difference only where it removes a real non-stationarity and does not destroy signal.
Scaling on full data
- Correction: fit scaler on train only and treat scaler parameters as artefacts.
Using centred moving averages that look into the future
- Correction: use trailing windows only.
Mixing time zones and trading calendars
- Correction: align timestamps and market sessions before transformations.

9. Mini-exercises

Exercise 1

Show that the sum of log returns equals the log price ratio.

Solution $\sum_{k=1}^{n}(\log P_{t-k+1}-\log P_{t-k})=\log P_t-\log P_{t-n}.$

Exercise 2 (trap)

You standardise a feature using mean and std from 2018–2024, then test on 2024. Is it leakage?

Solution Yes. The scaler used 2024 information. Fit on training only.

Exercise 3

When is de-trending more appropriate than differencing?

Solution When the trend is deterministic (e.g., linear growth) and you want to preserve low-frequency structure.

10. Summary

Prices are often non-stationary; returns are often closer to stationary.
Log returns are differences of log prices and are additive.
Differencing removes stochastic trends but can increase noise.
De-trending removes deterministic trends.
Scaling must be fit on training only to prevent leakage.

11. Out of scope (for this sub-lecture)

Unit root tests (ADF, KPSS) and full ARIMA theory.
Cointegration and error-correction models.
Advanced filters and state-space models.

Python lab (prices → returns, differencing, de-trending, scaling without leakage)


import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

np.random.seed(42)

# -----------------------------
# 1) Synthetic price series (non-stationary) + a trend component
# -----------------------------
T = 400
t = np.arange(T)

mu = 0.0008          # drift
sigma = 0.02         # volatility
logP0 = np.log(100)

# Random walk in log-price
logP = logP0 + np.cumsum(mu + sigma * np.random.randn(T))
P = np.exp(logP)

df = pd.DataFrame({"P": P})
df["logP"] = np.log(df["P"])

# -----------------------------
# 2) Returns: simple and log
# -----------------------------
df["R_simple"] = df["P"].pct_change()
df["r_log"] = df["logP"].diff()  # log return = diff(log price)

# Identity check: cumulative log returns equal log price ratio
# sum_{k=1..n} r_k = log(P_n / P_0)
cum_log_return = df["r_log"].cumsum()
log_price_ratio = df["logP"] - df["logP"].iloc[0]

# -----------------------------
# 3) Differencing and de-trending on a trending feature
# -----------------------------
# Create a "macro-like" trending feature with noise
trend = 0.05 * t
x = trend + 3.0 * np.sin(2*np.pi*t/50) + np.random.randn(T)

df["x"] = x
df["dx"] = df["x"].diff()  # differencing

# Linear de-trending: x_t = a + b t + u_t
A = np.column_stack([np.ones(T), t])
beta = np.linalg.lstsq(A, df["x"].values, rcond=None)[0]
x_trend_hat = A @ beta
df["x_detrended"] = df["x"] - x_trend_hat

# -----------------------------
# 4) Scaling: show "peeking via normalisation"
# -----------------------------
# We will split by time, then scale correctly vs incorrectly
df2 = df.dropna().reset_index(drop=True)  # remove initial NaNs from returns/diffs
split = int(0.7 * len(df2))
train = df2.iloc[:split].copy()
test = df2.iloc[split:].copy()

feature = "x_detrended"

# BAD: fit scaler on full data (leakage)
mu_all = df2[feature].mean()
sd_all = df2[feature].std()
test["z_bad"] = (test[feature] - mu_all) / (sd_all + 1e-12)

# GOOD: fit scaler on train only (no leakage)
mu_tr = train[feature].mean()
sd_tr = train[feature].std()
test["z_good"] = (test[feature] - mu_tr) / (sd_tr + 1e-12)

# -----------------------------
# 5) Plots
# -----------------------------
# Plot 1: Price vs log returns (non-stationary vs more stable)
plt.figure(figsize=(8, 4))
plt.plot(df["P"].values, label="Price P_t")
plt.title("Price level is non-stationary (trend + growing variance)")
plt.xlabel("t")
plt.ylabel("Price")
plt.legend()
plt.tight_layout()
plt.show()

plt.figure(figsize=(8, 4))
plt.plot(df2["r_log"].values, label="Log return r_t")
plt.axhline(0, linestyle="--")
plt.title("Log returns are differences of log prices (often closer to stationary)")
plt.xlabel("t")
plt.ylabel("Log return")
plt.legend()
plt.tight_layout()
plt.show()

# Plot 2: Identity check (additivity of log returns)
plt.figure(figsize=(8, 4))
plt.plot(cum_log_return.values, label="Cumulative log returns Σ r_t")
plt.plot(log_price_ratio.values, linestyle="--", label="log(P_t / P_0)")
plt.title("Identity: Σ r_t = log(P_t / P_0)")
plt.xlabel("t")
plt.ylabel("Value")
plt.legend()
plt.tight_layout()
plt.show()

# Plot 3: Differencing vs de-trending on a trending feature
plt.figure(figsize=(8, 4))
plt.plot(df["x"].values, label="x (trending feature)")
plt.plot(x_trend_hat, linestyle="--", label="linear trend fit")
plt.title("A trending feature: x_t = trend + structure + noise")
plt.xlabel("t")
plt.ylabel("x")
plt.legend()
plt.tight_layout()
plt.show()

plt.figure(figsize=(8, 4))
plt.plot(df2["dx"].values, label="Δx (differenced)")
plt.plot(df2["x_detrended"].values, label="x detrended (residual)")
plt.axhline(0, linestyle="--")
plt.title("Differencing vs de-trending: two ways to remove trend structure")
plt.xlabel("t")
plt.ylabel("Value")
plt.legend()
plt.tight_layout()
plt.show()

# Plot 4: Scaling leakage demonstration (train-only vs full-sample)
plt.figure(figsize=(8, 4))
plt.hist(test["z_bad"].values, bins=40, alpha=0.6, label="BAD: scaled with full-sample stats")
plt.hist(test["z_good"].values, bins=40, alpha=0.6, label="GOOD: scaled with train-only stats")
plt.title("Peeking via normalisation: test distribution changes if you scale incorrectly")
plt.xlabel("Scaled value (test)")
plt.ylabel("Count")
plt.legend()
plt.tight_layout()
plt.show()

# -----------------------------
# 6) Print a small numeric summary
# -----------------------------
print("=== Scaling summary (test set) ===")
print("BAD scaling:  mean =", float(test["z_bad"].mean()), " std =", float(test["z_bad"].std()))
print("GOOD scaling: mean =", float(test["z_good"].mean()), " std =", float(test["z_good"].std()))

# Sanity check: log-return identity difference should be ~0 up to numerical tolerance
max_diff = np.nanmax(np.abs(cum_log_return.values - log_price_ratio.values))
print("\nMax |Σ r_t - log(P_t/P_0)| =", float(max_diff))