1. Promise

One-sentence promise

After this lecture, you can recognise the most common time-series pitfalls—regime shifts, spurious correlation, microstructure noise, and timestamp issues—and you can implement basic checks that prevent false confidence.

End tag line

Most time series failures are operational, not statistical.

2. Learning objectives

After reading this lecture, you will be able to:

  • Explain non-stationarity and regime shifts and how they break models.
  • Recognise spurious correlation in trending series and avoid false relationships.
  • Describe microstructure noise (bid–ask bounce) and its impact at high frequency.
  • Handle missing timestamps and market closures without leaking information.
  • Implement a minimal “pitfall checklist” before modelling.

3. Prerequisites and setup

  • Basic knowledge of returns, differencing, rolling validation.
  • Python 3.9+ with numpy, pandas, matplotlib.
  • No external data needed.

4. Conceptual core

4.1 Non-stationarity and regime shifts

What it is

A process is non-stationary when its statistical properties change over time: - mean shifts, - volatility shifts, - correlations change.

A “regime shift” is a discrete or persistent change in the data generating process.

Finance reality

  • Calm vs crisis volatility.
  • Liquidity regimes (spreads widen).
  • Policy changes, index rebalances, fee changes, execution changes.

What it does to models

  • Parameters learned in regime A are wrong in regime B.
  • Scaling and thresholds drift.
  • Evaluation averages hide worst-window failure.

Controls

  • Rolling window validation.
  • Regime slicing (low/mid/high vol).
  • Drift monitoring (feature distributions, residuals).
  • Restricted usage statement: “valid only in these regimes”.

Validator sentence

A model is only as good as its worst regime that matters.

4.2 Spurious correlation

What it is

Two unrelated trending series can show high correlation. This is a classic time-series trap.

Why it happens

If both series are integrated / trending (random-walk-like), correlation in levels is not meaningful.

Finance examples

  • Price level of asset A correlates with price level of asset B.
  • Macro series with strong trend appears to “explain” anything.

Controls

  • Work in returns or differences.
  • Use time-aware validation.
  • Test stability across subperiods.
  • Demand an economic story.

Validator sentence

Correlation is not evidence when both series trend.

4.3 Microstructure noise (high frequency)

What it is

At high frequency, observed prices contain noise from: - bid–ask bounce, - discrete ticks, - latency, - venue fragmentation.

Impact

  • Very short-horizon returns can be dominated by noise.
  • Apparent negative autocorrelation is often bid–ask bounce.
  • Overfitting risk becomes extreme.

Controls

  • Use mid-price rather than last trade when possible.
  • Sample at lower frequency (e.g., 1-min instead of tick).
  • Use robust features (spreads, imbalance) with correct timing.
  • Validate with realistic execution assumptions.

Validator sentence

At high frequency, the data generating process includes the market microstructure.

4.4 Missing timestamps and market closures

What it is

Time series in finance have: - weekends, - holidays, - half-days, - missing ticks/bars, - vendor outages.

Pitfalls

  • Naive resampling creates artificial structure.
  • Forward-filling can leak information if applied incorrectly.
  • Centred windows use future information.
  • “Time since last observation” matters.

Controls

  • Use correct trading calendars.
  • Track “time gap” as a feature.
  • Use trailing windows only.
  • Define explicit missing-data policy per feature.
  • Separate “no trading” from “missing data”.

Validator sentence

A timestamp is part of the data, not metadata.

5. Control lens

In control, you distinguish: - model mismatch, - measurement noise, - actuator saturation, - sensor dropout.

Time series finance has the same: - regime shift (plant change), - microstructure (measurement noise), - execution limits (constraints), - missing timestamps (sensor dropout).

6. Python lab (regime shift, spurious correlation, microstructure noise, missing timestamps)

This lab is designed to run in 1–2 minutes.

It produces four demonstrations: 1) regime shift breaks a fixed model, 2) spurious correlation in trending levels, 3) microstructure noise creates artificial autocorrelation, 4) missing timestamps distort rolling statistics if mishandled.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

rng = np.random.default_rng(0)

# ---------------------------------------------------------
# Part 1: Non-stationarity / regime shift
# ---------------------------------------------------------
T = 600
t = np.arange(T)

# Regime shift: mean and volatility change at midpoint
mu = np.zeros(T)
sig = np.ones(T) * 0.8
mu[T//2:] = 1.0
sig[T//2:] = 1.6

y = np.zeros(T)
for k in range(1, T):
    y[k] = 0.4 * y[k-1] + mu[k] + sig[k] * rng.normal()

# Fit a simple AR(1) on first half, test on second half
split = T//2
y_tr = y[:split]
y_te = y[split:]

# OLS for AR(1): y_t = a + b y_{t-1}
X = np.column_stack([np.ones(len(y_tr)-1), y_tr[:-1]])
beta = np.linalg.lstsq(X, y_tr[1:], rcond=None)[0]

# Predict on test sequentially using true lag (one-step, not recursive drift)
yhat_te = beta[0] + beta[1] * y[split-1:-1]  # uses actual previous y

rmse = float(np.sqrt(np.mean((y_te - yhat_te)**2)))

plt.figure(figsize=(8, 3.8))
plt.plot(y, label="series y_t")
plt.axvline(split, linestyle="--", label="regime change")
plt.title("Regime shift: mean/volatility changes over time")
plt.xlabel("t")
plt.ylabel("y")
plt.legend()
plt.tight_layout()
plt.show()

plt.figure(figsize=(8, 3.8))
plt.plot(y_te[:120], label="y (test)")
plt.plot(yhat_te[:120], label="AR(1) trained in regime A")
plt.title(f"Fixed model breaks after regime change (test RMSE={rmse:.2f})")
plt.xlabel("test index")
plt.ylabel("value")
plt.legend()
plt.tight_layout()
plt.show()

# ---------------------------------------------------------
# Part 2: Spurious correlation in trending levels
# ---------------------------------------------------------
N = 500
a = np.cumsum(rng.normal(0, 1, N))  # random walk 1
b = np.cumsum(rng.normal(0, 1, N))  # random walk 2 (independent)

corr_levels = float(np.corrcoef(a, b)[0, 1])

# Differences (returns-like)
da = np.diff(a)
db = np.diff(b)
corr_diffs = float(np.corrcoef(da, db)[0, 1])

plt.figure(figsize=(8, 3.8))
plt.plot(a, label="A level (random walk)")
plt.plot(b, label="B level (random walk)")
plt.title(f"Two independent trends can look correlated (corr in levels = {corr_levels:.2f})")
plt.xlabel("t")
plt.ylabel("level")
plt.legend()
plt.tight_layout()
plt.show()

plt.figure(figsize=(8, 3.8))
plt.scatter(da, db, alpha=0.5)
plt.title(f"After differencing, correlation collapses (corr in diffs = {corr_diffs:.2f})")
plt.xlabel("ΔA")
plt.ylabel("ΔB")
plt.tight_layout()
plt.show()

# ---------------------------------------------------------
# Part 3: Microstructure noise (bid–ask bounce)
# ---------------------------------------------------------
M = 800
true_price = 100 + np.cumsum(rng.normal(0, 0.05, M))  # smooth latent price

spread = 0.04
# Trade price alternates between bid and ask with noise
side = np.where((np.arange(M) % 2) == 0, +1, -1)  # +1 ask, -1 bid
trade_price = true_price + side * (spread / 2) + rng.normal(0, 0.005, M)
mid_price = true_price + rng.normal(0, 0.002, M)

# Very short-horizon returns
r_trade = np.diff(trade_price)
r_mid = np.diff(mid_price)

def acf(x, max_lag=10):
    x = np.asarray(x, float)
    x = x - np.mean(x)
    denom = np.sum(x**2) + 1e-12
    out = []
    for k in range(max_lag+1):
        out.append(np.sum(x[k:] * x[:len(x)-k]) / denom)
    return np.array(out)

acf_trade = acf(r_trade, 10)
acf_mid = acf(r_mid, 10)

plt.figure(figsize=(8, 3.8))
plt.plot(trade_price[:200], label="trade price (with bid–ask bounce)")
plt.plot(mid_price[:200], label="mid price (cleaner)")
plt.title("Microstructure noise: trade price oscillates around latent price")
plt.xlabel("tick")
plt.ylabel("price")
plt.legend()
plt.tight_layout()
plt.show()

plt.figure(figsize=(8, 3.8))
plt.stem(np.arange(len(acf_trade)), acf_trade, basefmt=" ", label="trade returns ACF")
plt.stem(np.arange(len(acf_mid)), acf_mid, basefmt=" ", linefmt="C1-", markerfmt="C1o", label="mid returns ACF")
plt.axhline(0, linestyle="--")
plt.title("Bid–ask bounce can create artificial (often negative) short-lag autocorrelation")
plt.xlabel("lag")
plt.ylabel("ACF")
plt.legend()
plt.tight_layout()
plt.show()

# ---------------------------------------------------------
# Part 4: Missing timestamps / market closures
# ---------------------------------------------------------
dates = pd.date_range("2024-01-01", periods=200, freq="B")  # business days
x = np.cumsum(rng.normal(0, 1, len(dates)))

df_ts = pd.DataFrame({"x": x}, index=dates)

# Simulate vendor outage: remove a block of dates
df_missing = df_ts.drop(df_ts.index[60:80])

# Naive reindex with forward fill (common but risky)
df_reindexed = df_missing.reindex(dates)
df_ffill = df_reindexed.ffill()

# Rolling mean comparison
roll_true = df_ts["x"].rolling(10).mean()
roll_ffill = df_ffill["x"].rolling(10).mean()

plt.figure(figsize=(8, 3.8))
plt.plot(df_ts.index, df_ts["x"], label="original (complete)")
plt.plot(df_missing.index, df_missing["x"], ".", label="after outage (missing block)")
plt.title("Missing timestamps: outage creates a gap")
plt.xlabel("date")
plt.ylabel("x")
plt.legend()
plt.tight_layout()
plt.show()

plt.figure(figsize=(8, 3.8))
plt.plot(roll_true.index, roll_true.values, label="rolling mean (true)")
plt.plot(roll_ffill.index, roll_ffill.values, label="rolling mean (after reindex+ffill)")
plt.axvspan(df_ts.index[60], df_ts.index[79], alpha=0.2, label="missing block")
plt.title("Naive filling distorts rolling statistics around gaps")
plt.xlabel("date")
plt.ylabel("rolling mean (10)")
plt.legend()
plt.tight_layout()
plt.show()

print("=== Quick takeaways ===")
print("Regime shift test RMSE (fixed AR(1)):", rmse)
print("Spurious corr in levels:", corr_levels)
print("Corr after differencing:", corr_diffs)
print("Trade return ACF lag1:", float(acf_trade[1]), " | Mid return ACF lag1:", float(acf_mid[1]))

7. What the student should conclude from the lab

  1. Regime shift
  • A model trained in one regime can be systematically wrong in the next.
  • Rolling evaluation shows this early.
  1. Spurious correlation
  • Correlation in levels between trends is not evidence.
  • Differencing often removes the illusion.
  1. Microstructure noise
  • Trade prices carry bid–ask bounce.
  • Using trade returns at tick level can create artificial autocorrelation.
  • Mid prices are often a safer object.
  1. Missing timestamps
  • Gaps are not harmless.
  • Reindexing + forward fill can distort rolling features and evaluation.

8. Practical checklist (before you model)

  • Are you modelling levels or returns, and why?
  • Did you test stability by subperiod and volatility regime?
  • Did you check for spurious relationships (levels vs differences)?
  • If high frequency: are you using mid-price and realistic sampling?
  • Did you validate timestamp integrity and calendar alignment?
  • Are fills causal and documented?

9. Summary

  • Non-stationarity breaks fixed models.
  • Spurious correlation is common in trending time series.
  • Microstructure noise dominates at high frequency.
  • Timestamp gaps and closures change feature meaning.

10. Out of scope

  • Formal cointegration testing and error-correction models.
  • Full execution modelling and market impact.
  • Detailed market calendar libraries and corporate action handling.