1. Promise

One-sentence promise

After this lecture, you can speed up data and feature pipelines using vectorisation, control memory growth on large datasets, and tell a simple profiling story about where time is spent.

End tag line

Fast enough beats perfect.

2. Learning objectives

After reading this lecture, you will be able to:

  • Replace slow Python loops with vectorised pandas/numpy operations.
  • Understand common memory traps in pandas (copies, wide frames, object dtype).
  • Use lightweight profiling to find the top time consumers.
  • Communicate performance work as an engineering story: measure → change → re-measure.

3. Conceptual core

3.1 Vectorisation in pandas

Principle

Push computation into: - numpy arrays, - pandas vectorised operations, - groupby/rolling operations.

Avoid: - row-wise loops, - DataFrame.apply(axis=1) as a default, - per-row Python logic.

Typical wins

  • Lag features: shift.
  • Returns: pct_change or diff.
  • Rolling stats: rolling().mean(), rolling().std().
  • Conditional flags: np.where, boolean masks.

Validator sentence

If your pipeline uses Python loops over rows, it will not scale.

3.2 Memory awareness for large data

Performance is often limited by memory, not CPU.

Key traps:

  • Wide frames (thousands of columns).
  • Object dtype (strings, mixed types).
  • Unnecessary copies (df.copy() everywhere).
  • Intermediate frames kept alive.
  • Merging large tables without keys quality checks.

Simple controls:

  • Use float32 / int32 when acceptable.
  • Convert categorical strings to category.
  • Drop intermediates early.
  • Use chunked reads for raw data.
  • Prefer long format when it avoids huge sparse wide tables.

Validator sentence

Memory is a constraint. Treat it as a first-class design input.

3.3 A simple profiling story (where time goes)

Good profiling is simple:

  1. Measure baseline runtime.
  2. Identify top hotspots.
  3. Optimise the biggest hotspot first.
  4. Re-measure and report improvement.
  5. Stop when you meet constraints.

Tools:

  • time.perf_counter() for timing blocks.
  • cProfile for function-level profiling.
  • tracemalloc for memory allocations.

Professional narrative

  • “Feature generation consumed 80% of runtime.”
  • “We replaced row-wise apply with vectorised operations.”
  • “Runtime dropped from 12 minutes to 45 seconds.”
  • “We verified identical outputs on a small sample.”

4. Common performance traps in quant work

  • Doing rolling computations inside loops.
  • Recomputing the same features per fold.
  • Merging without filtering early.
  • Using high precision everywhere by default.
  • Materialising giant intermediate arrays.

5. Control lens

In control, you respect real-time constraints. If your controller computes too slowly, it is unstable in practice.

In quant, latency is a stability condition too: - missed market windows, - stale risk numbers, - failed batch jobs.

6. Python lab (vectorisation, memory, profiling)

This lab shows three things:

  1. Loop vs vectorised computation for a feature.
  2. Memory impact of dtypes and wide frames.
  3. A minimal profiling story using perf_counter().
import time
import numpy as np
import pandas as pd

rng = np.random.default_rng(0)

# ---------------------------------------------------------
# 1) Create a moderately large dataset
# ---------------------------------------------------------
N = 500_000
df = pd.DataFrame({
    "price": 100 + np.cumsum(rng.normal(0, 1, N)).astype(np.float64),
    "volume": rng.integers(1, 5000, N).astype(np.int64),
})

# ---------------------------------------------------------
# 2) Example task: compute simple returns and a rolling mean
# ---------------------------------------------------------
def loop_returns(prices):
    # slow Python loop
    out = np.empty(len(prices), dtype=np.float64)
    out[0] = np.nan
    for i in range(1, len(prices)):
        out[i] = (prices[i] - prices[i-1]) / prices[i-1]
    return out

def vector_returns(prices):
    # fast vectorised
    p = pd.Series(prices)
    return p.pct_change().values

# Timing: loop vs vectorised
prices = df["price"].values

t0 = time.perf_counter()
r_loop = loop_returns(prices)
t1 = time.perf_counter()

t2 = time.perf_counter()
r_vec = vector_returns(prices)
t3 = time.perf_counter()

print("Loop returns time (s):", round(t1 - t0, 4))
print("Vector returns time (s):", round(t3 - t2, 4))

# Sanity check: values should match (ignoring first NaN)
diff = np.nanmax(np.abs(r_loop - r_vec))
print("Max absolute diff:", float(diff))

# Rolling mean: vectorised pandas
t4 = time.perf_counter()
df["ret"] = pd.Series(prices).pct_change()
df["roll_mean_50"] = df["ret"].rolling(50).mean()
t5 = time.perf_counter()
print("Vectorised rolling features time (s):", round(t5 - t4, 4))

# ---------------------------------------------------------
# 3) Memory awareness: dtype choices
# ---------------------------------------------------------
def mem_mb(frame):
    return float(frame.memory_usage(deep=True).sum() / (1024**2))

print("\nMemory (MB) before dtype optimisation:", round(mem_mb(df), 2))

# Downcast where acceptable
df_opt = df.copy()
df_opt["price"] = df_opt["price"].astype(np.float32)
df_opt["volume"] = df_opt["volume"].astype(np.int32)
df_opt["ret"] = df_opt["ret"].astype(np.float32)
df_opt["roll_mean_50"] = df_opt["roll_mean_50"].astype(np.float32)

print("Memory (MB) after dtype optimisation:", round(mem_mb(df_opt), 2))

# Wide frame trap demo (small scale to avoid huge RAM usage here)
wide = pd.DataFrame(rng.normal(size=(50_000, 50)))
print("\nWide frame memory (MB):", round(mem_mb(wide), 2))

# ---------------------------------------------------------
# 4) A minimal profiling story: measure -> change -> re-measure
# ---------------------------------------------------------
def slow_feature(df_in):
    # a deliberately slow pattern: apply row-wise
    return df_in[["price", "volume"]].apply(lambda row: row["price"] * np.log1p(row["volume"]), axis=1)

def fast_feature(df_in):
    # vectorised equivalent
    return df_in["price"].values * np.log1p(df_in["volume"].values)

sample = df.iloc[:200_000].copy()

t6 = time.perf_counter()
_ = slow_feature(sample)
t7 = time.perf_counter()

t8 = time.perf_counter()
_ = fast_feature(sample)
t9 = time.perf_counter()

print("\nRow-wise apply time (s):", round(t7 - t6, 4))
print("Vectorised time (s):", round(t9 - t8, 4))

7. What to say in interviews (a clean narrative)

  • “I start by measuring runtime and memory.”
  • “I remove row-wise apply and Python loops.”
  • “I downcast dtypes and avoid wide intermediate frames.”
  • “I profile the top hotspot first and validate outputs match.”
  • “I stop when latency and memory constraints are met.”

8. Summary

  • Vectorisation is the first performance lever.
  • Memory awareness prevents crashes and swaps.
  • Profiling is an engineering loop: measure, optimise, re-measure.
  • Performance is a production constraint, not an afterthought.