1. Promise

One-sentence promise

After this lecture, you can make your quantitative runs reproducible by controlling randomness, snapshotting data deterministically, and producing run IDs with complete artefact folders.

End tag line

Reproducibility is the foundation of trust.

2. Learning objectives

After reading this lecture, you will be able to:

  • Fix random seeds and explain what they control and what they do not.
  • Create deterministic data snapshots and version datasets.
  • Generate run IDs and store complete artefact folders.
  • Build a minimal “run manifest” that allows a validator to reproduce outputs.
  • Avoid common reproducibility failures in quant workflows.

3. Conceptual core

3.1 Fixed seeds (where relevant)

Randomness appears in many places:

  • data simulation,
  • train/test shuffling (should be avoided for time series),
  • stochastic optimisers,
  • bootstrapping,
  • Monte Carlo,
  • model initialisation (deep learning),
  • multi-threaded libraries with non-deterministic reductions.

A seed is useful when:

  • you want two runs to differ only by a specific change,
  • you want to reproduce a backtest or evaluation exactly.

A seed is not a guarantee when:

  • computation is parallel and order of operations changes,
  • hardware differences matter (GPU kernels),
  • libraries are not deterministic by design.

Engineering rule

Record the seed in the config and write it into the run manifest.

Validator sentence

A fixed seed is necessary for reproducibility, but never sufficient.

3.2 Deterministic data snapshots / versioned datasets

In practice, models fail reproducibility because data changed.

You need a dataset policy:

  • Snapshot the exact data used for a run.
  • Store it immutably.
  • Assign a dataset ID.

Typical methods:

  • Save a parquet/CSV snapshot to a versioned folder.
  • Use a content hash of the file (SHA256).
  • Use a database snapshot or point-in-time query with an “as-of” timestamp.
  • Record vendor, extraction time, and filters.

For market data, “as-of” matters: - corporate actions get corrected, - historical bars get restated, - missing data gets backfilled.

Engineering rule

A run must reference a dataset ID that can be reloaded later.

Validator sentence

If the dataset cannot be reconstructed, the model cannot be reconstructed.

3.3 Run IDs + artefact folders

A reproducible run produces an evidence pack.

Minimum artefacts:

  • config.json snapshot,
  • manifest.json (run metadata),
  • metrics.json summary,
  • fold_metrics.csv (if CV),
  • plots (optional but useful),
  • model parameters (optional for baseline),
  • notes / assumptions.

Run IDs must be:

  • unique,
  • stable,
  • searchable.

Good practice:

  • run ID includes timestamp + short config hash,
  • folder is immutable after completion.

Engineering rule

A run is not complete until its artefacts are written.

4. Minimal run manifest (what must be recorded)

A manifest.json should include:

  • run_id
  • created_at (UTC)
  • git_commit (if available)
  • dataset_id + dataset_path
  • config_hash
  • seed
  • python version (optional)
  • library versions (optional)

This is the smallest reproducibility contract that works.

5. Common failure modes (and corrections)

  1. Seed is set but not recorded
  • Correction: store seed in config and manifest.
  1. Data is read “live” from a changing source
  • Correction: snapshot and version datasets.
  1. Artefacts are overwritten
  • Correction: run folders are write-once.
  1. Paths are hard-coded
  • Correction: paths come from config or environment variables.
  1. Environment drift
  • Correction: record package versions or use lock files.

6. Control lens

In control and experiments, you keep:

  • initial conditions,
  • controller version,
  • plant configuration,
  • measurement setup.

Quant modelling is the same. A model run is an experiment. You need a lab notebook. Run IDs and artefact folders are the lab notebook.

7. Python lab (seeds, dataset snapshot IDs, run ID + artefact folder)

This lab demonstrates:

  • deterministic synthetic data generation with a fixed seed,
  • writing a versioned dataset snapshot with a content hash as dataset ID,
  • producing a run ID with a short config hash,
  • writing a manifest + metrics into a run folder.
import json
import os
import hashlib
from datetime import datetime, timezone

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# -----------------------------
# Utilities
# -----------------------------
def sha256_bytes(b: bytes) -> str:
    return hashlib.sha256(b).hexdigest()

def sha256_file(path: str) -> str:
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(8192), b""):
            h.update(chunk)
    return h.hexdigest()

def save_json(path: str, obj: dict):
    with open(path, "w", encoding="utf-8") as f:
        json.dump(obj, f, indent=2)

def now_utc_iso() -> str:
    return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")

def short_hash(s: str, n: int = 8) -> str:
    return hashlib.sha256(s.encode("utf-8")).hexdigest()[:n]

# -----------------------------
# 1) Config (seed + dataset spec)
# -----------------------------
cfg = {
    "seed": 123,
    "data": {"T": 500, "freq": "B"},
    "target": {"horizon": 1},
    "features": {"lags": [1,2,3,5]},
}

cfg_str = json.dumps(cfg, sort_keys=True)
cfg_hash = short_hash(cfg_str, n=10)

# -----------------------------
# 2) Deterministic "dataset snapshot"
# -----------------------------
rng = np.random.default_rng(cfg["seed"])
T = cfg["data"]["T"]
idx = pd.date_range("2024-01-01", periods=T, freq=cfg["data"]["freq"])

# Synthetic series (deterministic given seed)
t = np.arange(T)
season = 0.3 * np.sin(2*np.pi*t/5)
y = np.zeros(T)
for k in range(1, T):
    y[k] = 0.2*y[k-1] + season[k] + rng.normal(0, 1.0)

df = pd.DataFrame({"y": y}, index=idx)

# Write dataset snapshot (parquet if available; CSV fallback)
os.makedirs("datasets", exist_ok=True)
dataset_path = os.path.join("datasets", f"dataset_{now_utc_iso().replace(':','')}_{cfg_hash}.csv")
df.to_csv(dataset_path)

dataset_id = sha256_file(dataset_path)

# -----------------------------
# 3) Run ID + artefact folder
# -----------------------------
run_id = f"{datetime.now().strftime('%Y-%m-%d_%H%M%S')}_{cfg_hash}_{dataset_id[:8]}"
run_dir = os.path.join("runs", run_id)
os.makedirs(os.path.join(run_dir, "plots"), exist_ok=True)

# -----------------------------
# 4) Minimal "run" output (metrics + a plot)
# -----------------------------
# Simple baseline: last value forecast (yhat_t = y_{t-1})
df2 = df.copy()
df2["yhat"] = df2["y"].shift(1)
df2 = df2.dropna()

rmse = float(np.sqrt(np.mean((df2["y"] - df2["yhat"])**2)))
mae = float(np.mean(np.abs(df2["y"] - df2["yhat"])))

metrics = {"rmse": rmse, "mae": mae}

plt.figure(figsize=(7.5, 3.5))
plt.plot(df2.index[:120], df2["y"].values[:120], label="y")
plt.plot(df2.index[:120], df2["yhat"].values[:120], label="yhat (last value)")
plt.title("Baseline forecast (first 120 points)")
plt.xlabel("date")
plt.ylabel("value")
plt.legend()
plt.tight_layout()
plt.savefig(os.path.join(run_dir, "plots", "baseline_preview.png"))
plt.show()

# -----------------------------
# 5) Write artefacts: config, manifest, metrics
# -----------------------------
save_json(os.path.join(run_dir, "config.json"), cfg)
save_json(os.path.join(run_dir, "metrics.json"), metrics)

manifest = {
    "run_id": run_id,
    "created_at_utc": now_utc_iso(),
    "config_hash": cfg_hash,
    "seed": cfg["seed"],
    "dataset_id": dataset_id,
    "dataset_path": dataset_path,
    "notes": "Reproducibility demo: fixed seed + dataset snapshot hash + run artefacts.",
}
save_json(os.path.join(run_dir, "manifest.json"), manifest)

print("Run dir:", run_dir)
print("Dataset path:", dataset_path)
print("Dataset ID (sha256):", dataset_id)
print("Metrics:", metrics)

8. What this lab proves

  • Same config + same seed produces the same dataset values.
  • Dataset snapshot is immutable and has a content hash.
  • Run ID ties together config hash and dataset ID.
  • A validator can rerun by loading:
    • runs/<run_id>/config.json
    • dataset at manifest.json["dataset_path"].

9. Mini-exercises

  1. Change only the seed and observe dataset ID changes.
  2. Keep seed fixed and change T. Observe config hash changes.
  3. Add library versions into manifest.json.
  4. Make the dataset snapshot path depend only on dataset ID (content-addressable storage).

10. Summary

  • Seeds control randomness but do not solve drifting data.
  • Deterministic data snapshots with dataset IDs solve “data changed” failures.
  • Run IDs and artefact folders are the lab notebook for quant systems.
  • Reproducibility is a validation prerequisite, not a nice-to-have.