Data Versioning, Replay, and Audit Trail

2.1 Market snapshot

Definition

A market snapshot is:

• a versioned, immutable bundle of market data used for pricing and risk at a given cut-off.

Think:

• interest rate curves,
• FX curves,
• vol surfaces,
• equity prices,
• credit spreads,
• fixings and historical data where needed.

Each snapshot has:

• snapshot_id (primary key),
• as_of datetime (market close / risk cut-off),
• source or vendor date (which feed, which vendor, which file),
• metadata (region, book scope, etc.).

Inside, you might have child tables:

• snapshot_curves
• snapshot_vol_surfaces
• snapshot_fixings
• snapshot_fx … all referencing snapshot_id.

Properties:

• immutable: once created, you do not “edit” it; you create a new snapshot to represent a correction or new cut-off.
• complete enough to price the relevant universe of trades at that time.

REMEMBER: “A snapshot is an immutable view of the market: the exact curves, vols, and fixings used for a given valuation run.”

2.2 Run ID

Definition

A run ID is:

• a unique identifier for a pricing or risk run that fixes:
  • which trades / portfolio you valued,
  • which market snapshot(s) you used,
  • which model versions you used,
  • which library (code) version you ran,
  • which configuration / parameters you passed.

In a table, it might look like:

• run_id
• run_type (EOD risk, intraday what-if, P&L explain, calibration, etc.)
• portfolio_id / book_id / trade_set
• snapshot_id
• model_set_id or model IDs per asset class
• library_version or git_sha
• pricer_settings (JSON) – tolerances, seeds, approximations, etc.
• timestamps: created_at, started_at, completed_at
• initiator (user or scheduler).

Each trade-level result then references:

• run_id + trade_id
• plus PV, Greeks, diagnostics.

CORE SENTENCE: “Every reported number should point back to a run ID, and a run ID is a recipe: which snapshot, which models, which library version, which configuration.”

2.3 Replay

Definition

Replay is:

• the ability to re-run a valuation in the future and get the same results (within controlled tolerances).

Formally:

• Given:

  • run_id
  • library version v
  • snapshot_id
  • model_ids
  • pricer_settings
  • the same trade data

• You can run:

  price(trade, snapshot, model, pricer_settings, library v)

and reproduce:

• PV, Greeks, P&L explain numbers, etc.

For Monte Carlo:

• you may allow statistical equivalence (same seed → same result; or same confidence interval),
• but you still treat the run as deterministic by controlling seeds.

REMEMBER: “Replay means I can go back to 31 Jan 2025, pull snapshot S, run R with library version v, and get the same numbers the desk saw that day.”

4.1 Raw vendor data vs curated artefacts

Raw vendor data:

• tick data, quotes, yields, prices, surfaces as delivered by vendors.
• may be noisy, incomplete, or inconsistent.
• used as input to curve-building and calibration.

Curated artefacts:

• cleaned, interpolated, arbitrage-free curves,
• fitted vol surfaces,
• calibrated model parameters,
• aggregated indices or risk factors.

In practice you want both:

1. Raw data store (with source and timestamp).
2. Curated artefact store (curves, parameters) with versioned build configs.

REMEMBER: “Raw vendor data lets you revisit cleaning and curve-building decisions; curated curves are what pricers actually consume.”

4.2 Versioning transformations

Transformations include:

• cleaning rules (outlier filters, bid/ask rules),
• curve-building algorithms (bootstrap, smoothing, extrapolation),
• calibration procedures and optimisers.

To version them:

• store a configuration object (JSON) describing:
  • algorithm choices (e.g. bootstrapping method, interpolation scheme),
  • parameters (e.g. smoothing weights, tolerance, maximum iterations),
  • code version (library version or git SHA).
• link each artefact to:
  • raw data snapshot (raw vendor set),
  • transformation config (cleaning + building),
  • artefact ID (curve ID, vol surface ID, model ID).

So:

• curve_id → (raw_data_snapshot, curve_build_config_id, library_version)
• model_id → (curve_id, calibration_config_id, library_version)

CORE SENTENCE: “We don’t just store the curve; we store which raw data and which build configuration produced it, so we can re-run or challenge that process later.”

5.1 What you need to store

For proper replay and audit, you store:

• Trade data as of the run:
  • trade IDs and their state at that time (payoffs, notionals, schedules).
• Market snapshots:
  • snapshot IDs and all dependent artefacts (curves, vols, fixings).
• Model artefacts:
  • model IDs and parameters.
• Run configuration:
  • run ID, run type, snapshot, portfolios, settings.
• Outputs:
  • PVs, Greeks, risk measures, P&L explain, etc.
• Provenance:
  • library version, git SHA, environment, user.

This establishes:

REMEMBER: “Every number in a report points to a run, every run points to snapshots and models, and every snapshot/model points to raw data and build configs.”

5.2 Replay workflow

To replay a past run:

1. Retrieve run_id from risk system (e.g. from a P&L number or limit breach).
2. From the run record, retrieve:
   • snapshot_id,
   • model_set,
   • library_version,
   • pricer_settings,
   • trade universe / portfolio.
3. Check out or install the same library version.
4. Load:
   • the referenced trades as of that date,
   • snapshot artefacts,
   • models and parameters.
5. Re-run valuations; compare to stored outputs.

If they match:

• you confirm reproducibility.

If they differ:

• you investigate:
  • environment changes,
  • dependency or compiler changes,
  • possible bugs or data corruption.

6.1 Late vendor corrections

Sometimes:

• a vendor publishes a corrected curve or price for a past date (e.g. mis-stamped fixings, corrected holiday).

You do not overwrite the original snapshot. Instead:

1. Create a new snapshot for that as-of date, e.g.:
   • snapshot_id = S_2025-01-31_original
   • snapshot_id = S_2025-01-31_corrected_v2.
2. Store a link that says:
   • corrects_snapshot_id = S_2025-01-31_original.
3. New runs for that date can choose:
   • “use corrected snapshot” (default going forward), or
   • “replay original snapshot” (for audit of what the desk saw at the time).

REMEMBER: “We never rewrite history; we add corrected versions with explicit links.”

6.2 Backfills

Backfills occur when:

• you did not have data at the time,
• or you decide to rebuild historical curves with improved methods.

Example: you improve your curve-building algorithm and want to rebuild a 1-year history.

You then:

1. Keep original snapshots as the “as used” history.
2. Create new curated artefacts:
   • curve_id with version “backfilled_2025”,
   • linked to new build configs and same raw data if available.
3. Use these for:
   • risk backtesting,
   • stress testing,
   • model validation.
4. But keep as-used valuations for P&L explain and accounting.

Key message:

CORE SENTENCE: “Backfills create a parallel history for analysis, but they never overwrite the as-used data that produced actual decisions and P&L.”

8. 30-minute question: “How do you store market data and ensure replay?”

You want a strong, structured answer.

9. 30-minute question: “How do you track what produced a number?”

Talk about run provenance.

10. Deep technical: “Do you store raw vendor data or curated curves? How do you version transformations?”

We already sketched it; condense it:

• Store both raw and curated:
  • raw for traceability and reprocessing,
  • curated for fast, consistent consumption by pricers.
• Version transformations via:
  • config objects (algorithm, parameters, thresholds) plus library version.
• Artefact IDs hold:
  • raw_data_snapshot,
  • transform_config_id,
  • library_version.

Sample answer:

“We keep a raw vendor store and a curated store. Curves and vols in snapshots are built from raw data using versioned transformation configs that describe cleaning rules and bootstrap or interpolation methods. Each curve or surface is tagged with the raw data snapshot, build-config ID, and library version. That way we can re-run or challenge any transformation and know exactly how a given artefact was produced.”

11. Deep technical: “How do you handle late corrections and backfills?”

Pull from §6; compress.

Sample answer:

“We never overwrite as-used snapshots. If a vendor corrects data for a past date, we create a new snapshot for that date with a link back to the original. New analyses can use the corrected snapshot, but we still can replay what the desk actually saw. For backfills, for example when we improve a curve-building method, we rebuild historical artefacts in parallel, tagged as backfill versions. Those are used for risk studies and model validation, but production P&L and historical decisions remain anchored to the as-used data.”

That answer signals:

• you understand history vs analysis,
• and how governance views data.

12. Cheat sheet – lines to memorise

You should be able to say these on demand:

1. Snapshot

   “A market snapshot is an immutable bundle of curves, vols, and fixings with an ID; every valuation run references a snapshot ID.”

2. Run ID

   “A run ID is the recipe for a valuation: it fixes the snapshot, models, library version, portfolio, and configuration.”

3. Replay

   “Replay means I can regenerate past numbers exactly by re-running the same trades, snapshot, models, and code version.”

4. Trust sentence

   “Without snapshot versioning and run provenance, the framework cannot be trusted. You cannot debug, you cannot reproduce, and you cannot govern.”

5. Raw vs curated

   “We store raw vendor data for traceability and curated curves for consumption, and we version the transformations that turn one into the other.”

6. Corrections and backfills

   “Corrections and backfills create new versions; they never overwrite the as-used data that produced actual P&L and decisions.”

7. Control analogy

   “In control theory you cannot control what you cannot observe; in analytics, snapshots are the observed state of the system.”