Architecture & Systems

Raw vs Curated Data and Transformations

Raw vendor data is the starting point; curated curves and models are versioned outputs of documented transformation pipelines.

intermediateraw datacuratedtransformationspipeline

Reliable analytics distinguish source observations from documented, versioned transformations.

Raw data stores vendor quotes and time series as delivered, preserving the original record for traceability.

Curated artefacts such as arbitrage-free curves, vol surfaces, and calibrated models are produced by cleaning and transformation steps.

INTERACTIVE INTUITION

Explore the mechanism

Which observations came from the source, and which values were cleaned, mapped, adjusted, or modeled?

Raw vs curated data transformations

Raw vendor quotes are stored as-is. Curated curves are versioned outputs of an explicit transform configuration.

Transformation configuration

Filter

40%

Smooth

55%

Bootstrap method

linearpiecewisespline

Interpretation

Raw data is the audit record. It should not be “cleaned in place”.
Curated artefacts are derived outputs: they must carry the transform config + version.
When you change filters or interpolation, you are creating a new curated artefact.

cfg

INFO

{"filter_strength":"40%","smoothing":"55%","bootstrap":"spline"}

Raw quotes (points) → Curated curve (line)

Higher filter removes outliers. Higher smoothness reduces kinks. Changing config should create a new curated artefact version.

Pipeline graph (toy)

Raw snapshot → Transform config → Curated artefact (versioned)

Raw vendor snapshot

immutable

Transform config

spline / f=40% / s=55%

Curated artefact

versioned output

raw timestampscurrent normalizercurated seriesconsistent model input

State is coherentThe curated dataset names the transformation that produced it.

UNDERSTAND

What to notice

Transformation logic is captured in configuration objects that describe filters, bootstrapping methods, interpolation, and calibration settings.

Each curated artefact links back to both its raw data snapshot and its transformation configuration, enabling replay and challenge of the pipeline.

TRANSFER

Where it appears

building auditable market-data pipelines
reprocessing data after rule changes
separating vendor facts from analytical outputs

CHALLENGE

Where the intuition breaks

What this view leaves out

Keeping raw data does not guarantee reproducibility unless transformation code, parameters, schemas, and dependencies are also versioned.

CONNECT

Where this concept sits

Follow the strongest prerequisites, applications, extensions, and contrasts.

Raw vs Curated Data and TransformationsYOU ARE HERE

Foundation ofMarket Snapshots as Bundles

Reliable snapshots should be assembled from curated, explicitly transformed market data rather than opaque raw feeds.

SupportsData Lineage and Audit Trail

Lineage records how raw inputs were transformed into curated data used downstream.

Supported bySchema Migrations and Backward Compatibility

Data pipelines depend on controlled schema evolution so transformations keep working across versions.

Implemented byScheduled Jobs and Worker Queues

Scheduled and queued workers are a common operational mechanism for running ingestion and transformation pipelines.

Foundation of · Market Snapshots as Bundles

Reliable snapshots should be assembled from curated, explicitly transformed market data rather than opaque raw feeds.

Explore concept

SOURCES

References

W3C Recommendation, PROV-O (opens in a new tab)

← Explore all concepts

Mental model

What to notice

What this view leaves out

Connections for Raw vs Curated Data and Transformations