Module 501 — Client delivery stories (consulting behaviour)

The control-room rules

• One idea per sentence.
• Use numbers, dates, and thresholds.
• Say what you did and what you refused to do.
• Name the main risks before the stakeholder does.
• End with a clear “what is safe to use” statement.

4.1 Objective + stakeholder

Objective

Reduce inventory blow-ups by flagging when the desk’s inventory is likely to become risk-expensive over the next $h=30$ minutes.

Stakeholder

Head trader and desk risk controller.

Success is measured in: - fewer limit breaches, - smoother inventory, - no loss of quote competitiveness.

4.2 Data reality (messy constraint)

• Quotes and trades arrive with timestamp jitter and occasional missing bursts.
• Order book snapshots are not continuous.
• Inventory is known, but decomposition by venue is delayed.
• Latency budget is strict: decision must be produced in < 50 ms.

Key constraint: - Anything requiring deep feature engineering across the full order book is too slow.

4.3 Modelling choice and why

I framed it as a classification with calibrated probability:

• Target: $$Y_{t+h}=\mathbf{1}\{\text{inventory risk cost over next 30 minutes exceeds threshold}\}$$
• Output: $\hat{p}_t = \mathbb{P}(Y_{t+h}=1\mid X_t)$

I chose a logistic model / monotone tree depending on explainability needs.

Why: - The desk needed a probability to gate actions. - AUC alone was not enough. - They needed stable behaviour and reason codes.

Feature set v0 was simple: - inventory level and change, - spread, short-term volatility proxy, - trade imbalance, - time-of-day buckets.

4.4 Validation and controls

Evaluation protocol: - Rolling time-series CV with embargo of $h$ to avoid overlap effects. - Stability checks across: - high-vol vs low-vol, - auction vs continuous trading, - news windows (proxy: volatility spikes).

Controls: - Hard output bounds: cap $\hat{p}_t$ to avoid extreme oscillation. - Missing data policy: if key feeds missing, fall back to baseline risk rule. - Monitoring: - calibration drift check weekly, - feature distribution drift daily, - alert if risk flags cluster abnormally.

I explicitly communicated two limitations: - It is not an alpha model. - It is not designed for new venues without recalibration.

4.5 What I delivered (code + doc + handover)

Deliverables (evidence pack): - src/ inference module with latency profiling. - config/ thresholds, horizons, filters. - metrics/ AUC + Brier + stability-by-regime tables. - plots/ reliability curves and failure-window plots. - docs/ model card: assumptions, limitations, operational runbook. - Handover session: 60 minutes with trader + risk controller.

I left a safe usage statement: - “Use as an alert and gating input, not a trading signal.”

4.6 What I would improve next

• Add a regime switch: separate calibration for high-vol regime.
• Improve label quality with a refined risk-cost proxy.
• Add online monitoring dashboard for drift and false-positive rate.
• Extend from 30-minute to multi-horizon outputs.

5.1 Objective + stakeholder

Objective

Improve loan approval decisions while staying within governance. Provide a probability of default that can feed pricing and provisioning.

Stakeholder

Credit risk committee, model validation, and business product owner.

Primary success: - improved risk ranking, - calibrated PDs, - audit-ready documentation.

5.2 Data reality (messy constraint)

• Multiple source systems with inconsistent definitions for “income” and “employment”.
• Missingness is not random: high-risk applicants have more missing fields.
• Label definition is regulatory: default within 12 months, but historical policy changed.

Constraints: - Interpretability is non-negotiable. - Any feature must have a clear lineage and justification.

5.3 Modelling choice and why

I framed it as classification with probability output.

• Target: $$Y=\mathbf{1}\{\text{default within 12 months}\}$$
• Output: $$\hat{p}=\mathbb{P}(Y=1\mid X)$$

I chose logistic regression with monotonic constraints / scorecard style as v1.

Why: - Governance acceptance. - Stable behaviour under policy shifts. - Clear variable effects and reason codes.

I used a strict feature set v0: - utilisation measures, - payment history aggregates, - debt-to-income buckets, - stable demographic proxies where allowed, - missingness indicators (explicitly treated).

5.4 Validation and controls

Metrics: - Discrimination: AUC. - Calibration: Brier, reliability plots, calibration slope/intercept. - Stability: by vintage (origination month), by segment, by macro proxy (unemployment bucket).

Controls: - Reject inference if key fields are outside training support. - Fairness and bias checks where required by policy. - Override framework: human decision can override, must log reason.

I positioned validation in committee language: - “We are not optimising AUC only. We need calibrated PD and stability under policy drift.”

5.5 What I delivered (code + doc + handover)

Deliverables: - End-to-end pipeline with versioned datasets. - Config with label definition and window rules. - Model card with: - assumptions, - limitations, - population and exclusions, - monitoring plan.

Handover: - Workshop with credit ops on reason codes. - Validator walkthrough with reproducibility pack.

Safe usage statement: - “Use PD for ranking and pricing within approved segments only.”

5.6 What I would improve next

• Revisit label drift using a consistent backfilled definition.
• Add challenger model (GBM) as shadow to test incremental uplift.
• Improve missing-data modelling with causal missingness patterns.
• Expand stress testing with macro scenarios.

6.1 Situation

Stakeholder says: - “We need a model that predicts next-day returns with Sharpe above 2. Use all available data. We need it in two weeks.”

6.2 Your pushback (the professional form)

I align first: - “Understood. You want a decision advantage that holds out-of-sample.”

Then I isolate the risky assumption: - “A Sharpe target is a decision-level outcome. It cannot be promised from a prediction task without a complete execution model and costs.”

Then I propose the reframing: - “We can deliver two things in two weeks: a baseline evidence pack and a stable signal that beats naive baselines after costs in a constrained setting.”

6.3 The alignment move

I offer options, not refusal:

Option A (fast, safe): - Deliver a baseline strategy with transparent assumptions. - Provide stability by regime and worst-window results. - Provide a monitoring plan and a kill-switch rule.

Option B (ambitious, higher risk): - Deliver a richer model but only as a shadow deployment. - No production usage until stability and drift checks pass.

I force clarity on constraints: - “Do you prioritise speed or governance acceptance?”

6.4 Resolution

We agree: - A two-week deliverable: baseline + evaluation discipline + artefact pack. - A second phase: complexity only if the baseline evidence shows signal.

6.5 What you learned (explicit)

• Pushing back is not saying “no”.
• It is saying “yes, under conditions that protect the client and the bank.”

7.1 A 90-second version (for interviews)

• Objective + who cared.
• One data constraint that mattered.
• One modelling choice and why.
• One validation control that prevented embarrassment.
• What you shipped.
• One improvement you would do next.

7.2 A 5-minute version (for committees)

Add: - split protocol and leakage controls, - stability across regimes, - limitations and safe-usage statement, - monitoring plan.