Topic 27 – Dynamic Programming Intuition

1. Core idea

Dynamic programming is for multi-period decisions with memory.

Four ingredients:

• State $s_t$: what summarises the past for the future.
• Decision $a_t$: what you control now.
• Transition: how state evolves: $$s_{t+1} = f(s_t, a_t, \omega_t)$$ where $\omega_t$ is randomness (prices, inflows…).
• Reward $r_t(s_t, a_t)$: cashflow at time $t$.

Value function:

$$V_t(s) = \max_{\{a_u\}_{u \ge t}} \mathbb{E}\Big[\sum_{u=t}^T r_u(s_u, a_u)\,\Big|\,s_t = s \Big]$$

Bellman equation:

$$V_t(s) = \max_{a \in \mathcal{A}(s)} \left\{ r_t(s,a) + \mathbb{E}\big[ V_{t+1}(s') \mid s,a \big] \right\}$$

“Optimal value today = best immediate reward + expected value of optimal future.”

2. Hydro reservoir as a DP problem

Hydro asset:

• State $s_t$: reservoir volume (water level) at time t.
• Decision $a_t$: generation/release (MWh or m³) at time t.
• Transition: $$s_{t+1} = \min\{ s_t - a_t + \text{inflow}_t, \; s_{\max} \}$$
• Reward: $$r_t(s_t, a_t) = P_t \cdot a_t$$ where $P_t$ is the power price.

Constraints:

• $0 \le a_t \le \text{turbine capacity}$,
• $s_{\min} \le s_t \le s_{\max}$ (environmental & physical),
• Possibly minimum generation or ramp constraints.

Interpretation:

• Water in the reservoir is an inventory with option value.
• Using water now → cash today but less flexibility later.
• DP chooses a policy $a_t(s_t)$: how much to generate for each water level and time.

3. Backward induction intuition

In a finite horizon with known prices/inflows:

1. At final time $T$:

   • No future value; all remaining water is either:
     • Used (subject to capacity) or
     • Left unused if capacity binding.
   • $V_T(s_T) = P_T \cdot \min(s_T, \text{capacity})$.

2. Step backwards:

   For each time $t$ and water level $s_t$:

   • Loop over feasible releases $a_t$.
   • Compute: $$\text{total}(a_t) = P_t a_t + V_{t+1}(s_{t+1})$$
   • Pick $a_t$ that maximises total.

Qualitative behaviour:

• If future prices are clearly higher than today → optimal to save water.
• If today’s price is very high and little time left → optimal to generate now.
• With inflows and capacity, you may need to pre-emptively generate to avoid spill.

4. Why this matters for a power quant

Many “options” in commodities are control problems:

• Hydro reservoirs,
• Gas storage,
• Swing contracts,
• Tolling (power plant dispatch vs spark spread).

Static option intuition (“just a call on the spread”) is not enough.

DP framing shows you get that:

• The value depends on the whole path of prices and inflows,
• Constraints (capacity, min flows, ramping) are central,
• Optimal policy is time- and state-dependent, not a single strike.

Interview line:

“For hydro and storage assets, I think in dynamic programming terms: reservoir level (or inventory) is the state, generation/injection is the control, and the Bellman recursion trades off immediate revenue against the option value of holding inventory for future high-price periods.”