Read the third clause carefully, because it is the whole lecture in one symbol:

What we seek is not a sequence of numbers $u_0,\dots,u_{K-1}$ but a rule $\gamma_k$ from state to control. Control theory insists on this from its very first slide, for the same reason Lecture 7 wanted a policy rather than a plan: a rule is still valid from wherever the system actually lands.

Applied directly, it gives the recursion — and it is Lecture 7's, letter for letter:

Each $u_k^*$ is read off as the argument attaining the right-hand side — what the source deck calls a single-shot optimisation. That phrase is worth keeping: a problem that unfolds over time has been reduced to one static optimisation performed once per instant. We meet the move again in Act 2 and again in Act 4.

Two of those three differences are cosmetic. The expectation $\sum_{s'}P[\cdots]$ collapses to a single point because the dynamics are deterministic — put the noise back and it returns unchanged. Cost versus reward is a sign.

Calculate and check. $dc/du=2(x+u)+2u=0$ gives $u^*(x)=\mathbf{-x/2}$. At $x=-2$, the command is 1. The rule is feedback: it recomputes the action from the current state.

Check your answer. $u^*=-(-1.5)/2=\mathbf{0.75}$. Reusing 0.5 ignores the new state measurement. Feedback revises the decision; it does not require the previous prediction to have been correct.

So the continuous action space is not a nuisance detail; it is the structural fact that makes this a different tradition. Everything that follows is a way of coping with it: first solve a quadratic problem explicitly (Act 2), then interpret the smooth Bellman condition (Act 3), or characterise its solution along one trajectory rather than everywhere (Act 4).

The cost-to-go is defined exactly as before, over the remaining interval:

The boundary condition is the terminal cost. In Lecture 7 the recursion was seeded at $V(K,\cdot)=g_K$; here it is seeded at $t=T$. Same seed, same backward direction.

"Sufficient" is a strong word and it is earned: a solution of the PDE is a certificate that every other admissible trajectory costs at least as much. The four-line verification argument is Backup 2, and it is the only place in this lecture where something is proved optimal rather than merely derived.

Three prices, and each is a limitation we will spend the rest of the course working around:

• It needs the model. $f$ and $g$ appear explicitly inside the minimisation. Delete them and there is no equation left to solve — which is Lecture 10.
• It assumes $V$ is differentiable. Often it is not; the honest fix is a weaker notion of solution (viscosity solutions), which buys existence at the cost of the clean formula. (Crandall & Lions, 1983)
• It does not survive dimension. A value field over $x\in\R^n$ is exponentially large — the same curse Lecture 7's table faced, now with the table replaced by a continuum.

So HJB is the characterisation, rarely the computation. LQR is a particularly useful solvable case: a quadratic value function closes the equation. Other special problems also admit explicit solutions.

This is Lecture 1's Lagrangian move, applied once per instant. Adjoin $\dot x = f$ with a multiplier $\lambda(t)$ — the costate — and collect the terms into the Hamiltonian:

One multiplier per instant, and a whole function $\lambda(\cdot)$ instead of a vector. The constraint set of Lecture 1 has become the dynamics themselves.

Read the shape of it. The state runs forward from $x(0)=x_0$; the costate runs backward from $\lambda(T)$; and they are coupled. This is a two-point boundary-value problem, not an initial-value problem — you cannot simply integrate.

For the scalar quadratic value $V(t,x)=P(t)x^2$, the costate is $\lambda=2P(t)x$. At the infinite-horizon solution $P=1$, a small state change $\Delta x$ changes the value by approximately $2x\Delta x$.

Feedback beats an open-loop plan for the reason Lecture 7 already gave: $\gamma(t,x)$ is valid from any state you actually reach, while $u^*(t)$ is valid only along the trajectory you predicted. But Pontryagin scales where HJB cannot, which motivates trajectory methods, alongside direct optimization approaches. Its machinery returns in Lecture 10 — REINFORCE is a trajectory view — and in Lecture 11, where a learned policy imitates an iLQR teacher.

Three mathematical tools with different roles; numerical solutions still need checking:

Lecture 10 follows the same teaching move as Lecture 8: keep the decision objective, but learn from experience when a complete model is unavailable. The linear gain $K$ is one feedback policy. A network $\mu_\theta(s)$ is a more general policy class trained using sampled performance; it is not guaranteed to recover the LQR solution or a global optimum.

A learned actor can avoid a fresh continuous action search at every step. The difficulty moves into training and evaluating that actor.

Where Lecture 7 had a stochastic transition over a finite set of states, control theory studies a system — usually continuous, often deterministic — driven by a control:

The goal is identical to Lecture 7's: not a plan, but a feedback law $u = \gamma(t,x)$ — the best control at every state and every time. The differences are of tradition, not of substance:

• finite action set $\to$ continuous control $u \in U \subseteq \R^m$;
• probabilistic transition $T(s,a,s') \to$ differential dynamics $f$;
• sum $\to$ integral; reward and $\max$ $\to$ cost and $\min$; a backup $\to$ a PDE.

That wall is not this lecture's starting point — we begin from Lecture 7, not Lecture 8. But it is one of the seams where the two lineages meet, and it is worth naming now because it recurs three times today:

Control theory has lived with the continuous $\arg\min$ since 1956, and its three answers are today's acts. Lecture 10's answer — learn a network that outputs the minimiser — is the fourth, and it is a descendant of the third.

Read it as one claim: HJB is the Bellman optimality equation made infinitesimal. Lecture 7's intuition transfers wholesale; only the calculus changes. And keep the last two rows in view — that is the seam where, in Lecture 10, samples will replace the solver.

Taylor-expand the second term, cancel $V(t,x)$ from both sides, divide by $\delta$ and take the limit. Assuming $V$ is continuously differentiable, what survives is the Hamilton–Jacobi–Bellman equation:

The full three-line derivation is Backup 1. Compare it with Lecture 7's: identical, with $\delta\to 0$ where Lecture 7 had a one-step sum.

For infinite-horizon LQR, standard sufficient assumptions are stabilizability of $(A,B)$ and detectability of $(Q^{1/2},A)$, with $R\succ0$. The stabilizing Riccati solution gives a stationary value and gain:

The discrete-time problem $x_{t+1}=Ax_t+Bu_t$, $J=\sum(x^\top Qx + u^\top Ru)$ runs the identical argument and lands on $P = Q + A^\top PA - A^\top PB(R+B^\top PB)^{-1}B^\top PA$ with $K = (R+B^\top PB)^{-1}B^\top PA$. Backup 3 carries both derivations in full.

Value iteration, with the value function carried in closed form instead of enumerated. The uniqueness of the positive-semidefinite $P$ plays the role of the uniqueness of the Bellman fixed point, and — the source deck's own remark — infinite-horizon LQR is just steady-state finite-horizon LQR. One backward sweep, run to convergence.

Try $V(t,x) = x^\top P_t\, x$. Then $\partial_x V = 2P_t x$ and $\partial_t V = x^\top \dot P_t x$, and the HJB equation reads

The inner minimisation — the continuum that Act 1 warned about — is now a convex quadratic in $u$. Set the gradient to zero: