Model error can affect both plans and learned policies. Follow the core model-use loop before differentiating an optimizer.

Bring: Bellman updates, feedback, SAC and the idea of a latent state.

The naive plan is to fit a model, plan in it, and execute. It works for exceptionally simple cases — those in which the fitted model generalises over the whole state space.

Before reaching for a network, note how much structure the control tradition already knows how to fit — and that every one of these was built to be planned in.

These are alternative structures, not a strict ranking of expressivity or planning difficulty. The advanced differentiable-control section studies models whose structure also helps optimization.

Append what actually happened, refit, repeat. If the model is wrong where the plan goes, then go there and collect data:

At each step, optimise a short horizon of actions, execute only the first, observe, re-plan:

Use $\hat x_{t+1}=\hat x_t+u_t$, start at $x_0=2$, and compare two-step plans with cost $u_0^2+u_1^2+x_2^2$.

Among these candidates, choose $(-1,0)$ and execute only $u_0=-1$. Suppose the measured next state is 1.4, not the predicted 1.

The planning model is $x'=x+u$, with cost $(x')^2+u^2$. Starting at x = −2, it chooses u = 1 and predicts x′ = −1. The real next measurement is −1.5.

Predict: If the model has not yet been refitted, should the next plan start from −1 or from −1.5?

At the measured state x = −1.5, change the learned dynamics to $x'=x+gu$ with g = 0.5. For cost $(x+gu)^2+u^2$, solve $g(x+gu)+u=0$ for u.

At the measured state $s_t$, compare candidate action sequences using the learned dynamics and the same objective:

Let the one-step model error be at most $\epsilon$ and let true dynamics amplify state differences by at most $L$. For the same actions and $e_0=0$,

For $\epsilon=0.1$:

Version 3 still acts on the model's mean: it takes actions believed good in expectation, which provides no explicit incentive to seek information. It may still visit unfamiliar states.

Split the error the way Lecture 0 split it, and the way Lecture 2 wrote it:

After one prediction, a future state is uncertain. Feeding only its mean into a nonlinear model generally loses information:

For an illustrative $X$ equally likely to be $-1$ or $1$ and $f(x)=x^2$, the true next-state mean is 1, while the plug-in mean prediction is $f(0)=\mathbf0$.

The source uses Monte Carlo propagation and Gaussian moment approximations. An ensemble represents alternative models; transition noise represents variability within a model. Neither a particle count nor ensemble agreement alone certifies calibration.

The simplest reunion: generate transitions with the learned model and use them in a compatible value or policy learner. The update formula can stay the same even though the data quality changes.

Let $Q(s,a)=2$, $\alpha=0.2$, $\gamma=0.9$, and next maximum value 5.

Keep collecting real experience to update the model. The synthetic buffer expands the learner's training opportunities; it does not add new measurements of the environment.

Count the trade-off: 100 real starting states produce 100 synthetic transitions at horizon 1, or 500 at horizon 5. The second option uses five times as many predictions, with more opportunities for compounding error. These are illustrative budgets, not recommended settings for every task.

Raw observations may be high-dimensional and partial. A useful latent representation should retain information needed to predict and control; it is not guaranteed to be low-dimensional or Markov.

The observation may be an image. Infer a compact model state from the observation history, then predict future latent states and rewards under the actor's actions.

From an inferred state, the actor and model predict rewards 2, then 1. The critic assigns value 4 to the final imagined state. With $\gamma=0.9$ and no termination:

PETS emphasizes probabilistic ensembles and trajectory sampling. Dreamer does not need to search action sequences at every real step: the actor has already learned from imagined futures.

The 2025 paper reports results across more than 150 tasks in eight domains and collecting diamonds in Minecraft without human demonstrations or curricula. This is evidence about the evaluated settings; it does not establish optimality or success on every new control problem.

A learned model can support planning, simulated experience or policy learning in imagination. These operations connect the course's value and control methods without making them identical.

These online methods use real experience to correct their learned models. Differentiating a planner for task learning or joint design is an advanced extension in the appendix.

For a convex QP planning subproblem $\min_z \tfrac12 z^\top Q z + q^\top z$ subject to $Az=b$, the optimum is characterised by its KKT system:

Rather than differentiate $z^*$ directly, take differentials of both conditions:

Each parameter perturbation has its own right-hand side; factorization can be reused. For a scalar outer loss, an adjoint solve yields all parameter gradients. Write $[d_z;d_\nu]$ for the solution of the transposed KKT system against $[-\nabla_{z^*}\ell;0]$. Then the chain rule gives $\nabla_Q\ell = \tfrac12(d_z z^\top + z\,d_z^\top)$, $\nabla_q\ell = d_z$, $\nabla_A\ell = d_\nu z^\top + \nu\,d_z^\top$ and $\nabla_b\ell = -d_\nu$.

LQR: the structured linear solve can reuse Riccati factorizations. Nonlinear MPC: differentiate a converged local approximation, with a stable active set and nonsingular KKT matrix. If the solver has not converged, differentiating its iterations describes that finite algorithm, not an exact optimum. (OptNet; Differentiable MPC)

Objective. Minimise trajectory cost while forcing the trajectory distribution to agree with the policy:

Lagrangian and alternation. Form $\mathcal L_{\mathrm{GPS}}(\theta,p,\lambda) = \E_{p(\tau)}[\ell(\tau)] + \sum_t \lambda_t D_{\mathrm{KL}}\big(p(x_t)\pi_\theta(u_t\mid x_t)\,\|\,p(x_t,u_t)\big)$ and minimise by turns:

1. w.r.t. $p(\tau)$ — trajectory optimisation by iLQG under time-varying linear-Gaussian dynamics; the LQR structure makes the local controller $p(u_t\mid x_t)=\mathcal N(K_t(x_t-\hat x_t)+k_t+\hat u_t,\Sigma_t)$ fall out automatically;
2. w.r.t. $\theta$ — supervised regression: minimise the weighted sum of KL divergences between $\pi_\theta$ and the local controllers;
3. duals $\lambda$ — dual gradient ascent (in practice often scheduled rather than updated).

Why the dual step is cheap. With $x^*(\lambda)=\argmin_x \mathcal L(x,\lambda)$ and $g(\lambda)=\mathcal L(x^*(\lambda),\lambda)$, the chain rule gives $dg/d\lambda = (d\mathcal L/dx^*)(dx^*/d\lambda) + d\mathcal L/d\lambda$, and the first term vanishes because $d\mathcal L/dx^*=0$ at the argmin. So $dg/d\lambda = d\mathcal L/d\lambda$ evaluated at $x^*$ — no derivative through the inner solve is needed. This envelope calculation differs from the full implicit derivative of an optimizer, developed in the advanced differentiable-control section. Why it is sample-efficient. Time-varying linear-Gaussian dynamics are a strong but reasonable local assumption for a physical system, so each trajectory is optimised with far fewer samples than general model-free search — the return on injecting control-theoretic structure.

Propagating uncertainty through a horizon. If the state input is itself uncertain, $x\sim\mathcal N(\mu_x,\Sigma_x)$, then $p(f(x)\mid \mu_x,\Sigma_x) = \int p(f(x)\mid x,\mathcal D)\,p(x)\,dx$, which is intractable and is handled either by Monte-Carlo — sample $x^t\sim p(x)$ and average $p(f(x^t)\mid x^t,\mathcal D)$ over $T$ particles — or by computing moments analytically where possible and approximating the result as Gaussian, which is PILCO's choice. Rolling a large number of such trajectories and taking their mean and standard deviation gives the fan of predictions that widens with horizon: compounding uncertainty, drawn.

The gain. A learned model can be queried without environment interaction, so one real transition informs the value or policy at many imagined states. Empirically this cuts the required real samples by orders of magnitude — decisive whenever interaction is expensive, slow or dangerous: robots, chemical processes, buildings, patients.

The hazard. Planning trusts $f_\theta$. Where $f_\theta$ is wrong the planner optimises a fiction, and errors compound over the horizon because each step's error is the next step's input. This is the dynamic cousin of Lecture 5's overestimation trap: an optimiser exploiting a model's blind spots.

The constrained program, with an augmented Lagrangian:

In the stochastic form, the local controller is constrained to stay near the previous one:

The naive version of route 2 collects a dataset from an MPC expert and fits the policy to it. That ignores the one thing imitation always breaks on.

The states visited by the teacher are not the states visited by the learner, so nothing guarantees long-horizon performance. PLATO repairs it by pulling the teacher toward the student at every step:

Every version in Act 2 fits $f_\theta$ by minimising $L\big(x_{t+1}, f_\theta(x_t,u_t)\big)$ — a loss that is task-independent and data-dependent. An aggregate prediction objective need not emphasize the errors that most affect action selection. Its weights and the training distribution determine which errors it prioritizes.

For a convex quadratic planning subproblem — an LQR, or a QP —

With a locally unique optimum and a nonsingular KKT system (and a stable active set for inequalities), the solution is differentiable locally. Do not differentiate $\tau^*$; differentiate the conditions it satisfies:

A one-step planner chooses $u^*(\theta)$ by minimizing $\tfrac12(u-\theta)^2+\tfrac12u^2$.

The optimality condition is $2u^*-\theta=0$. Differentiate that condition:

If the training goal is an expert action 1, let $\ell=\tfrac12(u^*-1)^2$. At $\theta=0$, $u^*=0$ and $d\ell/d\theta=(-1)(1/2)=-0.5$.

• Imitation learning. Fit $\theta$ so that $\pi_\theta(s)\approx\hat\pi(s)$, inheriting a controller's structure while learning from demonstrations rather than from a hand-written cost.
• Structure as inductive bias. The constraint $x_{t+1}=f_\theta(x_t,u_t)$ is baked into the architecture, so far less data is needed than for a generic policy network, and extrapolation improves.
• Task-aligned models. Accuracy lands where the task needs it, which is the direct repair of Act 2's "prediction-accurate, control-useless" failure.

Every version so far fits a model and hopes the planner copes. Turn the question round: what property should a learned model have so that planning inside it is easy?

Read the first two as connections to control. Then follow Dyna → MBPO and compare Dreamer's latent imagination. Dreamer also trains a critic; these categories describe uses of a model, not mutually exclusive algorithm families.

Compose policy and model along the horizon and differentiate the whole chain:

Starting from $D_\theta s_0=0$, propagate these sensitivities and sum $\nabla_\theta J=\sum_t(r_sD_\theta s_t+r_aD_\theta a_t)$, with discount factors when needed.

PILCO propagates uncertainty in a GP model. Dreamer later learns a latent state model and combines imagination with a critic. Neither removes the need to learn useful dynamics from real experience.

Lecture 1 wrote an optimisation. Lecture 9 added the dynamics as a constraint. Add one more constraint and you have this lecture.

Lecture 1 optimised a wind-farm layout with a trust region, and Lecture 5 designed from a fixed dataset. Neither could touch the controller. Differentiable control closes that loop.

Many real problems are bilevel — an upper-level design $p$ wrapping a lower-level control problem: