Values and policies work together in actor–critic methods. A learned actor supplies actions; its training objective and data determine how it improves.

Bring: Log derivatives, expectations, Bellman targets and feedback.

Parameterise a stochastic policy $\pi_\theta(a\mid s)$ and maximise the expected return:

The trouble is where the parameter lives. It is not in the integrand; it is in the distribution being integrated against, and that distribution contains the dynamics:

One identity does all the work: $\;\nabla p = p\,\nabla \log p$. It turns the gradient of an expectation into an expectation of a gradient:

Put the two halves together and the whole of Act 1 is one line:

For $J(\theta)=\E[\sum_{t=0}^{T-1}\gamma^t r_{t+1}]$, define $G_t=\sum_{k=t}^{T-1}\gamma^{k-t}r_{k+1}$. Then

REINFORCE: sample complete episodes under the current policy, average this expression, and take a gradient-ascent step. The full-return estimator is unbiased under the usual differentiation and integrability assumptions.

The equivalent $Q^\pi$ theorem averages over the policy's state visitation weights. The appendix makes those weights explicit.

A one-step problem has actions A and B. Let $\pi_\theta(A)=p=\operatorname{sigmoid}(\theta)=0.5$. Then the log-probability derivatives are $1-p=0.5$ for A and $-p=-0.5$ for B.

With step size 0.1 and baseline 2:

Look again at what one episode contributes:

Every score term is multiplied by the same scalar — the whole episode's return. One unlucky tail rescales all $T$ of them together. And the return itself is a sum of $T$ random rewards along a random trajectory.

An action at time $t$ cannot affect a reward at time $t' < t$. So those cross-terms have mean zero and contribute nothing but variance. Drop them:

Subtract from each reward-to-go a quantity $b(s_t)$ that does not depend on the action taken:

For two actions, let $Q(\mathrm{off})=-6.7$ and $Q(\mathrm{heat})=-2.9$. The policy heats with $p=\sigma(\theta)$. At θ = 0, p = 0.5 and $dp/d\theta=0.25$. Treat the Q estimates as fixed for this actor step.

Predict: Which way should θ move? Does the fact that both action values are negative change the preferred direction?

Change the fixed Q estimates to −106.7 and −102.9. Keep the same policy parameter. Recalculate the actor gradient and the value baseline.

Put the two together. With $b(s)=V^\pi(s)$ and $\hat Q_t \to Q^\pi(s_t,a_t)$, what multiplies the score is the advantage:

Suppose the observed full return is 4.6, the baseline is 2, and the next-state critic is 3. With reward 1 and $\gamma=0.9$:

At a true terminal state, continuation value is zero. GAE combines multiple TD errors to control this trade-off.

• A3C / A2C — run many actors in parallel on separate copies of the environment. Their rollouts provide more diverse recent experience without a large replay buffer. A2C synchronises updates; A3C permits parameter lag, so neither parallelism nor freshness means exact sample independence. (Mnih et al., 2016)
• $n$-step returns — bootstrap after $n$ steps rather than at the end: $\;R_t = \sum_{i=1}^{n}\gamma^{i-1}r_{t+i} + \gamma^{n}V(s_{t+n})$. A full Monte Carlo return continues to termination with zero bootstrap; one-step actor–critic uses $n=1$.
• GAE — do not pick $n$; average all of them geometrically, $\;\hat A_t^{\mathrm{GAE}(\gamma,\lambda)} = \sum_{l\ge0}(\gamma\lambda)^l\,\delta^V_{t+l}$. (Schulman et al., 2016)

Each worker interacts with its own environment copy and forms an $n$-step target. For the main lecture's undiscounted episodic convention,

If an episode terminates within the window, stop the reward sum there and use zero continuation value. Discounted versions retain the corresponding powers of $\gamma$.

At a fixed state, data from behaviour policy $\beta$ can estimate an expectation under $\pi_\theta$ using importance ratios:

This requires $\beta(a\mid s)>0$ wherever the target policy assigns probability. If $\pi(A\mid s)=0.8$ but $\beta(A\mid s)=0.4$, observed A actions receive weight 2.

The source deck states the debt in one sentence: "although DQN solves problems with high-dimensional observation spaces, it can only handle discrete and low-dimensional action spaces". Consider two common choices for using a Q-network with continuous actions.

Make the actor deterministic, $a = \mu_\theta(s)$, and let it ascend the critic's action-gradient: (Silver et al., 2014)

At one state let $Q(a)=5-(a-2)^2$ and let the actor output $a=\theta$. Initially $\theta=0$.

A step of size 0.1 gives $\theta_{\mathrm{new}}=0+0.1(4)=0.4$. The critic's score rises from 1 to 2.44.

Put Lecture 9's Act 3 and this act side by side. Both produce feedback, using different assumptions and procedures.

A deterministic policy has no randomness to explore with, and the obvious repair — act uniformly at random — is a poor one. The source deck gives three reasons, and the third is unusual in this course:

• random actions drive the policy update into regions where the critic is not accurate (Lecture 5's adversarial-optimiser hazard, in an RL costume);
• exploring unseen states needs action that is consistent along the episode, not resampled each step;
• and on real hardware, rapid random input changes can strain actuators.

A DDPG actor can exploit an erroneously high critic value. TD3 changes the target and update timing. Here $d$ is the terminal indicator and $\epsilon$ is zero-mean Gaussian target noise.

Let $r=1$, $\gamma=0.9$, and $d=0$. A target actor proposes 0.5; the clipped target noise is 0.1, so the smoothed action is 0.6 within the bounds $[-1,1]$.

Using the first estimate alone would give 8.2 at this action. If the transition were terminal, both targets would instead be 1. An illustrative delay of two means two critic updates for each actor update.

SAC trains a stochastic actor from replay. Its discounted episodic objective includes policy entropy:

Here $\alpha_H>0$ is the entropy temperature, not the gradient step size. Consider a one-step, two-action illustration with rewards 2 and 1.8, and $\alpha_H=0.5$:

Draw a next action $a'\sim\pi_\theta(\cdot\mid s')$ for each replayed transition. Train both critics toward

The actor minimizes, over replayed states and newly sampled current-policy actions,

In supervised learning a bad step costs one bad update; the next minibatch arrives from the same fixed dataset and pulls you back. In RL it is worse, because

Maximise an importance-weighted advantage, subject to staying inside a KL ball around the policy that collected the data: (Schulman et al., 2015)

Practical TRPO approximates a KL-constrained step. PPO uses a simpler surrogate with a similar intent: with $r(\theta) = \pi_\theta(a\mid s)/\pi_{\theta_{\mathrm{old}}}(a\mid s)$, (Schulman et al., 2017)

Let $\epsilon=0.2$. For one sampled action, compare $r\hat A$ with $\operatorname{clip}(r,0.8,1.2)\hat A$, and keep the smaller number.

Both can use stochastic neural policies in continuous control. PPO clipping is not a hard KL guarantee; SAC's replay is not a guarantee that arbitrary offline data will suffice.

Use a fixed start distribution $\mu$, bounded rewards, and $\gamma<1$: $J(\theta)=\sum_s\mu(s)V^\pi(s)$. Define $\phi(s)=\sum_a\nabla_\theta\pi_\theta(a\mid s)Q^\pi(s,a)$.

Differentiate $V^\pi(s)=\sum_a\pi_\theta(a\mid s)Q^\pi(s,a)$ and use the Bellman equation:

The transition law has no direct parameter derivative, but the future policy still does. Repeated substitution produces discounted state visitations. This is different from differentiating an objective with an arbitrary stationary start distribution that itself depends on $\theta$.

Let $d_\gamma^\pi(s)=(1-\gamma)\sum_{t\ge0}\gamma^tP_\pi(S_t=s\mid S_0\sim\mu)$. This is a normalized discounted visitation distribution.

The factor $1/(1-\gamma)$ can be absorbed into a learning rate; the state weighting cannot simply be discarded. Finite-horizon episodic versions use a sum over time instead.

At a fixed state, put $z=\nabla_\theta\log\pi_\theta(a\mid s)$. Since $\E[z\mid s]=0$, subtracting a fixed $b(s)$ leaves the mean unchanged.

For the variance of one score-weighted contribution, minimize $\E[\lVert z\rVert^2(G-b)^2\mid s]$. Differentiation gives

This equals $V^\pi(s)=\E[G\mid s]$ when the score weight is constant or suitably uncorrelated with return. Correlations across time add further considerations for a whole-trajectory variance optimum.

With $\delta^V_t = r_{t+1} + \gamma V(s_{t+1}) - V(s_t)$, the $k$-step advantage estimators telescope:

$k=1$ is the one-step TD advantage: low variance, biased by whatever error $V$ carries. $k=\infty$ is the Monte-Carlo advantage: unbiased, and as noisy as REINFORCE. GAE takes the exponentially weighted average of all of them, (Schulman et al., 2016)

which collapses to a single running sum of TD errors. $\lambda=0$ recovers one-step actor–critic, $\lambda=1$ recovers Monte-Carlo.

This is TD($\lambda$) applied to the advantage rather than to the value, and the discount $\gamma$ is doing double duty: as the problem's discount, and as a variance-reduction parameter that downweights delayed effects at the cost of bias. Lecture 8's Backup 1 drew the same dial for MC-versus-TD; it is the same dial.

Maximise $L(\theta) = \E[r(\theta)\hat A]$ subject to $\bar D_{\mathrm{KL}}(\theta_{\mathrm{old}},\theta)\le\delta$.

Step 1 — local models. To first and second order about $\theta_{\mathrm{old}}$,

with $g=\nabla_\theta L$ and $H$ the Fisher information matrix. The KL has zero gradient at $\theta_{\mathrm{old}}$, which is why its leading term is quadratic.

Step 2 — a linear objective on an ellipsoid. The maximiser lies along the natural-gradient direction $H^{-1}g$, scaled to the KL radius:

Step 3 — practice. $H^{-1}g$ is obtained by conjugate gradient without ever forming $H$, and the step is backtracked until the sample-estimated KL and surrogate improvement pass the line-search checks — which is Lecture 1's accept-or-shrink test, verbatim.

The derivation assumes an invertible positive-definite local Fisher matrix and a nonzero gradient. Implementations use damping and estimated quantities. PPO removes the second-order solve, but does not inherit a hard trust-region or monotonic-improvement guarantee.

These methods share policy optimization, but differ in objectives, estimators, data distributions and update rules. SAC explicitly includes entropy; PPO uses a clipped surrogate. A single vanilla policy-gradient formula does not specify every algorithm below.

With several agents, a critic can use the joint state and actions during training, while each actor uses its own observation when deployed.

This is centralised training with decentralised execution. Other agents' changing policies make the learning problem different from single-agent DDPG; actor gradients do not by themselves guarantee convergence to a Nash equilibrium.

Every object of Lecture 9 has a data-driven counterpart. Keep this in sight all lecture.

The most literal idea, and the exact rhyme of the detour Lecture 8 took: estimate the dynamics from data, then run Lecture 9 unchanged.

This is system identification, it is the oldest idea in adaptive control, and it is perfectly valid.