Linearity is the property used most often, and the one most often used without noticing:

It is why a return can be split term by term, why subtracting a baseline can be separated algebraically (zero bias additionally needs the score identity), and why the covariance of $AY$ is $A\Sigma A^\top$. Many probabilistic objectives in the course are expectations: the return $\E_\pi[\sum_t\gamma^t r_t]$ (Ch 7–11), expected improvement $\E[\max(0,f-f^{\max})]$ (Ch 4), the ELBO $\E_{q}[\log p(x\mid z)]$ (Ch 6), the policy objective $\E_\tau[R(\tau)]$ (Ch 10). Manipulating expectations is the technical core of the course.

Both say the same thing from two heights: conditional means average back to the original mean. Conditional variance is smaller on average over $V$, not necessarily for every observed value. Everything on the next two slides is those two sentences, cashed.

Calculate and check. $\mathbb E[Y\mid D]=\mathbf{21.6}$. Total variance gives $\operatorname{Var}(Y\mid D)=\mathbb E[\operatorname{Var}(Y\mid\theta,D)\mid D]+\operatorname{Var}(\mathbb E[Y\mid\theta,D]\mid D)=1+0.8=\mathbf{1.8}$.

Check your answer. Parameter uncertainty becomes zero, while the new-reading variance remains 1. Knowing the temperature exactly does not eliminate the specified sensor noise.

↪ used in Ch 2 (the prior–posterior identity), Ch 7 (every Bellman equation), Ch 8 and 10 (every sampled backup).

Total variance explains the first two readings: how much did I learn and what uncertainty remains in an observation. For a policy-gradient baseline, also use the score identity and a variance calculation; total variance alone does not prove unbiasedness.

The distinction is not pedantry. It is the MLE-versus-MAP, point-versus-distribution choice that Chapter 2 was built on, and it is why a Gaussian process reports a credible band rather than a confidence statement.

Lecture 0 named the pair on the slide Two kinds of uncertainty — name them once; Chapter 2's Act 4 ended by writing them as the two terms of $\sigma_Y^2+\tau_1^2$. That is the law of total variance again — the within part is aleatoric, the between part epistemic — which is why this appendix keeps circling the same identity.

Each is a one-line fact. Together they are why so much of the course can carry uncertainty analytically rather than by sampling: a Gaussian goes in, a Gaussian comes out, and the parameters update by matrix algebra rather than by an integral you cannot do.

For general random variables this is false. Take $X\sim\mathcal N(0,1)$ and $Y=X^2$: then $\mathrm{cov}(X,Y)=\E[X^3]=0$, and yet $Y$ is a function of $X$ — as dependent as two variables can be.

↪ every i.i.d. noise model in the course leans on this — it is what lets a diagonal $\Sigma$ stand in for genuine independence.

↪ the reparameterisation trick $z=\mu_\phi+\epsilon\sigma_\phi$ (Ch 6), and linear-Gaussian dynamics $x_{t+1}=Ax_t+Bu_t+w_t$ (Ch 9) — in both, a Gaussian is pushed through a linear map and must come out Gaussian for the method to close.

Property 4 — conditionals are Gaussian. The one that makes GP regression work. For

Read the two braces. The mean moves by how far the observation fell from its own mean, geared by the correlation; with fixed covariance parameters, the covariance drops by a term independent of the observed value — which is why a GP's error bars can be planned before any data arrives, and why Chapter 4 could choose where to sample next.

Chapter 4 quantified this at two correlations from its own kernel. At $\mathrm{corr}(f_1,f_2)=0.966$ one observation shrinks the neighbour's standard deviation to $\sqrt{1-0.966^2}=0.259$ of the prior — a narrow spike. Four points apart, at $\mathrm{corr}=0.573$, it shrinks only to $0.820$ — barely narrower than the prior, a value you still have to buy. Both are $\sqrt{1-\rho^2}$ and nothing more.

This is the quiet foundation of model-free reinforcement learning. When the expectation over unknown dynamics cannot be computed, average over sampled trajectories instead: Monte-Carlo returns and TD targets (Ch 8), REINFORCE and advantage estimates (Ch 10), and the ELBO's reconstruction term, which the reparameterisation trick turns into a one-sample average (Ch 6). ↪ Ch 8 and 10 — the entire data-driven half of the course.

Unbiased for any legal $q$. The catch is entirely in the variance: $\mathrm{var}_q(wg)$ can be far smaller than $\mathrm{var}_p(g)$, far larger, or infinite — and unbiasedness will not warn you which.

• Ch 6, the VAE. The ELBO's KL term pulls the encoder onto the prior, which is what makes the latent space samplable: without it $q_\phi$ is free to encode anywhere and new draws decode to nothing.
• Ch 6, generative training. Minimising KL to the data distribution is maximum likelihood; minimising JS instead is the GAN. Same shape, different divergence, different failure mode.
• Ch 10, trust regions. TRPO maximises an importance-weighted advantage subject to $\E[\mathrm{KL}(\pi_{\text{old}}\|\pi_\theta)]\le\delta$ — the largest step that keeps the new policy close in behaviour, not close in parameters. PPO discourages some large changes with clipping; it imposes no hard KL bound.

Entropy $H(p)=-\E_p[\log p(x)]$ measures spread rather than distance. Maximising it keeps a policy exploratory — the seed of maximum-entropy RL — and it is the term that turns a hard argmax into a soft, probabilistic best response.