Learning route — one backup before a whole algorithm
Bring: Expectation, a discounted sum and finite tables.
First pass
What to do
Follow the idea
State/action/return → Bellman backup → evaluate and improve → PI/VI
Work without the solution
Calculate a temperature-control backup, then repeat after changing a transition.
Return later
The contraction proof and detailed convergence assumptions are second-pass reading.
For the temperature thread: predict → calculate → reveal and check → change one condition. Complete the core calculation before reading the research extensions.
Q1. A greedy rule scores an action by its own reward. Build the object that scores a situation instead.
ACT 1The arena, and the value of a stateLecture 7
Why a sequence is not many single choicesfrom the deck's opening example
The choice, with numbers
Standing in your final undergraduate year, three actions:
action
reward now
get a job
$4,000 / month
go to graduate school
$500 / month
start a company
$0 / month
A one-step-greedy rule takes the job and never looks up.
Why that is the wrong reading
Graduate school pays 500 — and buys a state. From it a further action (publish X papers) opens a distribution over researcher, professor, private company, officer, no job.
The 500 was never the point. The state it bought was.
Actions are coupled through the state: what you do now changes the options and rewards you will have later. So an action cannot be scored by its own reward — only by the whole future it unlocks.
Which forces a new object. Instead of a number f(x) attached to a decision, we need a number attached to a situation — the total future reward a good plan can extract from it — plus an equation tying each situation's number to its successors'. That equation is Bellman's, and it is the lecture.
Q1 Why not greedy?Q2 What must value satisfy?Q3 How do we solve it?Q4 Why does it converge?
A finite MDP is the tuple ⟨S,A,T,R,γ⟩:
states s∈S and actions a∈A, both finite;
a transition functionT(s,a,s′)=P(St+1=s′∣St=s,At=a)=P(s′∣s,a) — the source deck's own gloss: also called the model, or the dynamics;
a reward functionR(s,a,s′), or its expectation E[rt+1∣St=s,At=a,St+1=s′] if the reward is itself random;
a discountγ∈[0,1), and a start state s0 (plus terminals sT for episodic tasks).
Stationary means T and R do not depend on t. That is what lets a chain of time slices collapse into a single recursive equation rather than one equation per step — and the two spellings T(s,a,s′) and P(s′∣s,a) name the same object. Lecture 8 will use the second, and delete it.
The goal is not a trajectory. It is a policy π:S→A that names the best action at every state.
Given the present state, the future and the past are independent. A state signal that retains all the relevant information is called Markov — the position and velocity of a projectile under known dynamics. A game state must also include the player to move and any history required by its rules.
The consequence is the one that matters, and it is not obvious:
π∗(st,st−1,…,s0)=π∗(st)
For our finite discounted MDP, an optimal stationary policy can use only the present state. For a finite horizon, the best action can also depend on how much time remains: use πt(s) or include t in the state.
Without it, "the value of a state" would be meaningless — value would have to be indexed by every path that could have led there, and there are exponentially many. The Markov property is what makes a table with one row per state a sufficient answer. Lecture 3 arrived at the same condition from the other side: it is a missing-edge pattern in a Bayesian network over S1,…,ST.
The agent maximises accumulated reward, not immediate reward. Two settings:
Episodic — there is a last step
Ut=rt+1+⋯+rT=k=0∑T−t−1rt+k+1
A maze, a game of Go, a game of chess. The sum is finite because the episode is.
Continuing — there is not
Ut=rt+1+γrt+2+⋯=k=0∑∞γkrt+k+1
With bounded rewards, γ<1 makes the sum finite. Our convention throughout: take at at st, then receive rt+1 and st+1.
γ=0 — myopic: only the immediate reward counts, and we are back to the greedy rule that failed;
γ→1 — far-sighted: distant reward counts almost as much as near reward.
A warning from the source deck worth repeating: the reward signal communicates what you want achieved, never how. Reward a chess engine for taking pieces and it will take pieces; it will not win. Every reward specification is a modelling decision of the kind Lecture 1 taught, made under the extra hazard that the optimiser has many steps in which to find the loophole.
Those three words are the whole distance from Lecture 1. We do not want a policy that is best on average over some start distribution; we want one that is simultaneously best from every state at once. It is not obvious that such a policy exists. That it does is the first gift of the Bellman equation.
A policy may be deterministic, a∗=π∗(s), or stochastic, p(a∣s)=π∗(s,a). For our finite, discounted, stationary MDP with bounded rewards, a deterministic stationary optimal policy exists, so we take π deterministic throughout. Stochastic policies return in Lecture 8, where randomness buys exploration, and again in IE579, where against an opponent it buys unpredictability.
The expected return from following π starting at s.
Qπ — how good is it to do this here
Qπ(s,a)=Eπ[Ut∣St=s,At=a]
Take a now — whatever π would have said — then follow π forever after.
Eπ is not an expectation over the policy; with π deterministic it is an expectation over every stochastic transition the policy leads you through. And both are defined with respect to a policy: there is no such thing as the value of a state, only the value of a state under a way of behaving.
This is exactly what the greedy rule lacked. A value scores a state by the entire future a policy can extract from it, not by one step's reward.
A corridor with a small prize one step to the left of START and a large prize six steps to the right. Turn γ and watch the arrows flip one cell at a time, from the far end inwards. The frontier is not a matter of taste — each cell has its own threshold, and they are exact: START switches at γ=10−1/5≈0.631, where γ5⋅10 overtakes 1; its neighbour at 10−1/3≈0.464; and the three cells nearest the large prize never point left at all. Same MDP, same optimality principle, opposite answer.
The Markov property says the future depends on the past only through the current state. Why does the whole apparatus need it?
Without a sufficient state, the value of "here" can depend on earlier observations; one may need to include history or a belief state. The Markov property is what collapses an exponential tree of histories into a graph of states, and everything after it — the Bellman equation, dynamic programming, every algorithm in Lectures 8 to 12 — is built on that collapse. Choosing the state representation is choosing whether the assumption holds.
Q2. The value of a state sums an infinite future. Turn that sum into a relation between neighbours.
ACT 2The Bellman equationLecture 7
Collapse the tree
Q1 Why not greedy?Q2 What must value satisfy?Q3 How do we solve it?Q4 Why does it converge?
Write out what Vπ(s) actually averages over. From s, every action; from each action, every successor; from each successor, every action again — a tree that never stops branching.
all trajectoriesevery action, every outcome, forever
fix πthe decision branches close; only chance branches remain
one step, then a valueR+γVπ(s′)
Fixing the policy prunes every decision branch. What remains is one layer of chance — and beyond it, a subtree that is itself a value function, because the Markov property says the future from s′ depends on nothing but s′.
The infinite tree closes on itself. That is the entire trick.
The value of a state = the expected immediate reward + the discounted value of where you land.
Read as a picture: a state nodes leads to a chance node(s,π(s)), whose branches carry the weights T(s,π(s),s′) and end at successors holding Vπ(s′). Every equation in this lecture is that diagram with one thing changed.
For a finite S this is a system of ∣S∣ linear equations in ∣S∣ unknowns — solvable in principle by matrix inversion, in O(∣S∣3). It has turned "sum an infinite future" into "relate each state to its neighbours". That is the trade dynamic programming lives on.
Q is the more general object: it keeps the first action free, then defers to π. Force a=π(s) and it collapses back to V.
Why we will end up storing Q, not Vthe source deck's sentence, kept whole
"The Q function effectively caches the results of all one-step-ahead search."
That cache is worthless while the model is free — with T in hand you can recompute the lookahead whenever you like. It becomes the whole ballgame the moment T is gone, because the lookahead is precisely the thing you can no longer perform. Lecture 8's first structural decision follows from this line and nothing else.
The two maxima choose actions at different times. We observe s′ before choosing a′. This is not permission to swap a maximum and an expectation arbitrarily.
A single optimal continuation policy works from every successor in our finite discounted MDP. That is why the remaining value can be written as V∗(s′).
Temperature thread — calculate a temperature-control backupshared teaching example · predict before revealing the calculation
At x=−2, choose off (u=0) or heat (u=1). For now, x′=x+u is known. Reward is r=−[(x′)2+u2], discount is 0.9, and the current continuation estimates are V(−2)=−3, V(−1)=−1.
Predict: Compare the immediate cost and the continuation value before choosing the action.
Calculate and check.
Action
Reward
One Bellman target
Off
−4
−4+0.9(−3)=−6.7
Heat
−2
−2+0.9(−1)=−2.9
This backup selects heat. The supplied V values are inputs to this sweep, not a claim that convergence has occurred.
An action is evaluated by what happens now and the value of the state it reaches.
Try it — heating succeeds only 20 percent of the timework independently · reveal only after writing an answer
Off behaves as before. Heat reaches x′=−1 with probability 0.2 and remains at x′=−2 with probability 0.8, while still paying effort u2=1. Keep the same continuation estimates. Does the backup still choose heat?
Check your answer. A failed heating step has reward −5 and target −5+0.9(−3)=−7.7. The heat target is 0.2(−2.9)+0.8(−7.7)=−6.74. Off gives −6.7, so this backup chooses off.
Changing a transition changes both next-state cost and continuation value. Take the expectation over complete outcomes.
The principle of optimality, and the policy read off
Bellman's principle of optimality
An optimal policy has the property that, whatever the first action and the state it leads to, the remaining decisions must themselves be optimal from that state.
Which is this lecture's thesis in formal dress: a good plan stays good from wherever you land. And it is what makes a daunting search over plans tractable — because it licenses solving the problem one state at a time. The optimal policy then falls straight out:
The source deck's gloss: any greedy policy with respect to V∗ is optimal, because V∗ already accounts for the reward consequences of all possible future behaviour. Greed was never the enemy. Greed applied to the immediate reward fails; greed applied to the right quantity is exactly optimal. Act 1's question is now answered: we were not too greedy, we were greedy about the wrong number.
Check the timing — guessing a coin before or after the toss
A fair coin will be tossed. Choose heads or tails; a correct guess pays 1.
Decide before observing
Either guess has expected reward 0.5.
amaxE[R(a,C)]=0.5.
Decide after observing
You can match every outcome and collect 1.
E[amaxR(a,C)]=1.
Putting the current action's maximum inside the expectation grants extra information. In Bellman's equation, only the next action is chosen after the next state is observed.
The Bellman optimality equation contains both a max over actions and an expectation over next states. Why can the two not be swapped?
Moving the max inside the expectation gives E[max…], which values the ability to pick an action after the dice are thrown. That quantity is an upper bound and it describes a clairvoyant, not an agent. The gap between the two is the value of information, and keeping them in the right order is what makes the equation describe a decision made under genuine uncertainty.
Q3. The Bellman equations are conditions, not procedures. Turn them into iteration.
ACT 3Solving it with the modelLecture 7
Two halves of one machine
Q1 Why not greedy?Q2 What must value satisfy?Q3 How do we solve it?Q4 Why does it converge?
Dynamic programming is, in the source deck's words, "a collection of algorithms that compute optimal policies given a perfect model of the environment as a Markov decision process". Its key idea is to use value functions to organise the search over policies. It has exactly two moves:
Policy evaluation — make V agree with π
Vk+1(s)←s′∑T(s,π(s),s′)[R+γVk(s′)]
Sweep the expectation backup until it stops changing.
Policy improvement — make π greedy on V
π′(s)=aargmaxs′∑T(s,a,s′)[R+γVπ(s′)]
One argmax over a finite set, per state.
Set the right-hand box beside Lecture 1's x∗=argmaxxf(x) — the source deck does exactly that, on one slide. Policy improvement is Lecture 1's argmax, executed once per state over ∣A∣ candidates instead of over a continuum. What is new is not the optimisation. It is the quantity being optimised.
Initialise V₀(s) ← 0 for every s ∈ S
Repeat for k = 0, 1, 2, …
for each state s:
V_{k+1}(s) ← Σ_{s'} T(s, π(s), s') [ R(s, π(s), s') + γ V_k(s') ]
until max_s | V_{k+1}(s) − V_k(s) | ≤ ε
This is a full backup: each sweep backs up the value of every state, from every successor, once. Nothing is sampled, nothing is skipped. Hold that phrase — Lecture 8's central move is to replace it with a sample backup from one successor.
Two-array update
Read from Vk, write into Vk+1. The mathematically clean version: the sweep order cannot matter.
In-place update
One array. A state updated early in the sweep is read by states updated later, so fresh information travels within a single sweep. Usually faster, and half the memory.
The in-place variant is the first crack in the idea that DP must proceed in lockstep. Act 4 widens it into a hole.
The gridworld — the source deck's worked example4×4, undiscounted, reward −1 on every move
The MDP
S={1,…,14} plus two shaded terminal corners;
A={↑,↓,←,→}, deterministic;
a move off the grid leaves the state unchanged — so T(7,→,7)=1;
R=−1 on every transition, and γ=1.
The question
Under the equiprobable random policy — all four actions equally likely — what is Vπ?
Reward −1 everywhere means −Vπ(s) is simply the expected number of steps a random walker takes to fall into a corner.
γ=1 is admissible here only because the random policy being evaluated reaches a terminal corner with finite expected hitting time; with a policy that never terminates, undiscounted evaluation simply runs to −∞. Both corners are terminal, so value propagates inward from two sources at once. Watch the next slide's k=1 frame: after one sweep every non-terminal state reads exactly −1, because a step costs 1 and everything it can reach is still worth 0. The information has not moved yet; after that it moves one ring per sweep.
The deck's own figure, stepped. Expectation backup reproduces its printed table — −1.0, then −1.75 (which the printed figure truncates to −1.7), then −2.4, and at last −14, −20, −22. Switch to the max backup and the same sweep becomes value iteration: it settles in three sweeps at V∗= minus the distance to the nearest corner. The arrows are the greedy policy with respect to the numbers beside them, recomputed every sweep.
The source deck draws a blue cross through the last two rows of its own figure. It is the most useful thing on the slide.
What the numbers do
At k=3 the values read −2.4,−2.9,−3.0. They are heading for −14,−20,−22 and will take dozens more sweeps to get there.
What the policy does
At k=3 the greedy policy is already optimal, at every state, and never changes again. At k=2 it is not.
Evaluating to convergence was wasted work. The argmax only needs the values ordered correctly, not converged.
This is the whole licence for truncating the inner loop, and everything in the rest of this act follows from it. It is also a warning: "the policy stopped changing" is a weaker certificate than "the value converged", and only the second one bounds how suboptimal you might still be.
That inequality is cheap — it says one deviation, then back to π, is no worse. The theorem is the leap from there to always:
Policy improvement theorem
If π′(s)=argmaxaQπ(s,a) then Vπ′(s)≥Vπ(s) for every s, with equality at all states only when π is already greedy with respect to its own value — which is exactly the Bellman optimality condition.
The proof (Appendix, Backup 3) is a telescope: substitute the inequality, expand one step, substitute again. Each pass converts one more step from π to π′ and the inequality never turns round. Two consequences. Improvement can never hurt; and since a finite MDP has finitely many deterministic policies and each nonterminal round strictly improves at least one state (keep the old action when it ties for best), policy iteration terminates exactly.
Alternate the two moves until the policy stops changing. Each round gives an exact value for an exact policy, and finite termination follows with consistent tie handling; the number of rounds need not be small.
The bottleneck
Every arrow marked PE hides a whole iterative solve — many sweeps over the entire state set. The source deck flags it bluntly: policy evaluation is slow to converge.
The mitigation
Each evaluation warm-starts from the previous policy's value, and consecutive policies are similar, so later solves are far cheaper than the first.
And the previous slide has already told us the deeper answer: since the greedy step only needs the values ranked, running the inner loop to convergence is buying precision we throw away.
Policy iteration evaluates a fixed policy before improving it. Value iteration directly applies the optimality backup:
Vk+1(s)←amaxs′∑T(s,a,s′)[R(s,a,s′)+γVk(s′)].
Use Vk to score every action with a one-step lookahead.
Store the largest score as Vk+1(s).
Repeat; then read off a greedy policy from the final values.
This is the Bellman optimality equation used as an update rule. The two-state example just performed it twice.
Lecture 8 replaces the model-weighted expectation with sampled transitions and incremental updates. A neural network is a later extension; tabular Q-learning comes first.
One MDP, one stopping rule, both algorithms — and a dial for the number of evaluation sweeps per improvement. At m=∞ the schedule is policy iteration; drive m down and it slides continuously toward value iteration. Policy iteration reaches π∗ in 9 improvements but spends 16,124 backups getting there; value iteration needs 17 sweeps and only 1,972. Same V∗, same π∗, an eightfold difference in work — and the cheapest schedule, at m=2, is neither endpoint.
Q4. Two processes that keep undoing each other, and an answer they nevertheless agree on.
ACT 4Why it worksLecture 7
Generalised policy iteration — the dance behind both
Q1 Why not greedy?Q2 What must value satisfy?Q3 How do we solve it?Q4 Why does it converge?
evaluationmake V consistent with π
improvementmake π greedy with respect to V
stabilisedV∗,π∗ — the solution of the Bellman optimality equation
Each move breaks the other. Making π greedy makes V wrong for the new π; re-evaluating makes π no longer greedy. The source deck draws this as two lines in the plane of (V,π) — the locus V=Vπ and the locus π=greedy(V) — with the algorithm zigzagging between them in ever shorter hops.
The optimal value function is unique. Several optimal policies can share it when actions tie. Policy iteration and value iteration are different algorithms organised by the same evaluation–improvement idea.
Value iteration is nothing but Vk+1=TVk. And T is a γ-contraction in the sup-norm:
∥TV−TV′∥∞≤γ∥V−V′∥∞
Two value functions disagreeing by δ are pushed to within γδ. By Banach's fixed-point theorem, γ<1 buys three things at once:
a unique fixed point V∗ — one answer, not a family;
convergence from any V0, with no condition on the initialisation;
a rate: ∥Vk−V∗∥∞≤γk∥V0−V∗∥∞.
γ is not a modelling afterthought bolted on to keep a sum finite. It is the number that makes the whole machine well-posed, and it is also its convergence rate.
For a finite MDP with bounded rewards and 0≤γ<1, value iteration converges from any finite V0. What guarantees that?
The Banach fixed-point theorem does the whole job. Each application of the operator brings any two value functions closer by a factor no greater than γ, which forces both a unique fixed point and convergence to it from anywhere. The error after k sweeps decays like γk — which is also why a γ near 1 is not free, and why Lecture 8 has to worry when the operator is no longer a contraction.
∥Vk−V∗∥∞ against sweep number, on a log axis. In this demonstration, the measured decay is close to a straight line of slope log10γ. The theorem guarantees only that the curve remains below the dashed a priori bound γk∥V0−V∗∥. Move the dial and watch it tilt. Reaching an error of 10−3 takes 10 sweeps at γ=0.5 and more than 60 at γ=0.9: far-sightedness is not free, and γ is the invoice.
Every sweep so far updated all states in lockstep. Nothing required that.
In-place: overwrite V(s) immediately, and reuse it within the same sweep.
Asynchronous: back states up in any order whatsoever, using the latest available values (or delayed values whose information does not remain permanently stale) — repeating some, skipping others. Provided every state is updated infinitely often, V→V∗ still holds.
The motivation is brute size. Backgammon has about 1020 states and Go has 319×19; a single full sweep over either will not finish, and even one complete sweep is prohibitively expensive. Partial backups can still provide useful information.
The source deck's own arrow forward, exclamation marks included
"Make it easier to intermix computation with real-time interaction: to solve a given MDP, we can run the iterative DP algorithm at the same time that an agent is actually experiencing the MDP (Reinforcement Learning !!!!)"
Sampled RL is asynchronous DP driven by whichever states the trajectory happens to visit. That is not an analogy — the "visit every state infinitely often" condition in Lecture 8's convergence theorem builds on this slide's condition, and also needs suitable learning rates and coverage of every state–action pair.
The machine is complete and provably correct. Every part of it borrows something.
what value iteration does
what it needs
what happens without it
∑s′T(s,a,s′)[⋯]
the transition model T
the expectation cannot be evaluated
argmaxa∑s′T[R+γV(s′)]
T again, for the lookahead
V alone cannot name an action
one entry V(s) per state
a table the size of S
no storage, and no generalisation to unseen states
Three debts, and the next lecture defaults on all three. Notice they are not independent: the first two are the same debt — the model — appearing once in evaluation and once in improvement. The third is why the last act of Lecture 8 exists at all.
Value iteration and policy iteration both solve the MDP exactly. What do they need that the next lecture will not have?
Every backup in this lecture evaluates ∑s′P(s′∣s,a)[…], which you can only write down if you own P and R. Lecture 8 assumes the full model is unavailable and uses observed transitions, so the question becomes: how do you take an expectation you cannot compute? Sampling the backup is the starting point for the value-based methods in Lecture 8.
The equation is proved, the solvers are exact, and everything rested on owning the world.
CLOSINGLecture 7
Where we are — one parent established
Lineage A · OR / Dynamic Programming
Lineage B · Control Theory
Model-based (origin)
MDP & Dynamic ProgrammingLecture 7
Optimal Control / PlanningLecture 9
Data-driven (extension)
Value-Based RLLecture 8
Policy-Based RLLecture 10
We hold the Bellman equation, two exact solvers, and a convergence guarantee — and every line of it leaned on knowing T and R. Two roads lead out, and the course takes both.
Down — Lecture 8
Keep the equation, delete the model. Replace the expectation with a sample, V with Q, the table with a network. Value-based reinforcement learning: this lecture with (T,R) removed.
Right — Lecture 9
The other parent. Control theory solves the same dynamic problem through continuous-control examples with dynamics f, instead of today’s finite tables — and Lecture 10 will delete f exactly as Lecture 8 deletes (T,R).
A sequence of decisions seemed to need a plan for every possible future.
Bellman showed it needs only a value for every state — because an optimal plan, by its nature, stays optimal from wherever you happen to land.
Questions?
Value, defined recursively, turns an intractable search over plans into local backups over states. It is the founding move of operations-research decision making, the equation Lecture 8 keeps after throwing the model away, and — read one column to the right — the problem Lecture 9 will solve all over again in a different language.
APPENDIX
Backup slides
The derivations, kept out of the narrative.
APPENDIXBackup slidesLecture 7
Reading guide — understand the backup before the algorithmone main idea to explain, one comparison, one application
How does one expected return become a Bellman backup?
Compare
Policy iteration versus value iteration on the same MDP
Which operation evaluates a policy, and which improves it?
Apply
The original gridworld, with the same transition and reward conventions
How many backups are needed before the preferred action changes?
Calculate one backup and two sweeps before reading the convergence proof. The figures and widgets use the same mathematical objects; a newer research paper is not needed to replace this foundation.
The third line is where the Markov property is spent: the inner expectation depends on the whole history only through s′, so it isVπ(s′).
Optimality form. Maximise over the first action and use the principle of optimality for the remainder: π(s) becomes maxa and Vπ becomes V∗. Q form. Condition on (s,a) before the transition instead of after it; the max then lands at the successor, maxa′Q∗(s′,a′), which is why V∗ maxes before the expectation and Q∗ after.
since ∑s′T(s,a,s′)=1. Taking the maximum over s gives the claim. ■
Consequence (Banach). For γ<1, T has a unique fixed point V∗, and Vk+1=TVk converges to it from any V0 with ∥Vk−V∗∥∞≤γk∥V0−V∗∥∞. The max inequality is the only step that needs care. Finiteness ensures maxima are attained; analogous suprema also work under suitable continuous-space assumptions — Lecture 9 works in a setting where that maximum is over a continuum. Q-learning combines this contraction with stochastic-approximation, coverage and step-size conditions (Lecture 8, Backup 2).
Each pass converts one more step of the trajectory from π to π′, and the inequality never turns round. Why policy iteration terminates. A finite MDP has finitely many deterministic policies, and each improvement is strict unless π is already greedy with respect to its own value — which is precisely the Bellman optimality condition. So π∗ is reached in finitely many rounds, not merely in the limit.
In-place value iteration. Keep one array; overwrite V(s) immediately and let later states in the same sweep read the new number. It converges, and usually faster, because fresh information propagates within a sweep instead of waiting for the next one. The sweep order now matters for speed — never for the answer.
Asynchronous DP. Back states up in any order, repeating some and skipping others. As long as every state is updated infinitely often, V→V∗. This decouples the computation from any rigid sweep and is what lets DP be prioritised — order the backups by how large their Bellman residual was last time (prioritised sweeping), and effort follows the states where the value is still wrong.
Why it matters downstream. Sampled RL is asynchronous DP whose update order is chosen for it by the environment: Q-learning backs up whatever (s,a) the trajectory delivers, in whatever order experience delivers them. The "every state infinitely often" condition here is the model-based ancestor of the exploration requirement in Lecture 8 — and the reason ε must not be allowed to reach zero too soon.
The empty cells are outside this course’s scope. Continuous-time reinforcement learning also exists.
These columns describe the examples chosen for this course: finite-action MDPs in Lecture 7 and continuous-control systems in Lecture 9. MDPs can also have continuous spaces; policy methods can also handle discrete actions. The common question is how to choose actions over time.
Lecture 9 is the mirror of this lecture, not its sequel. Lecture 8 sits directly below it; Lecture 10 directly below Lecture 9.