| Previous step | This chapter's question |
|---|---|
| Lectures 8–11 could collect new transitions during learning. | Now the training log is fixed. |
Lecture 5 already showed how optimizing a learned score can exploit error. Here unsupported action values also enter repeated Bellman targets.
Bring: TD3, Bellman fitting and a basic probability ratio.
| First pass | What to do |
|---|---|
| Follow the idea | Diagnose missing support → TD3+BC → CQL/IQL → evaluate |
| Work without the solution | Estimate a policy's return from a log and identify an unsupported action. |
| Return later | Full CQL guarantees, model/sequence alternatives and advanced OPE estimators are references. |
We are handed D={(si,ai,ri,si′)}i=1N, collected by some behaviour policy β that may be a human operator, an old controller, or a mixture of several. No more data will arrive. Run Lecture 8's update on it, unchanged:
The max ranges over every action, including actions β never took at s′. There is no datum anywhere in D that contradicts whatever Q happens to say there; a function approximator will extrapolate cheerfully; and the max is not a passive reader of that extrapolation — it actively seeks out whichever one is highest. The result is written into the next target, and bootstrapped again.
The transition is real: s, a, r, s′ were all measured. The highlighted term was not — it is the model's opinion about an action at a state, and for most (s′,a′) that opinion rests on no evidence whatsoever.
Call the gap ϵ(s′,a′)=Q(s′,a′)−Q∗(s′,a′). Three properties make it lethal together:
(Fujimoto, Meger & Precup, 2019) name the first of these extrapolation error. Their experiment: a DDPG agent trained on the replay buffer of another, concurrently running DDPG agent — the same task, the same algorithm, near-identical data — fails, while the agent that generated the buffer succeeds. The only difference is who chose the actions.
Off-policy learning has always been hard. What follows is not that difficulty; it is the removal of the thing that made it survivable.
Q over-rates (s,a) ⇒ the policy tries a at s ⇒ the environment returns a disappointing r,s′ ⇒ the target drops ⇒ Q falls.
New visits can provide corrective evidence; online learning still needs exploration and a stable update method.
Q over-rates (s,a) ⇒ the policy would try a at s ⇒ but it may not ⇒ no transition from (s,a) is ever added ⇒ the target never drops.
The log supplies no direct corrective observation at that pair. Without other structure or regularization, the error may persist and grow.
This is Lecture 4 against Lecture 5 again, in the dynamic world. There, uncertainty was an opportunity while the oracle remained and a hazard once it was gone. Here, exploration was an opportunity while the environment remained, and the same width in Q is now a hazard.
Lecture 8's Act 4 named three ingredients — function approximation, bootstrapping, off-policy data — whose combination can cause divergence. Standard bootstrapped offline value learning has all three: the data is not merely off-policy, it comes from a policy we did not choose and cannot re-run. And what online RL had, the ability to go and visit the state–action pair it is wrong about, is precisely what has been withdrawn.
The log contains a transition with reward 1 to state B. At B, action stay was observed and has estimated value 2; action jump was never observed but the network predicts 20. Let γ=0.9.
| backup | continuation used | target |
|---|---|---|
| unrestricted maximum | 20, from the unsupported action | 1+0.9(20)=19 |
| maximum over observed actions | 2 | 1+0.9(2)=2.8 |
The optimiser exploits fθ at out-of-distribution inputs.
The policy exploits Q at out-of-distribution actions.
Lecture 5 promised this slide would return with a policy in place of an optimiser and a Q-function in place of a surrogate. It has. The analogy replaces a design x by a state–action pair (s,a). CQL also includes a Bellman target and a specific action-distribution penalty, so it is not simply COMs with renamed variables. Act 3 develops the loss.
Online interaction can supply corrective data for visited actions. Offline, that feedback loop is cut. Maximization can select optimistic unsupported values and feed them back as targets, causing estimates to inflate. It is Lecture 5's adversarial optimiser, wearing a policy.
The trivial offline algorithm is behaviour cloning — supervised learning of π^(a∣s) from D, no Bellman equation anywhere:
It directly fits demonstrated actions. Exact replication has the same expected performance as the behavior policy, but approximation, generalization, and action selection can make it better or worse. It has no reward-based mechanism for selecting better actions and no automatic safety guarantee.
Reward-guided stitching. A Markov state can connect the good first half of one run to the good second half of another. Bellman backups use rewards to identify those continuations. A state-based clone can also produce unseen combinations, but does not explicitly select them by return.
So the offline problem is not "imitate the data" but seek better expected return using supported actions and transitions. Hold those two clauses together: support reduces extrapolation risk; it does not by itself establish safety or improvement.
Take TD3 — Lecture 10's overestimation-hardened DDPG — exactly as it stands. Add one term to the actor loss:
A behaviour-cloning term, and a normaliser λ that reduces sensitivity to the scale of Q; it does not eliminate task dependence. No generative model, no divergence estimate, no extra network.
Fujimoto & Gu report competitive performance against more elaborate offline methods on their D4RL evaluation, with substantially lower training overhead. This supports using a simple baseline before adding machinery; it is not a claim of dominance on every offline dataset. A Minimalist Approach to Offline Reinforcement Learning (NeurIPS 2021)
A tight distribution constraint can limit improvement. A support constraint can still select better actions than the behavior policy by changing their probabilities. The right restriction depends on coverage and estimation error.
| method | what it constrains | the mechanism |
|---|---|---|
| BCQ (Fujimoto, Meger & Precup, 2019) | the support of π | fit a generative model Gω(s)≈β, sample n candidate actions, allow a small learned perturbation ξϕ, and take the max only over those |
| BEAR (Kumar et al., 2019) | the support, softly | an MMD penalty between π(⋅∣s) and β^(⋅∣s) — encourages similarity using finite-sample MMD; MMD is not literally a support-only distance |
| BRAC (Wu, Tucker & Nachum, 2019) | the distribution | an explicit divergence, usually KL, penalised in the actor loss or subtracted from the reward |
The distinction in column two is the one that matters. A support constraint says only propose actions β might plausibly have taken; a distribution constraint says propose them about as often as β did. A support restriction allows reweighting of observed choices; a tight distribution penalty limits it — at M, both continuations are in β's support, so BCQ may take the better one, while a tight KL to β drags the choice back toward the mixture.
Written as a modified backup, BCQ is one edit to the target: y=r+γmaxa′∈AG(s′)Q(s′,a′) with AG(s′) the sampled candidate set. That is the escape hatch from Act 1's widget, made into an algorithm.
Cloning maximizes action likelihood rather than return. Because the Bellman equation propagates value across trajectories through shared states, offline RL can discover that the first half of one poor trajectory joins the second half of another to make a good one — a route nobody in the dataset ever drove. That is the whole reason to accept the difficulty.
CQL still evaluates Q at out-of-distribution actions in order to push them down. IQL declines to evaluate them at all. Fit a state value V by expectile regression toward Q on in-data actions, then bootstrap through V:
with the asymmetric loss L2τ(u)=τ−1{u<0}u2.
τ=21 is ordinary least squares and V becomes Ea∼β[Q] — policy evaluation of β. Raise τ and under-predictions are penalised more heavily, so V is pulled toward the upper expectile of Q over the actions β took. As τ→1 it approaches maxa∈suppβQ(s,a).
The policy is then extracted separately by advantage-weighted regression, π=argmaxπED[exp(ϰ(Q(s,a)−V(s)))logπ(a∣s)] — a weighted behaviour clone, which is why IQL avoids one important source of training extrapolation. The price is that the in-support maximum is estimated from however many actions β happened to try at s; where β was nearly deterministic, there is nothing for the expectile to climb.
At one state, suppose the log contains the two actions equally often, with Q values 2 and 6. For an expectile m between them:
| expectile level | fitted value | interpretation |
|---|---|---|
| τ=0.5 | 4 | ordinary mean |
| τ=0.8 | 5.2 | pulled toward the higher observed value |
| τ→1 | approaches 6 | approaches the observed maximum |
| too little | too much | |
|---|---|---|
| COMs, α | the optimiser escapes the data and returns a hallucinated design | the surface flattens; ascent cannot move at all |
| CQL, α | the max escapes the support and Q diverges | Q^ tracks logβ^; the argmax becomes β's modal action — behaviour cloning, arrived at by accident |
| IQL, τ | V→Eβ[Q]; mean-value fitting; actor weighting can still favor better actions | the expectile chases the largest sampled Q and re-imports the over-estimation it was designed to avoid |
CQL's Lagrangian variant turns the penalty into a constraint with a budget. TD3+BC instead normalizes the Q term to reduce sensitivity to its numerical scale. Both address balancing terms, but normalization is not a constrained optimization derivation; cross-task transfer still needs checking.
All three start with a fixed log and face the same risk: the learned policy may prefer actions whose value is poorly supported by that log.
| Main method | What is changed? | Question to ask when reading its loss |
|---|---|---|
| TD3+BC | The actor objective | How strongly does copying logged actions oppose maximizing the critic? |
| CQL | The critic objective | Which candidate actions have their values pushed down relative to the data? |
| IQL | Value fitting and policy extraction | How do expectile fitting and weighted cloning use logged actions without maximizing over new actions during value training? |
Both chapters study optimization with imperfect models and limited data. COMs regularizes a design surrogate; CQL regularizes action values inside Bellman learning. Their conservative objectives are related, but their guarantees and assumptions differ. Excessive conservatism can suppress useful improvements.
You have a candidate policy π and a fixed log D. If new evaluation trials are unavailable, estimate Vπ using the log and explicit coverage assumptions. Offline RL restricts interaction during training; it does not by definition prohibit every later deployment.
Unbiased with correct behavior probabilities, matching environment dynamics, and target-policy support covered by the data policy. The weight is a product of ratios.
A reward at step t cannot depend on later actions, so it should not be reweighted by them. This avoids unnecessary future-action ratios; it often reduces variance, without a universal strict ranking.
In the independent, identical-step example below, let Eβ[ρ2]=q. Then E[W2]=qH and Var(W)=qH−1, so the estimator's standard error grows like qH/2 — geometric in the horizon, with a base fixed by how far π has moved from β. For a fixed dataset, this can make the estimate too uncertain to compare policies reliably.
One decision, two actions. The behavior policy uses probabilities (0.5, 0.5); the target policy uses (0.8, 0.2). Rewards are (1, 0).
| logged action | target / behavior weight | weighted reward |
|---|---|---|
| A | 0.8/0.5=1.6 | 1.6 |
| B | 0.2/0.5=0.4 | 0 |
With one observation of each, IS gives (1.6+0)/2=0.8, matching the target's true value. This exact agreement is specific to this balanced sample.
Use only this one-step log; the learner has no simulator formula. Behavior chooses off/heat with probabilities (0.5, 0.5). The log contains one off reward −4 and one heat reward −2. The proposed policy uses probabilities (0.2, 0.8).
Predict: Should giving the lower-cost action more probability improve the estimated return?
Calculate and check. Weights are 0.2/0.5=0.4 and 0.8/0.5=1.6. The IS estimate is [0.4(−4)+1.6(−2)]/2=−2.4, compared with logged mean −3. This exact calculation uses the balanced sample shown.
Suppose behavior always chose off, but the proposed policy still heats with probability 0.8. Can the existing log identify its value without extra assumptions about heating outcomes?
Check your answer. No. The heating probability ratio has zero behavior probability in its denominator, and no heating rewards were observed. Many different heating outcomes are compatible with the same all-off log.
| Check | What to report |
|---|---|
| Coverage | Does the log include actions the proposed policy could choose? |
| Estimated return | State the estimator, behavior probabilities and assumptions. |
| Stability | Inspect large importance weights and sensitivity to the evaluation data. |
| Comparison | Compare with the logged policy and a simple baseline under the same protocol. |
Our main optimization and RL methods use one decision maker's objective. With several decision makers, outcomes also depend on their interaction: objectives may align, conflict, or partly overlap. Equilibrium concepts become relevant, but adding an agent does not automatically turn every optimization problem into a Nash-equilibrium problem. IE579 develops these strategic models.
| Question | Core tools | What still needs checking |
|---|---|---|
| Which actions should the policy prefer? | TD3+BC and behavior restrictions | The trade-off between staying near the data and improving on it. |
| How should values guide improvement? | CQL and IQL | Their different objectives and the effect of conservatism. |
| Does the new policy perform better? | Off-policy evaluation | Coverage, behavior probabilities and estimation uncertainty. |
Model-based alternatives and sequence models remain in the appendix. Each adds its own modeling assumptions; none creates missing observations.
| Role | Read or revisit | Question to answer |
|---|---|---|
| Core | Kumar et al., Conservative Q-Learning for Offline Reinforcement Learning (NeurIPS 2020) | How can the critic discourage actions unsupported by the data? |
| Compare | Fujimoto & Gu, A Minimalist Approach to Offline Reinforcement Learning (NeurIPS 2021) and Kostrikov et al., IQL (ICLR 2022) | What is constrained: the actor update, the value, or the action queries? |
| Apply | The logged-route stitching example and off-policy evaluation calculations | Can measured evidence support the claimed improvement? |
| On-policy | Off-policy | Offline | |
|---|---|---|---|
| who collects the data | the policy being learned | some other policy, but we still act | a fixed log; we never act |
| an over-estimate is | corrected next episode | corrected once visited | never corrected |
| exploration | required | required | impossible |
| the design principle | on-policy correction | replay and stability | conservatism |
| representative method | SARSA, PPO | Q-learning, DQN, TD3 | BCQ, CQL, IQL, MOPO |
The middle column is where the confusion usually sits. Q-learning has always been off-policy, and Lecture 8 showed it converging happily while behaving ε-greedily. What made that work was not the algorithm but the clause "with every (s,a) visited infinitely often" in Watkins & Dayan's theorem. A finite log generally cannot supply fresh samples from every true transition law. Replaying every covered pair infinitely often can solve an empirical MDP; it does not remove uncertainty about the real MDP. A fully covered deterministic finite problem, such as the stitching example, can still be solved from a finite log. (Levine, Kumar, Tucker & Fu, 2020)
Choosing μ. Leaving μ free and adding an entropy regulariser R(μ)=H(μ) gives μ∗(a∣s)∝expQ(s,a), and the penalty collapses to the closed form used in practice — CQL(H):
The first term is a soft max over all actions; the second is the empirical behaviour average. For discrete actions, the difference is H(β^)+DKL(β^∥softmax(Q)). Its minimum is H(β^), not generally zero. A dominant penalty encourages softmax scores to match behavior frequencies; greedy extraction then favors the mode.
The Lagrangian form. The maxα with a budget τ is Lecture 5's move again: τ is read in the units of the value function and is interpretable on the value scale, but is not automatically transferable across tasks.
Why the counter-term is not optional. The counter-term protects supported actions from excessive pessimism. With fixed Bellman targets, squared error anchors observed values; a linear pessimism term alone does not imply that every Q value diverges downward.
For a random variable X, the τ-expectile mτ is the minimiser of E[L2τ(X−m)] with L2τ(u)=∣τ−1{u<0}∣u2. Setting the derivative to zero,
so m1/2=E[X], and for bounded X, as τ→1 the expectile approaches esssupX. IQL applies this with X=Q(s,a), a∼β^(⋅∣s) — hence the supremum is over the support of the behaviour policy at s, never over the whole action set.
The two losses, in order.
repeat:
V ← argmin_V E_D[ L₂^τ ( Q(s,a) − V(s) ) ] in-data actions only
Q ← argmin_Q E_D[ ( r + γ V(s′) − Q(s,a) )² ] target uses V(s′), not max_a
π ← argmax_π E_D[ exp( ϰ (Q(s,a) − V(s)) ) log π(a|s) ]
Note what is absent from every line: an action that is not in D. IQL is the only method in this lecture whose Q-network is never evaluated at an out-of-distribution input during training, which is why it needs no explicit conservatism term at all — and why its ceiling is set by how much β varied its actions.
Let ρt=π(at∣st)/β(at∣st) and W=∏tρt. Assume target support is covered and environment dynamics are shared.
| estimator | main calculation | practical limitation |
|---|---|---|
| IS | average WG | unbiased with correct ratios; variance can be very large |
| PDIS | weight each reward by its prefix ratios | avoids unnecessary future ratios; can still have high variance |
| WIS | ∑jWjGj/∑jWj | finite-sample bias; direction varies; few weights can dominate |
| DR | model estimate + weighted TD residuals | correct ratios or exact Q can preserve unbiasedness with appropriate fitting; no universal variance cure |
| FQE | regress on r+γEa′∼πQ(s′,a′) | depends on coverage, horizon, function class, and fitting |
WIS stays within the observed return range when its denominator is positive; that does not imply closeness to the true policy value. Detailed IS, PDIS, and DR formulas are in Act 4.
The theoretical result concerns a policy value, averaged over that policy's actions:
It requires the theorem's coverage, estimation-error, update, and sufficiently large penalty assumptions. The practical neural-network loss does not automatically certify those assumptions.
Under the stated assumptions, the estimated value of the evaluated policy is conservative.
Every action value is a lower bound; any extracted policy is safe; or every trained network has the guarantee.
Lecture 11 handed us a learned dynamics model P^. Offline, its bias becomes acute for the same reason everything else does: a rollout that leaves the data is never contradicted. So penalise the reward by the model's own uncertainty and plan in the penalised MDP:
u(s,a) is the maximum standard deviation across a bootstrapped ensemble of dynamics models — Lecture 5's ensemble-alarm, now measuring disagreement about where you will end up rather than about how good it is. A theoretical lower bound requires a valid model-error bound; empirical ensemble uncertainty is a proxy, not an automatic certificate.
Harder-edged: an unknown state–action detector partitions the space, and every pair it flags is routed to an absorbing state with the worst possible reward. Planning then avoids the unknown region because the model says it is a cliff.
Every method so far has kept Bellman and defended it. The last family does not keep it.
Model the trajectory as a sequence: R^1,s1,a1,R^2,s2,a2,…, where R^t=∑t′≥trt′ is the return-to-go. Train a causal transformer to predict at. At test time, condition on the return you want and let it autoregress.
It avoids Bellman bootstrapping and its particular feedback loop; training and generalization can still fail. The failure mode moves instead: ask for a return the data never achieved and it will confabulate.
Go further: learn a diffusion model over whole trajectories and generate one, guided by a reward gradient. Planning becomes sampling; the learned distribution guides the proposal, but feasibility and dynamics consistency are not guaranteed for free.
This is Lecture 6 arriving in the dynamic world. There, search a forward model (Lec 5) sat opposite sample an inverse model (Lec 6), and the spine promised the pair would return as value against policy. Here it returns a third time, in its sharpest form: learn Q and search it, or learn p(τ∣return) and draw from it. Conditioning on a desired outcome and sampling a design is exactly what Lecture 6's condition-shift did — including the way the samples thin out as the condition leaves the data.
With support and appropriate independent fitting, DR is unbiased if the ratios are correct or the relevant Q model is exact. Weights multiply Bellman residuals; an accurate model can reduce their variance.
Regress Qπ directly: Q←r+γEa′∼π[Q(s′,a′)], fitted on D. FQE avoids products of trajectory ratios, but its error still depends on horizon, coverage, function approximation, and fitting. More data can reduce estimation error; misspecification or missing support can leave irreducible error.
Compare estimators and inspect coverage and importance weights. Their disagreement is a diagnostic, not a confidence interval. Any reported uncertainty interval needs an explicit construction and its assumptions; agreement alone cannot establish accuracy.
In an industrial deployment this act comes first. Before anyone asks whether to use CQL or IQL, someone asks: if we hand you six months of logs and a proposed controller, can you tell us whether it is better than the one we are running? The course has not addressed that question until now, and it is the question that gates the rest.