Data-Driven Design Optimization — Surrogate-Based
Offline, part 1 of 2 — learn the objective from a fixed dataset, then decide what to trust
Where we are — the query is taken away
| Lecture 4 · Observe | Lecture 5 · Trust | Lecture 6 · Generate | |
|---|---|---|---|
| the question | which experiment to run next | what to believe about designs no one has measured | how to produce valid, good designs |
| access to f | one evaluation per round | none — a fixed dataset | none — a fixed dataset |
| uncertainty is | a reason to look | a reason to be careful | a reason to stay on the data's manifold |
Learning route — two sessions, one question
Bring: regression and gradient ascent (Lecture 1), the GP posterior and UCB (Lecture 4).
| Session | What it does |
|---|---|
| 1 · Why it fails | Delete the evaluation → watch the optimiser exploit the surrogate → split the decision loss into two errors → conservatism, with COMs worked to the end |
| 2 · Other grounds for trust | Uncertainty: estimate what the surrogate does not know and decide with it → system knowledge: give it structure or physics → compare the three |
Lecture 4's loop, with one line deleted
Every black-box method so far closes its loop through the real objective — Bayesian optimisation, and equally a genetic algorithm, CMA-ES, a policy gradient, or gradient ascent on a fitted proxy. Lecture 4's loop:
Online — Bayesian optimisation (Lecture 4)
for t = 1 … T:
fit the GP on D_t
x_{t+1} = argmax_x a_t(x) ← acquisition, uses σ
y_{t+1} = f(x_{t+1}) ← query
D_{t+1} = D_t ∪ {(x_{t+1}, y_{t+1})}
Offline (this lecture)
fit a model f_θ on D ← once
x̂ = argmax_x f_θ(x) ← no query
return x̂
One line deleted, and with it every correction. Online, an over-rated point is queried, found wanting, and the model repairs itself — which is why BO could afford to seek uncertainty. Offline, the model is fitted once and never contradicted again.
Source alignment: Part 1 pp. 3–17 (the online methods — GA, CMA-ES, BO, policy gradient; tabulated in the appendix). The source's p. 5 box, gradient ascent on a proxy with a query per step, is the naive offline method of slide 8 with the query put back.
What breaks offline
Offline model-based optimisation
Evaluating f is expensive or prohibitive — a molecule to synthesise, a protein to express, a chip or an aircraft to build. The source figure is one deleted arrow: data flows into an offline optimiser, a design flows out to superconductors, DNA, proteins and robot bodies, and the return arrow from the design back to the data is struck through — no additional interactions.
1 · Surrogate-based — this lecture
Approximate f from D, then optimise over the surrogate. A forward model, and a search.
2 · Generative-based — Lecture 6
Learn a representation or distribution of plausible designs, then search in it, condition it, or guide it toward better ones.
Source alignment: Part 1 pp. 18–19; Part 2 pp. 3–4.
The bar to beat is the best design you already ownthree ways to answer with the same data
| rule | the design it returns | what it risks |
|---|---|---|
| Best observed | argmaxxi∈Dyi | nothing — and it can never improve |
| Naive surrogate | argmaxxf^θ(x) | believing f^θ where no data checked it |
| Reliable offline optimisation | the maximiser of a score that accounts for out-of-distribution risk | this chapter |
The obvious method — fit a proxy, then climb it
Fit a neural network fθ to D — this is the surrogate, or proxy — and then, because fθ is differentiable where the real f was not, run gradient ascent on it, starting from the best design in D.
Source alignment: Part 1 p. 20.
What the benchmark asks for, and how it is scoredthe six Design-Bench tasks as used by NEMO, COMs and RoMA (2021)
| task | dimension | type | dataset size |
|---|---|---|---|
| Superconductor | 81 | continuous | 21,263 |
| GFP (green fluorescent protein) | 238 | categorical (20) | 5,000 |
| MoleculeActivity | 1,024 | binary | 4,216 |
| HopperController | 5,126 | continuous | 3,200 |
| AntMorphology | 60 | continuous | 25,009 |
| DKittyMorphology | 56 | continuous | 25,009 |
Three things to keep apart. The offline data D is all a method may use. The candidates — 128 designs in the papers of this lecture — are what it returns. The oracle scores them afterwards, for the benchmark only, and is never fed back: the physics simulator for the robot tasks, a model fitted to a larger held-out dataset for tasks such as Superconductor. The usual score, the best of the 128, is a benchmark number; deploying one design would still need a way to pick it without the oracle. HopperController has 3,200 designs in 5,126 dimensions: almost every direction the ascent can take was never checked against anything.
Source alignment: Part 2 p. 23 (task table). These are the 2021 versions of the tasks, the ones the COMs, NEMO and RoMA results use. The final benchmark (Trabucco, Geng, Kumar & Levine, ICML 2022, Table 1) revised them: Superconductor is 86-dimensional there, and GFP and MoleculeActivity were moved to its appendix as tasks not kept in the final suite.
Source alignment: Part 2 pp. 25 and 38 (128 candidates, 50th and 100th percentile).
Two different problems
Problem 1 — values off the data · every offline problem
The training data constrains the model where it has support. Away from it, a small training loss controls nothing — and the search must go there, because the goal is a design better than anything in D. The failure is the task.
Problem 2 — the valid set · only with a constraint
When designs must be real molecules, foldable proteins or buildable layouts, only a thin sliver of input space is valid, and ascent can leave it: a design off it is not merely poor but meaningless. If every input is a legitimate design, this problem does not exist.
| valid | invalid | |
|---|---|---|
| near the data | the data themselves | a molecule string with one character changed — when the valid set is thin |
| far from the data | the design we hope to find | where unchecked ascent ends |
Independent questions. A known constraint is Lecture 1's constrained optimisation. An implicit one — known only through valid examples — must be learned, and judging it off the data is Problem 1 again, for the constraint. This lecture attacks Problem 1; Lecture 6 learns the valid set.
Source alignment: Part 1 pp. 21, 23, 25.
The two pictures
Source alignment: Part 1 p. 22 (the fan of extrapolations) and p. 24 (the valid manifold, redrawn as a thin curve). "Near the data" here means close in the input space — the sense a GP's σ or a trust region uses. Measured instead by the support of the data distribution, a point inside it is valid whenever the data are.
Watch it happen
The optimiser is an adversary
Gradient ascent on f^θ is the same computation aimed at a regression model: it moves the input in whatever direction raises the model's output fastest. In a high-dimensional space with little data, many of those directions lead to errors.
Source alignment: Part 2 p. 29.
A good fit can still make a bad decisionthe signed decision bound
Let x^ be the design the optimiser returns and x∗ the true optimum. Add and subtract f^ at both points:
A — at the true optimum
How much the surrogate under-rates the best design. A model too pessimistic about the right answer pays here.
B — at the chosen design
How much the surrogate over-rates the design it picked. The optimiser's exploit pays here.
The middle term, the optimiser gap, is never positive when x^ maximises f^ — that is all the inequality uses. The exact loss keeps it, so A+B can sit well above the loss. A and B themselves can be negative.
Where the loss comes from, on the same fit
Temperature thread — what the surrogate promised, and what it deliveredshared teaching example · predict before revealing
Predict: Which command does maximising the surrogate pick — and is it the better one?
It picks u=2, the command it rates higher. Press → to run the heater and see why that is the worse choice.
Numbers behind the picture: hidden score f(u)=−[(u−2)2+u2]; predictions f^(1)=−2.1, f^(2)=−1.0; truth f(1)=−2, f(2)=−4. So A=0.1, gap =−1.1, B=3, and the loss 2=A+gap+B.
Try it — the largest error is no longer the selected actionwork independently · reveal only after deciding
Which command is chosen now — and does the remaining error at u=2 cost anything?
Check your answer. It picks u=1, the right command. The optimiser maximises truth plus error, not error alone: the error at u=2 is still there, but an error costs the decision only when it changes the choice. Press → to see it.
Numbers behind the picture: errors 0 at u=1 and +1 at u=2, so predictions −2 and −3; the pick is u=1, and A=B=0.
The cure is not a better optimiser
Two repairs suggest themselves first, and both treat the search as the problem.
Search harder
A stronger, more thorough optimiser finds a higher point of f^ — on this surface, a point still further from the data and still more over-rated. It makes B larger.
Optimisation strength is on the wrong side of the problem.
Search less
Restricting the search to measured designs caps the result at the best recorded score. A neighbourhood limit can still allow improvement, but it also excludes the novel designs we came for.
The trade is between evidence and novelty.
Check — prediction quality versus decision quality
Validation error averages over designs like those in D. The decision loss is bounded by A+B: the errors at the true optimum and at the chosen design, which the optimiser picks because the prediction is high there. A model can win on the first number and lose on the second — the surrogate in Watch it happen had an RMSE of 0.058 and B=4.44.
Three ways to give a surrogate grounds for trustthe map for both sessions
Conservatism
Change what f^ says where the optimiser goes: a training penalty on the designs it would chase.
COMs (ICML 2021) · PRIME · RoMA — Session 1
Uncertainty
Estimate what f^ does not know, and decide with it: μ−κσ, Lecture 4's UCB with the sign turned.
Deep Ensembles (NeurIPS 2017) · NEMO — Session 2
System knowledge
Give the model structure or physics, so that its extrapolation has grounds other than the data.
FGM (AISTATS 2024) · Kennedy & O'Hagan — Session 2
The papers of this lecture — why each is here
Conservatism: train the surrogate against its own optimiser
COMs — what we want, and the obstacleConservative Objective Models · Trabucco, Kumar, Geng & Levine, ICML 2021
We want a model that does not over-rate the designs an optimiser would chase — a small B. The obstacle is knowing which designs those are, and the answer is to generate them by running the attack we fear.
μθ is where T ascent steps from the data end up; the loss uses it only through this average.
Because fθ changes at every training step, μθ is regenerated as training proceeds — an inner attacker chasing an outer defender, as in adversarial training for robustness.
Source alignment: Part 2 pp. 26–29. The paper describes μ as "several steps of gradient ascent" from data points; its implementation uses a single step (Trabucco et al., 2021, §3.4). Written as a distribution, μθ=M1∑jδxT(j) — a point mass 1/M at each end point — which is why its expectation is the plain average above. The toy widgets collect every point the 25-step ascent visits, not only the end points; with the paper's one step the two coincide.
Three terms, three fitsadd a term to the loss and watch the surrogate move · the same fifteen points, α = 0.3
(i) fit the data
(ii) + push the attacker's designs down
(iii) + hold the data up
Source alignment: Part 2 p. 30 ("Problem: underestimation. Solution: regularise by maximising function values under the data distribution"). The three panels are one widget in three instances, each trained for real on the same fifteen points at α = 0.3. Panel (ii): the first attempt's mean error on D is −0.30 — exactly −α: lowering the whole surface by c cuts the penalty by αc and costs the fit ½c², so the balance is c = α; 14 of the 15 points sit above it, by up to 0.81 at the right end of D, where the attacker's designs (the grey strip) crowd. Panel (iii): with the data term the mean error is 0.00 — a constant shift moves E_μ[f] and E_D[f] alike, so the penalty no longer rewards it. The full COMs curve is the first attempt lifted by 0.30 exactly inside D, so the attacker's paths, and any later search, are unchanged: only the level was wrong. If asked why that matters: without the data term the surrogate is biased low even on the designs we measured, and α would set the level as well as the relative push; with it α sets only the gap. The attacker never goes past x ≈ 5.8, and the far edge is still over-rated (1.98 for the first attempt, 2.31 for full COMs, against a true −0.14) — the point of "Conservatism is learned where the attacker went". No search is drawn: Algorithm 2 comes two slides later.
The loss, term by term
- (i) ordinary regression — be right about the designs we measured;
- (ii) prevent over-estimation out of distribution;
- (iii) prevent under-estimation in distribution.
COMs as an algorithm — train against an attacker, then search slowly
Algorithm 1 — train the conservative surrogate
initialise f_θ; pick η, α
repeat:
sample (x₀, y) ~ D
x_K ← K ascent steps from x₀ on f_θ
μ ← empirical distribution of the x_K
L = ½ E_D (f_θ(x₀) − y)²
+ α E_μ[f_θ] − α E_D[f_θ]
θ ← θ − λ ∇_θ L
Algorithm 2 — search it, slowly
x₀ = argmax_{(x,y)∈D} y ← the best design we own
for t = 0 … T−1: ← T = 450 in the paper
x_{t+1} = x_t + η ∇_x L_opt(x_t)
← L_opt: f̂ with a brake, next slide
return x̂ = x_T
Left — learn. Ordinary regression on D plus the penalty of the last two slides. The attacker is re-run on the current fθ every round, so the surrogate is flattened exactly where it was tempting. Out comes one model, f^θ∗.
Right — search. Start from the best design we own and take T small steps uphill — not on f^θ∗ itself but on Lopt, a braked copy of it that distrusts what the surrogate promises one step further on.
Source alignment: Part 2 p. 35 (Algorithms 1 and 2). The toy in the next widgets: the attacker takes K = 25 plain ascent steps from every data point; the search is Algorithm 2 with β = 0.9 and T = 450. With the paper's one-step attacker this 1-D toy is too local to protect anything.
The brake — what Algorithm 2 climbsa trust region, as in Lecture 1: do not trust the model further than one step
To second order, Lopt≈(1−β)f^−βη∥∇f^∥2: a tenth of the climb, minus a penalty on steepness. T=450 steps is a budget, not convergence.
Trabucco et al. (2021), Eq. 4 and §3.4: β = 0.9, T = 450; the paper also reports that using the trust-region procedure to build μ performs similarly. On flat ground the trial step promises about what the surrogate says, so nine tenths of the same number is subtracted and a tenth is left: the search climbs at a tenth of the pace. On a steep slope the promise is larger by about η‖∇f̂‖², and nine tenths of that is subtracted too, so L_opt falls where f̂ is steep — the edge beyond the data, where a plain fit extrapolates, is downhill for the search. In the toy of the next widgets η = 0.34 and the conservative surrogate's slope beyond the data is about 0.07, so the brake acts mostly through the pace: 450 steps travel about 1.1 units.
Turning the dial
Numbers on the dial: B falls from 4.44 at α=0 to below zero by α=0.15; A rises to 1.41 at α=1.3; the loss is 2.79, 0.33 at 0.15, 0.03 or less from 0.3 to 0.9, and 0.70 at 1.3. At α=0 the trust-region search passes the optimum (worth 2.66) near step 50 and reaches the edge by step 450 — the same end point as plain ascent. Why the search stops: the trust-region gradient is about 0.1×f^′, so on the conservative surrogate (slope about 0.07 beyond the data, against 0.41 for the plain fit) each step moves about 0.0024 and 450 steps travel about 1.1. Run longer at α=0.3: x=7.41 after 900 steps (worth 1.90), 9.78 after 1,800, the edge by 4,500. Only α=1.3 stops at a real peak of the surrogate (x=4.43). The attacker's farthest point at α=0.3 is x=5.84, so x^=6.32 is just outside μ.
The exact three-term split f(x∗)−f(x^)=A+gap+B always holds. Dropping the gap to get the bound A+B is valid only when the gap is non-positive — when the search ends at a point the surrogate rates at least as high as x∗. A local search need not, which is why the dial shows A, B and the loss but not the bound.
The same shape on a real taskCOMs against naive gradient ascent · HopperController, 5,126 dimensions
Naive ascent improves at first — near 2,800 after about seven steps in this figure — then deteriorates as further steps carry it further from the data.
COMs climbs slowly and stays high over the 450 steps — as in the toy, where a flattened surrogate and a slow search keep the design near the data for that budget.
Source alignment: Part 2 p. 38. The paper reports the same comparison for the evaluation budget (p. 40): COMs reaches about 2,400 within 25 candidates, naive ascent needs all 128 to reach about 2,100.
Conservatism is learned where the attacker wentso the final search must not go far from there
The 2 × 2 of model and search, 450 steps from the best design in D: plain fit + plain ascent → x=10, worth −0.14; plain fit + trust region → x=10, −0.14 (it passes 2.66 near step 50); conservative fit + plain ascent → x=10, −0.14, where the surrogate still promises 2.31 (it passes 2.65 near step 40); conservative fit + trust region → x=6.32, 2.62. In this toy neither half works alone — and the slow search is still rising when its steps run out: run far longer, it reaches the edge too.
What conservatism promises, and what it does nota lower bound on average over the attacker's designs — not design by design
Where the formula comes from, in one line: the penalty pushes a design down in proportion to μ(x), holds it up in proportion to d(x), and the fit term resists in proportion to d(x), so with one free value per design the balance is f̂ = y − α(μ/d − 1). The two-line derivation is the appendix slide "Backup — the tabular COMs solution". Numbers behind the picture (α = 0.5): d = (0.5, 0.3, 0.2), μ = (0.1, 0.3, 0.6), y = (1, 2, 3), so f̂ = (1.4, 2.0, 2.0); E_μ[f̂ − y] = −0.56 ≤ 0, E_d[f̂ − y] = 0. Proposition 1 (Trabucco et al., 2021) is the network version of the same statement: with α large enough, the surrogate's average over the optimiser's own outputs is at most the truth's, under Lipschitz and kernel-style assumptions. The calculation holds μ fixed and needs μ(x) > 0 only where d(x) > 0: where the attacker leaves the data altogether the tabular loss has no minimum, so this explains the mechanism inside the data and does not show that OOD is solved.
Conservatism is a dial — make it a budgetchoosing α is hard, so state the conservatism in the objective's own units
α's right value depends on the scale of y; τ is a budget on how far the surrogate may rate the attacker's designs above the data. It bounds a gap between two of the surrogate's own averages — not its error, which is never observed offline. With the objective normalised first, the paper uses τ=0.05 on every task, and α becomes the constraint's Lagrange multiplier.
τ=0.1 barely moves for forty steps — conservative enough to stay home. τ=1 and τ=2 climb fastest. Neither end is free.
Turning a penalty into a constraint with an interpretable budget is a move this course makes more than once — Lecture 1's trust region, and TRPO's bound on the policy step in Lecture 10.
Source alignment: Part 2 pp. 36 (the constraint form) and 39 (the τ ablation).
The same disease, one rung up the coursean analogy Lecture 12 takes up
COMs — a conservative objective
An optimiser exploits fθ at designs outside the data.
CQL — a conservative value
A policy exploits Q at actions outside the data.
Source alignment: Part 2 p. 33, where the source deck draws this comparison itself.
Check — what conservatism costs
Conservatism trades B for A. Too little and the optimiser exploits the error; too much and the model refuses to rate anything it has not already seen, which is a very safe way to be useless. In the toy, α=1.3 gives A=1.41 and a returned design worse than the best in D. Writing the conservatism as a budget τ makes the trade visible; it does not remove it.
PRIME — a conservative surrogate for hardware designKumar, Yazdanbakhsh, Hashemi, Swersky & Levine · ICLR 2022
Here f is a latency, which is minimised — so conservative means pushing the predicted latency up at the designs the optimiser finds, and at designs the log records as infeasible. Training over many target applications lets one model serve unseen ones.
The search uses only a log of past simulations; the baseline calls the simulator as it searches.
Source alignment: Part 2 pp. 54–55. The source writes PRIME's loss without COMs' data term; it is reproduced as written.
RoMA — make the surrogate smooth where the optimiser standsYu, Ahn, Song & Shin · NeurIPS 2021
Its assumption: over-estimates come from a network that is not smooth, with spurious spikes between and beyond the data. So it changes the surrogate's shape near each candidate, not its level.
Stage 1 — train it smooth
Noise on the inputs and the worst weights nearby: the fit must hold on a neighbourhood of each point, not at the point alone.
Stage 2 — re-smooth at each step
Flat at the candidate (the gradient term) without moving its prediction there (the second). Then one ascent step on f(⋅;θt).
Source alignment: Part 2 pp. 41–49. The picture is the paper's Figure 2; the objectives are the source's pp. 47 and 49, with B(θ)={θ~:∥θl−θ~l∥F≤ϵ∥θl∥F ∀l} (Stage 1 is the double-perturbation loss of Adversarial Weight Perturbation, NeurIPS 2020; the inner maximisation runs by projected gradient ascent). Stage 1 smooths only where the data is, which is why Stage 2 re-adapts at every candidate — the paper's pipeline figure is on the appendix slide "Backup — RoMA's two stages".
Session 1 — what conservatism fixed, and what it left
Mitigated
B, the optimiser's exploit — on the designs the training attacker visits, with a dial (α) or a budget (τ) to set how hard, and a short, slow search that does not get far from there.
Left
- the protection is tied to the region its attacker explored — and to the search's budget;
- too much of it inflates A;
- it never says how much the model knows, so it cannot tell a gap in the data from a region that is truly poor.
Uncertainty: decide with what the model does not know
Session 2 — where Session 1 left off
What conservatism did
It lowered the scores the optimiser would reach — B shrank — on the designs its own attacker visited, and a slow search kept the final design there.
What it could not tell us
How much the model knows. A penalty treats a gap in the data and a genuinely poor region alike: both are simply rated low.
The papers of Session 2
Uncertainty is Lecture 4's tool, with the sign turned
Online — Bayesian optimisation (Lecture 4)
Optimism. Go where the band is wide: if the model is wrong there, the next measurement says so and the band collapses.
Offline — this lecture
Pessimism. Stay where the band is narrow: no measurement will ever correct a wrong guess, so a wide band is a risk, not an opportunity.
One model, two signs
Where σ comes from for a neural surrogate — Deep EnsemblesLakshminarayanan, Pritzel & Blundell · NeurIPS 2017
A GP gives σ in closed form. A neural network does not. The simplest widely used answer: train several networks and read their disagreement.
The bet
Members trained from different random starts (or resampled data) agree where the data pins the function down, and disagree where it does not.
The offline rule
Plug the ensemble's μ and σ into μ−κσ. If the bet holds, the rule avoids the designs the members argue about.
The paper also has each member predict its own noise variance and adds it to σ2; the disagreement term above is the part that grows away from the data.
In our toy, the alarm is too quiet
NEMO — uncertainty as how easily the model could be talked into any answerFu & Levine · ICLR 2021
θ^D∪(x,y) is the model refit on the data plus one pretend point (x,y) — a different refit for every label. Source alignment: Part 2 pp. 15–16. Computing it exactly would mean refitting for every label; NEMO quantises y into K bins and keeps K models that it updates while it optimises x (Part 2 pp. 17–20; see the appendix backup).
The NML formula, one label at a time
The model class here is kernel ridge regression — the GP mean of Lecture 4 (linear trend plus RBF, noise sd 0.1); refitting with one extra point (x,y) moves the prediction at x by the share s2(x)/(s2(x)+0.12) of the way to y. Numbers: at x=3 one new point moves the model 26% of the way, and the NML spread is 0.14 with mean 1.45 (the model's own prediction); at x=8.5 it moves it 99%, the NML is flat over the labels [−0.5,3.5] (spread 1.17) and its mean is 1.53, where the model says 3.50. The NML mean peaks at x=5.52 (worth 2.56; best in D 2.43); the model's own mean peaks at the edge, x=10 (worth −0.14). The pull to mid-range comes from the bounded label set, as in NEMO's quantised labels.
NEMO's own test — against the ensemble
Source alignment: Part 2 p. 21.
How far can an uncertainty estimate be trusted?Ovadia et al., "Can you trust your model's uncertainty?" · NeurIPS 2019
What they found
Across many methods, calibration — whether a 90 % band holds the truth 90 % of the time — gets worse as the test data drift away from the training data. Ensembles held up best, but they also degraded.
What it means here
An optimiser does not drift gently away from the data: it searches for the most extreme design it can find. Offline MBO puts the uncertainty estimate exactly where such studies find it least reliable.
Not an MBO paper; it is the standard large-scale study of predictive uncertainty under dataset shift, and it tested deep ensembles among other methods.
What uncertainty buys, and what it costs
Assumption added
σ tracks the error: wide where the model is wrong, narrow where it is right.
What it solves
The decision can see the difference between I have checked this and I have not — and avoid the second.
What it leaves
σ is itself a model output. Off the data it can be as wrong as μ, and in the same direction.
Check — reading an ensemble's agreement
Agreement is evidence only where the members could have disagreed. Off the data, members built the same way from the same points tend to extrapolate the same way — our toy's ensemble agreed to 0.24 where it was wrong by 4.4. Uncertainty estimates also degrade under shift for every method studied (Ovadia et al., 2019), and an optimiser seeks out exactly the most shifted designs.
System knowledge: give extrapolation grounds
Knowledge, ordered by how much of it you have
Structure — count what it takes to see the spacefunctional graphical models · Grudzien Kuba, Uehara, Abbeel & Levine, AISTATS 2024
The paper's result is a regret bound for offline optimisation that depends on how well the data cover each clique's marginal, rather than the joint; it also gives an algorithm that infers the structure from data, over the inputs or over a latent representation. The 10,000-versus-210 count is the intuition, not the bound.
Structure — a design better than any in the data
A simulator — Kennedy & O'HaganBayesian calibration of computer models · JRSS-B 2001
Kennedy & O'Hagan is a calibration paper — it fits the simulator's parameters θ and the discrepancy jointly; it is not itself an MBO algorithm. Here θ is fixed and only the discrepancy is learned. Lecture 4's GP plays the role of δ(x).
The equations — physics-informed surrogateshPINN · Lu, Pestourie, Yao, Wang, Verdugo & Johnson · SIAM J. Sci. Comput. 2021
hPINN solves inverse-design problems in optics and fluids with physics-informed networks, imposing hard constraints by penalty and augmented-Lagrangian methods. It is adjacent work, not fixed-dataset offline MBO: it uses the equations, not just logged designs.
What knowledge buys, and what it costs
Assumption added
The structure, the simulator or the equations are right — beyond the data, where they cannot be checked.
What it solves
Extrapolation gets grounds other than the data: parts combine, a simulator supplies the shape, physics rules out the impossible.
What it leaves
Knowledge is often partial or missing, and wrong knowledge does not announce itself — it is trusted most exactly where the data cannot correct it.
Check — when to trust a stitched design
The prediction rests on the structure, not on the fit: if x1 and x2 interact, a small training error can coexist with a badly wrong stitched value. With the right structure and coverage of each part, combining parts that were each seen is exactly how offline optimisation can beat the best design in the data.
Closing
Three answers, side by side
| Conservatism | Uncertainty | System knowledge | |
|---|---|---|---|
| in depth | COMs | Deep Ensembles, NEMO | FGM |
| mechanism | lower the scores the optimiser would reach | estimate what the model does not know; decide with μ−κσ | constrain how the model may extrapolate |
| assumption | the attacker finds the tempting designs | σ tracks the error | the structure, simulator or physics is right |
| strength | curbs optimism where the search goes | separates unchecked from checked | can beat the best design in the data |
| weakness | too much inflates A; tied to its search | σ fails where it matters most | wrong knowledge biases everything |
What all three leave — and what Lecture 6 takes up
Performance
Is the design good? — the objective
Reliability
Can we trust the predicted score? — this lecture
Validity
Is the design possible at all — a molecule, a foldable protein? — Lecture 6
Where we are — a forward model, then a search
| the model | the decision | |
|---|---|---|
| Surrogate (Lec 5 ✓) | forward: fθ(x)≈f(x) | search fθ — conservatively, with uncertainty, or with knowledge |
| Generative (Lec 6) | a distribution of plausible designs, plus a performance model | search its coordinates, refit it, condition it, or steer its sampler |
Hold the shape of this lecture: a forward model searched carefully. Lecture 6 keeps a performance model in the loop and adds a generator that decides where to look — and this lecture's failure reappears there wherever the performance model steers the generator. Lecture 12 meets it again, with a policy exploiting a Q-function.
Backup slides
Reading guide — Session 1 papers: the problem and conservatism
| paper | why we read it | how it connects |
|---|---|---|
| Adversarial examples · Goodfellow et al., ICLR 2015 | the mechanism: gradient search finds where a model is wrong | COMs generates this attack during training and trains against it |
| Design-Bench · Trabucco et al., ICML 2022 | defines offline evaluation; every method here is scored on it | we use its 2021 task versions |
| COMs · Trabucco et al., ICML 2021 — in depth | the clearest conservative surrogate | one-step twin of CQL; PRIME and LCOMs build on it |
| PRIME · Kumar et al., ICLR 2022 | conservatism in real chip design, from logged simulations | builds on COMs |
| RoMA · Yu et al., NeurIPS 2021 | another lever: shape, not level | compared with COMs |
| CQL · Kumar et al., NeurIPS 2020 | the same disease, with a policy as the optimiser | Lecture 12 |
| LCOMs · Qi et al., ICLR 2023 workshop | a conservative surrogate searched in a learned space | Lecture 6 |
Reading guide, continued — Session 2 papers: uncertainty and system knowledge
| paper | why we read it | how it connects |
|---|---|---|
| Deep Ensembles · Lakshminarayanan et al., NeurIPS 2017 — in depth | the standard σ behind μ−κσ | Lecture 4's UCB, sign turned |
| NEMO · Fu & Levine, ICLR 2021 | uncertainty built for offline MBO | same group and year as COMs |
| Uncertainty under shift · Ovadia et al., NeurIPS 2019 | calibration degrades with shift, even for ensembles | stress-tests the uncertainty stream |
| Functional graphical models · Grudzien Kuba et al., AISTATS 2024 — in depth | when structure lets us beat the best design in the data | Lecture 3's graphs |
| Kennedy & O'Hagan · JRSS-B 2001 | a simulator inside a surrogate | Lecture 4's GP as the error term |
| Hard-constraint PINN · Lu et al., SIAM J. Sci. Comput. 2021 | the governing equations themselves | needs a known PDE |
Most of these papers come from one research programme — Levine's group at Berkeley: CQL (2020), COMs and NEMO (2021), PRIME and Design-Bench (2022), LCOMs (2023), FGM (2024). The map slide says so in its Session 2 text.
Backup 1 — offline MBO against Bayesian optimisation
The same goal, argmaxxf(x), under different access — and the difference dictates the method.
| Bayesian optimisation (Lec 4) | Offline MBO (Lec 5) | |
|---|---|---|
| data | active — query f each round | fixed — a static dataset D |
| a mistake is | corrected by the next query | uncorrectable — it is returned as the answer |
| uncertainty drives | where to sample | where not to trust |
| the core risk is | slow convergence | overestimation → an invalid design |
| the design principle | exploration | conservatism |
Both build a surrogate; both maximise something over it. What differs is whether the loop closes. When it does, optimism is self-correcting and therefore cheap — an over-rated point gets queried, found wanting, and the posterior repairs itself. When it does not, optimism is a one-way door.
Backup 2 — NEMO, made tractable
The estimator. pNML is the minimax solution of argminhmaxy′(logp(y′∣x;θ^D∪(x,y′))−logh(y′∣x)), where θ^D∪(x,y)=argmaxθN+11∑(x,y)∈D∪(x,y)logp(y∣x,θ).
Problem. The denominator requires training an MLE for every possible y and then integrating over them — impossible twice over for a deep network.
Fix 1 — quantise. Floor each y into one of K bins, so the integral becomes a sum: ∫yp(y∣x,θ^D∪(x,y))dy≈B∑k=1Kp(⌊yk⌋∣x,θ^D∪(x,⌊yk⌋)).
Fix 2 — amortise. Keep K models and update them incrementally while optimising x, rather than retraining from scratch at each iterate:
for t = 1 … T:
for k = 1 … K: D' ← D ∪ (x_t, ⌊y_k⌋); θ^k ← θ^k + α_θ ∇ LogLik(θ^k, D')
p̂_NML(y | x_t) ∝ p(y | x_t, θ^y) / Σ_k p(⌊y_k⌋ | x_t, θ^k)
x_{t+1} ← x_t + α_x ∇_x E_{y ~ p̂_NML(y|x)}[ g(y) ]
Quantisation flattens the landscape and kills the gradient, so NEMO's head outputs one minus the CDF of a logistic distribution sampled at intervals of 1/K and takes the mean; gradients then flow through the logistic mean μ(x), and Proposition 4.1 guarantees ⟨∇xμ(x),∇xymean(x)⟩≥0 — the smooth surrogate gradient never points against the one we want.
Backup — one COMs loss calculationfreeze the sampled candidates while inspecting the update
One measured design has target y=4 and prediction u=3.5. One adversarial design has prediction v=8. Use α=0.2.
At the measured design
Gradient descent increases u, improving the fit and the relative data score.
At the adversarial design
Gradient descent decreases v, reducing the unsupported high prediction.
Backup — the tabular COMs solutionone free value per design, for intuition
Take a finite set of designs, one free value f^(x) for each, data frequencies d(x) and attacker frequencies μ(x). The loss is a sum over designs, so each f^(x) is solved on its own:
Setting ∂L/∂f^(x)=d(x)(f^(x)−y(x))+α(μ(x)−d(x))=0:
Average the error under each distribution, using ∑xμ(x)=∑xd(x)=1:
Assumed: μ and d are distributions on the same finite set, μ(x) > 0 only where d(x) > 0, y(x) is the mean measured value at x, and μ is held fixed. Where the attacker visits less often than the data, f̂ is pushed above y. If the attacker reaches a design with d(x) = 0 — genuinely out of distribution — the tabular loss has no minimum there at all: this explains the mechanism; it does not show that OOD is solved. The paper's Proposition 1 is the network version of the same average statement, under Lipschitz and kernel-style assumptions.
Backup — RoMA's two stagesYu, Ahn, Song & Shin · NeurIPS 2021
Stage 1 — train it smooth
Noise on the inputs, worst-case perturbations of the weights, B(θ)={θ~:∥θl−θ~l∥F≤ϵ∥θl∥F}; the inner maximisation by projected gradient ascent.
Stage 2 — re-smooth at each step
Flatten the model at the current candidate, without moving its prediction there far.
What the benchmark sayssource-reported best-of-batch task scores; Avg uses the source's normalised aggregate
| method | GFP | Molecule | Supercond. | Hopper | Ant | DKitty | Avg |
|---|---|---|---|---|---|---|---|
| Dataset max | 3.152 | 6.558 | 73.90 | 1361.6 | 108.5 | 215.9 | 1.000 |
| Gradient ascent | 2.894 | 6.636 | 89.64 | 1050.8 | 399.9 | 390.7 | 1.237 |
| MINs | 3.315 | 6.508 | 80.23 | 746.1 | 388.5 | 352.9 | 1.304 |
| CbAS | 3.408 | 6.301 | 72.17 | 547.1 | 393.0 | 396.1 | 1.324 |
| COMs | 3.305 | 6.876 | 110.0 | 2395.7 | 378.8 | 341.4 | 1.589 |
| NEMO | 3.359 | 6.682 | 127.0 | 2130.1 | 393.7 | 431.6 | 1.687 |
| RoMA | 3.357 | 6.890 | 103.9 | 2466.5 | 468.5 | 384.3 | 1.705 |
Naive gradient ascent is not useless — it has the lowest aggregate score among the six methods shown, and on HopperController it returns less than the best trajectory already in the dataset. Every method that beats it does so by adding a constraint on what the surrogate is allowed to believe, not by searching harder.
LCOMs — a conservative surrogate searched in a learned spaceLatent Conservative Objective Models · Qi et al., ICLR 2023 workshop on ML for materials
Chemical space is not a vector space, so a CD-VAE encoder ϕ(x,c) maps a crystal into one, and a conservative surrogate of the lattice energy is optimised inside that latent space, then decoded:
Mean energy improvement 2.25 against supervised learning's 1.10. A generative model supplies the coordinates; a conservative surrogate does the searching — the bridge Lecture 6 crosses.
Five ways to optimise a black box, and what every one of them needswhat the source lecture spends fifteen slides establishing
| method | what it does | what it costs |
|---|---|---|
| Gradient ascent on a proxy | fit fθ to the data so far, step xt+1=xt+η∇xfθ(xt), evaluate | one query per step |
| Genetic algorithms | population, truncation selection of the top E, crossover, mutation | N queries per generation |
| CMA-ES | sample from an adaptive Gaussian N(mt,σt2Ct); update mean, covariance and step size | n queries per generation |
| Bayesian optimisation (Lec 4) | GP posterior, then xt+1=argmaxxAt(x) | one query per round |
| Policy gradient | learn πθ(x) by ∇θEx∼πθ[f(x)]=E[∇θlogπθ(x)f(x)] | n queries per round |