Curiosity-Critic: Cumulative Prediction Error Improvement as a Tractable Intrinsic Reward for World Model Training

Reward an agent only for the prediction error its model can still reduce.

Prediction error splits in two: the part that more training removes, and the noise no amount of training removes. A small side network learns to predict how much of that noise each transition carries, and subtracting its estimate keeps a curious agent off transitions it can never predict.

A curiosity-driven agent, given 35,000 steps in a small grid world, spent the last 20,000 of them in front of a wall of random pixels, learning nothing.

Model-based reinforcement learning takes a detour. Instead of learning a policy directly from experience, it learns a world model first, a network that predicts what the environment does next, and then plans against that model. The detour pays because a model you can query is cheaper than an environment you have to act in: MuZero searches thousands of imagined futures per move without touching the real game. The approach goes back to Sutton's Dyna (1990), and the name most people know it by comes from Ha and Schmidhuber's World Models (2018), which itself credits an older line of Schmidhuber's work for the idea.

The detour only pays if the model is accurate, because errors accumulate along a rollout. A model that is slightly wrong about one step is more wrong about two, and how fast that grows depends on whether the dynamics stretch small differences or shrink them.⁵ So the practical question in model-based RL is less about how to plan and more about how to get a good model out of a finite number of environment steps.

That comes down to which transitions you feed the model. Let an agent wander at random and it spends most of its budget re-confirming things the model already predicts perfectly. Schmidhuber's answer, from a February 1990 technical report and the 1991 conference paper that followed it, was to pay the agent an intrinsic reward, one the environment never provides, for going where the model is unreliable.

The obvious version of that reward has one failure mode stark enough to have its own name: the noisy-TV problem. Point the agent at a television showing static and the reward never runs out, because static is not merely unpredicted, it is unpredictable. There is nothing to learn there, so the error never falls, so the bonus never falls, so the agent never leaves. Curiosity-Critic subtracts that part of the error before paying for it.

Getting from "subtract the part you can never fix" to a number you can compute at every step takes three ideas: a change in the order a sum is added up, an identification of the one term that survives that change, and a second network that estimates the surviving term while the agent is still exploring.

Reward raw error, get trapped by noise

Fix notation first, because everything below is one formula with one term swapped. The agent is in state $s_t$, takes action $a_t$, and the environment moves it to $s_{t+1}$. The world model is $\theta_t$, and $\theta_t(s_t,a_t)$ is its guess at $s_{t+1}$. The prediction error is how far that guess lands from the truth:

The bars are the ordinary Euclidean length of the residual vector, not its square, a distinction that returns when the noise floor gets identified. Schmidhuber's first curiosity reward is this error, handed straight to the agent as payment: $r_t = e(s_t,a_t \mid \theta_t)$. The paper calls this Curiosity V1 and extends the label to the whole family of later variants that keep the same shape and only change how error is measured.

Reading that reward as an assumption makes the flaw visible. Paying for error assumes that error marks something worth learning. Schmidhuber put it plainly when he looked back on the work: "it was implicitly and optimistically assumed that the predictor will indeed improve whenever its error is high."² Some error does not mean that. A coin flip you are about to observe has high prediction error at every visit, at every point in training, forever, so an agent that finds one gets paid well there for standing still and the payment never shrinks.

The figure below is the paper's own test environment, and you can walk it. The left half of the grid is learnable: every cell shows the same fixed pattern of 200 black-and-white pixels every time you visit it, so a model can memorize it and drive its error to zero. The right half re-flips all 200 pixels on every visit, so no amount of training helps. Press play with the Curiosity V1 rule selected and watch where the agent settles. The other rules are the alternatives this page works through; come back to them once you have met each one.

V1's agent is not malfunctioning. It is doing what it was paid to do, under a reward that assigns the highest and most durable value in the world to a wall of random pixels.

Reward improvement instead

Schmidhuber diagnosed this himself, in April 1991, before anybody had a name for it: "in non-deterministic environments the controller will focus on parts of the environmental dynamics which are inherently unpredictable. This is because the adaptive model usually will produce incorrect predictions for the uncertain parts of the environment. Therefore the controller will receive reinforcement although it cannot be expected that the world model will improve."³

His fix was to change what gets paid for. Do not pay for being wrong, pay for getting less wrong. Take one gradient step on the transition you just saw, then measure the error again on that same transition, and hand the agent the difference:

Both terms use the same observed $s_{t+1}$, so the environment is sampled once and the second evaluation is free: you already have the updated weights and you already have the target. On a learnable transition the step buys real ground and the difference is positive. On a coin flip the step fits this particular flip and nothing else, so across visits the improvements average out to roughly nothing. The paper calls this Curiosity V2.

Two things weaken (2) as an estimate of "how much is left to learn here". It is a single sample, and a difference of two noisy numbers is noisier than either one. And the quantity it subtracts, $e(\cdot \mid \theta_{t+1})$, is the error of the model you have right now, so how well that quantity stands in for the error of a fully trained model depends on how close to fully trained your model already is. Early in training, when the guidance would help most, it is least trustworthy.

Schmidhuber saw both problems and did something about them, and the paper's framing skips it. The 1991 system did not subtract a single post-update sample. It trained a second network whose job was to output the model's expected change, and paid the agent the magnitude of that expectation rather than the magnitude of one draw. His reason is the modern one: "although noise was unpredictable and led to wildly varying target signals for the predictor, in the long run these signals did not change the adaptive predictor parameters much, and the predictor of predictor changes was able to learn this."² Equation (2) is the one-step instantiation Curiosity-Critic names for comparison, and the paper says so once, but it then argues against 1991 as though the single sample were all there was.

Score the whole history

Both rewards read one transition: the one the agent is standing on. That leaves an odd gap between what the agent is paid for and what you actually want from it. What you want is a world model that is accurate over everything it will be asked about, and a gradient step taken here changes the model's predictions everywhere, sometimes for the better and sometimes for the worse. None of that enters (1) or (2).

So write that down instead. Let $\mathcal{D}_t$ be the history: every transition the agent has visited up to time $t$. Pay the agent, at each step, the total error reduction its update produced across that entire history:

Read the summand as a before-and-after on one old transition: how wrong the model was about it a moment ago, minus how wrong it is now that the model has been updated on the current step. Sum that over the entire past and you have credited the current visit with everything it did for the model, not just what it did for itself. An update that sharpens one cell and quietly degrades fifty others scores badly here and scores perfectly under (1) and (2).

The noisy transitions mostly drop out of (3), for a reason that runs like this: suppose the model already predicts the average of a noisy transition's outcomes, and a gradient step taken somewhere else nudges that prediction by a small amount in some direction. Whether the error on a stored noisy sample goes up or down depends on which side of the average that particular sample fell, and stored samples fall on both sides equally often. So the first-order effect cancels and the contribution is about zero. It is an approximation, not an identity: it needs the noise to be symmetric around what the model predicts, and the second-order term is small but strictly positive, so unrelated updates make noisy transitions very slightly worse rather than exactly neutral. The cancellation argument also says nothing about the transition the agent is standing on, whose own error the update always shrinks a little whether or not there was anything to learn. Sum over everything else and what survives in expectation is the improvement on transitions that were learnable.

Then there is the cost. Computing (3) once means running the world model over every stored transition twice, before and after the update, so a step at time $t$ costs on the order of $t$ forward passes and needs the history in memory. Do that at every step of a run of length $T$ and the total is on the order of $T^2$, which at the paper's 35,000 steps is roughly 1.2 billion extra forward passes spent computing a scalar that gets multiplied into a policy gradient, which is not a trade anyone will take.

Neither the objective nor its cost is new. Schmidhuber's formal theory of creativity states this whole-history quality measure, insists that "both the old and the new model have to be tested on the same data, namely, the history so far", and then concedes that "in practical applications one cannot frequently measure predictor improvements by testing predictor performance on the entire history so far."⁴ That is as far as the argument had gone. Moving it is what the next section does.

Why the history sum collapses

Stop asking what (3) costs at a single step and ask what it adds up to over a run. An RL agent does not maximize one reward, it maximizes the discounted sum of all of them, with a discount $\gamma$ that says how much a reward later is worth now:

Set $\gamma = 1$ for a moment, so no reward is discounted, and look at what (4) is: a double sum over pairs $(t, t')$ with $t' \le t$, which is a triangle. Summing it row by row is the expensive reading, one row per step, each row as wide as the history. Summing it column by column is the same number arranged differently, and a column has a structure a row does not.

Follow one column, transition $t'$, down through the steps. At step $t$ that column's entry holds $+e(t' \mid \theta_t)$ and $-e(t' \mid \theta_{t+1})$. At step $t+1$ it holds $+e(t' \mid \theta_{t+1})$ and $-e(t' \mid \theta_{t+2})$. The minus of one row and the plus of the next are the same number, the error of the same transition under the same weights, so they cancel. They cancel all the way down. Two terms are left standing, the plus at the top and the minus at the bottom:

Nothing was approximated. A transition's total lifetime contribution to cumulative error improvement is its error when the agent first stood on it, minus its error at the end of training, and every intermediate step it lived through cancelled against its neighbour. The whole history disappeared from the bookkeeping because the history was being double-counted with opposite signs.

Switch the figure between the two summation orders and select a column to watch the pairs cancel. Push the horizon slider to both ends: at $T = 3$ the triangle is small enough that the row-wise cost looks harmless, and at $T = 10$ it is 66 entries against 11 surviving terms.

Equation (6) identifies a per-step reward that any RL algorithm can consume under whatever discount it uses:

Away from $\gamma = 1$ the identity becomes an inequality rather than breaking. Equation (5) in the paper splits $C(T)$ into the per-step term above plus a leftover history term carrying the factor $(1 - 1/\gamma)$. For $\gamma$ strictly below 1 that factor is negative, and the errors it multiplies are norms and so never negative, which makes the leftover term negative and the per-step term an upper bound on the true objective. Optimizing the bound is the usual move; the bound is tight exactly at $\gamma \to 1$, which is also the discount the paper wants on principle, since what you care about is the model's accuracy when training stops, not next week.

The gap between bound and objective does not shrink uniformly. That leftover term is roughly $(1-\gamma)\,\bar{e}\,T^2/2$ for $\gamma$ near 1, with $\bar{e}$ the average per-transition error, so it is small when $(1-\gamma)T^2 \ll 1$ and not otherwise. Fix any $\gamma < 1$ and run long enough and the slack stops vanishing. So $\gamma \to 1$ means the horizon is held fixed and the discount is taken to exactly 1, in that order, not that a discount of 0.99 is close enough over 35,000 steps.

The irreducible noise floor

Equation (7) is per-step and exact and still not computable, because $\theta_{T+1}$ is the model you will have when training ends, and the run is only partway through. Two substitutions bridge that. Replace the end-of-training model with its convergence limit $\theta_\infty$, which is exact as the run gets long. Then replace the single-sample error under that limit with its average over the transition's possible outcomes, which trades a noisy quantity for a stable one and is what makes the next section possible at all:

The subtracted term is the paper's asymptotic error baseline. The world model is trained on squared error, and the constant that minimizes expected squared error is the mean, so a converged model outputs the average outcome: $\theta_\infty(s_t,a_t) \to \mathbb{E}_{\mathcal{P}}[s_{t+1} \mid s_t, a_t]$. Put that back into the error and the baseline becomes the average distance from an outcome to the average outcome:

Read that as a property of the environment alone: it has nothing to do with the model, the architecture, or how long you train. A transition whose outcome is fixed has a baseline of zero. A transition whose outcome is a coin flip has a baseline set by how far the flips scatter. Subtract it from the current error and what remains is error the model can still remove, which is the only kind worth paying for.

The quantity in (9) is the mean Euclidean distance to the mean, which the paper calls the mean absolute deviation; those agree for a single number and part ways for a 200-dimensional vector, where the per-coordinate absolute deviation is a different quantity entirely. And the quantity in (9) is a floor only for a model trained the way this one is trained. Because the error metric in (1) is the unsquared norm while the training loss is the squared one, the two have different minimizers: squared error is minimized by the mean, unsquared distance by the geometric median. Where those differ, a differently-trained model would score lower on the metric, so (9) is the floor for an MSE-trained model rather than the floor over all models. In this paper's grid world the two coincide exactly, because a fair coin flip is symmetric about its own mean.

Numbers make the floor concrete. Take a transition that emits 200 bits, of which $k$ are re-flipped at random on every visit and $200-k$ are fixed. A converged model outputs 0.5 on each random bit and the true value on each fixed one, so it is off by exactly 0.5 on $k$ coordinates and 0 on the rest. The floor is $0.5\sqrt{k}$, and it is that value on every single draw, not on average, because a bit that lands on 0 and a bit that lands on 1 are both exactly 0.5 away from 0.5. At $k = 200$ the floor is $0.5\sqrt{200} = \sqrt{50} \approx 7.071$, which is the paper's noisy half. At $k = 0$ it is zero, which is the learnable half.

Drag $k$ across its full range in the figure. Both ends are the paper's actual environment and everything between is the partially-learnable case the method is built to grade. Watch the teal band, the reward, shrink to nothing as $k$ reaches 200: at that setting the curve starts on the floor and never leaves it, so there is no reward to collect on the first visit or the thousandth.

An appendix bounds the floor from above. Jensen's inequality gives $\mathbb{E}\lVert X\rVert \le \sqrt{\mathbb{E}\lVert X\rVert^2}$, so the floor is at most the square root of the summed per-coordinate variance, with equality only when the distance comes out the same on every draw. That equality happens to hold exactly in this grid world and does not hold in general, so the floor is not the standard deviation and it is not the variance.

A critic that learns the floor

The baseline in (9) needs two things nobody has: the environment's transition distribution and a converged model. What the agent does have, at zero extra cost, is the post-update error. It already ran the model before the step to get $e_{\text{before}}$, it already took the step, and running the model once more on the same input gives $e_{\text{after}}$. As the model converges, that number approaches the floor.

Handing that single number to the agent as the baseline is Curiosity V2, with the two weaknesses from earlier. So instead of using it, learn from it: train a small network $\phi$ to predict $e_{\text{after}}$ from $(s_t, a_t)$, and use its prediction as the baseline. The reward becomes

The natural objection is that this looks circular: a second learned thing, trained on the output of the first learned thing, to help train the first learned thing. It works because the two jobs are not the same size. The world model has to produce a 200-dimensional vector. The critic has to produce one number, and that number answers a much cruder question, how hard is this transition to predict at all. It has far less to learn, so it gets there far sooner, and while the world model is still bad the critic is already sorting the learnable from the hopeless.

Notice which critic is queried in (10). It is $\phi_{t+1}$, the critic after its own update on this step's target, so the estimate already includes the sample that was just taken. The loop is nine lines:

# one environment step of Curiosity-Critic, batch size one
a = policy.act(s)                    # eps-greedy over the four neighbours
s_next, obs = env.step(a)            # observe this cell's 200 bits

e_before = norm(world(s) - obs)      # L2 error, before the update
world.step(s, obs)                   # one Adam step on MSE loss
e_after  = norm(world(s) - obs)      # same sample, updated model

critic.step(s, e_after)              # the critic regresses e_after
r = max(0.0, e_before - critic(s))   # Eq. (10), then clipped at zero
policy.update(s, r / run_std(r))     # V(s) += 0.05 * (r_norm - V(s))
s = s_next

This beats a raw $e_{\text{after}}$ in two ways, and they are different mechanisms doing different work. First, variance: $\phi$ is a regression fit over many visits, so it returns something close to the mean of $e_{\text{after}}$ while (2) subtracts a single draw of it. A lucky draw on a noisy transition hands V2 a big reward, the agent's stored value for that cell remembers it, and the agent goes back. Second, coverage (this is what separates the neural critic from a tabular one that stores a separate baseline per cell): the critic sees the same factored input as the world model, one-hot row concatenated with one-hot column, so a visit to one cell moves the estimate for every cell sharing that row or column. A cell the agent has never stood on already has a baseline, inherited from its neighbours, which means the reward there is informative on the first visit rather than the twentieth. That shows up in how fast the run gets going and where it spends its time, more than in the error it finishes at.

Switch between the four baselines in the figure below with the noisy cell selected. The bars are all the same current error, split into the part that gets subtracted and the part that survives as reward. Curiosity V1 subtracts nothing and pays 7.07 on every visit for the entire run. The oracle subtracts the true floor and pays exactly zero. The critic lands close to the oracle without ever being told the floor, which is the claim; V2 lands in the same place on average with a whisker around it, which is the difference.

The paper's claim that the critic finishes first is checkable, and the released training traces check out. On the noisy cells the critic's mean estimate first lands within 5% of 7.071 at step 400, about 1% of the run, while the world model at that moment is still at roughly 7 and has learned essentially nothing. On the learnable cells the critic's estimate climbs for the first 600 steps before it starts falling, which is not a bug: it is tracking the post-update error, and early in training that error really is large there.

Both earlier rewards now read as one formula with the baseline set differently. Assume the environment is deterministic and the baseline is zero, and (8) is Curiosity V1. Approximate the baseline by the single post-update error and (8) is Curiosity V2. The framing holds for any choice of error metric, so variants that measure the error in a learned feature space, ICM among them, sit in the same family. The one thing it understates is how much of the apparatus was already there: Schmidhuber's 1991 confidence module was a network trained to output the model's expected error, and in the reported experiments it regressed the absolute deviation, which is the scalar Equation (9) is about.

The word "asymptotic" is exact in one place and loose in another, and the figure above shows where. What $\phi$ regresses is $e_{\text{after}}$, the model's error now, after one step. On a noisy cell that number never moves, so it already equals the asymptotic floor and the critic locks onto 7.07 within 400 steps. On a learnable cell it is still falling at the end of the run, so the critic there is tracking the model's current error rather than its eventual zero. The paper's released traces make that visible: the critic's mean estimate over learnable cells peaks at 7.14 around step 600 and is still descending at step 35,000, ending at 1.18.

That difference is what the gap between the critic and the oracle measures, and it shows up in the occupancy numbers. On a learnable cell the oracle subtracts a hard zero, so it keeps the full error as reward. The critic subtracts its estimate of the error one step from now, which on a learnable cell is most of the error, so it keeps only the sliver on top. The contrast the critic draws between a learnable cell and a noisy one is real but much weaker than the oracle's, and the agent spends 70.9% of its late run in the learnable half against the oracle's 95.3%. On the noisy side the same design leaves a small permanent residue for the opposite reason: a gradient step always fits the sample it just saw, so $e_{\text{after}}$ sits a little under the floor and the subtraction under-corrects. In the released traces the critic's late estimate on noisy cells averages 6.551 against the floor of 7.071, low by 7.35%.

One grid, two halves

The environment is deliberately small. In a 30×30 grid where you built the noise yourself, any difference between methods has to come from the reward signal, because nothing else differs: every method trains the same architecture with the same optimizer starting from the same warm-up weights.

Columns 0 to 14, all 30 rows, are the learnable half: 450 cells, each emitting a fixed 200-bit pattern. Columns 15 to 29 are the noisy half: each visit draws 200 fresh fair coin flips. The agent starts at cell (15, 15), one step onto the noisy side of the border, and picks one of four directions per step. Movement is deterministic; all the randomness lives in what the cells show you.

The world model does something narrower than Equation (1) suggests, and the paper says so in an appendix. It takes a 60-dimensional input, one-hot of the row concatenated with one-hot of the column, and predicts that cell's 200 pixels through a single hidden layer of 1024 units. The action is not an input at all. Since movement is deterministic, predicting the next state would be trivial, so the model predicts the observation instead. This is a legitimate simplification for the question being asked, which is whether a reward can sort learnable transitions from unlearnable ones. It does mean the experiment never exercises transition modelling, action conditioning, or the compounding-rollout error that motivated the paper's opening.

One visit to a cell in each half, with the quantities the loop computes:

# a learnable cell, visited for the sixth time
s        = onehot(row=7) | onehot(col=3)     # (60,)  two ones, 58 zeros
obs      = patterns[(7, 3)]                  # (200,) fixed 0/1, 88 ones
e_before = 4.31   # the model has partly learned this pattern
e_after  = 4.02   # one gradient step helped, and the help persists
critic   = 3.98   # pooled estimate of e_after here
reward   = 0.33   # positive: there is still error to remove

# a noisy cell, visited for the six-hundredth time
obs      = bernoulli(0.5, size=200)          # re-drawn on every visit
e_before = 7.071  # ||0.5 - obs|| is sqrt(50) for EVERY draw
e_after  = 6.98   # the step fits this draw, and the next draw undoes it
critic   = 6.98   # pooled estimate of e_after here
reward   = 0.09   # near zero, and it never grows

The policy is a lookup table, not a network. A 30×30 table holds one value per cell, updated as a running average of the normalized reward received there, with step size 0.05. The agent takes the highest-valued of its four neighbours, or a random one 30% of the time. The discount is zero in the released configuration, so the table never propagates value across the grid: the agent compares four adjacent cells and nothing further. The $\gamma \to 1$ limit back in “Why the history sum collapses” is the discount on the curiosity reward stream used to derive the per-step form, and it is a different $\gamma$ from this one, so the zero discount here neither contradicts the derivation nor puts it to work.

A few implementation details shape the numbers. The world model and the critic each take one Adam step per environment step at learning rate 0.001, and the model is warmed up for 100 random steps first, from identical weights for every method within a seed. Rewards are clipped at zero before use, which binds only on the methods that subtract a baseline and can dip negative. And rewards are divided by a running estimate of their own standard deviation, so a method whose rewards are large in absolute terms does not effectively get a larger learning rate. That last control has a side effect the paper does not mention: dividing by a spread without subtracting a mean rescales each method's rewards by a different and drifting factor, so the value table's shared initialization of 3.0, which the paper offers as giving every method the same starting optimism, does not do that.

What actually happened

All nine methods, five seeds each, 35,000 steps. The metric is the world model's mean L2 error over the 450 learnable cells, measured by querying every one of them directly, which is a fair test because the agent never gets to choose what it is graded on.

The neural critic wins among the methods that are not told the answer, at 1.858 ± 0.080, with the tabular ablation next at 1.912 ± 0.070; the oracle, handed the floor outright, reaches 1.736 ± 0.063. Every non-critic method is worse than all three. The neural critic is also the fastest to any error threshold: below 3.0 at step 13,400, against 21,700 for the best non-critic baseline and 34,400 for Curiosity V2. It spends 70.9% of its last 5,000 steps in the learnable half, against the oracle's 95.3%, with no knowledge of which half is which.

That speed ladder needs one addition and one hedge, both from the authors' own released traces. The addition: an undirected random walk reaches 3.0 at step 16,800 and 2.5 at 24,500, which is faster than the strongest non-critic baseline and faster than the paper's own tabular critic, and it is the one method the published ladder leaves out. The hedge: with five seeds the gap between the neural and tabular critics is not statistically separable, at p = 0.35 on a permutation test over the per-seed finals. What five seeds do support is that every critic-based reward beats every non-critic reward, and that the oracle still beats the learned critic.

Curiosity V1 finishes at 7.114 ± 0.147 and never improves. The paper calls that indistinguishable from an untrained model, and the arithmetic says something sharper. An untrained network here outputs roughly zero and the fixed patterns have 88 ones out of 200, so an untrained model scores about $\sqrt{88} \approx 9.38$. The 7.114 is instead almost exactly $\sqrt{50} \approx 7.071$, which is the score of a model that outputs 0.5 at every pixel. The noisy half can teach exactly one thing, the average of a coin flip, and a model that has learned it is off by 0.5 on every bit of any binary target, learnable or not, which is what 35,000 steps of curiosity bought.

Visitation count is the row that does not fit the story told around it. Its bonus is $1/\sqrt{N(s)}$, larger for cells you have seen less, which ought to spread coverage evenly. It ends at 5.588 ± 0.794, worse than an undirected random walk, and the paper explains that as the agent continuing to spend substantial time on noisy cells. Recomputing from the released trajectories says the opposite: visitation count puts 62.2% of its 35,000 gradient steps on learnable cells, more than the neural critic's 56.4% and more than the random walk's 51.9%. It had the largest learnable-half budget of any non-oracle method and got the second-worst model.

The trajectories do explain the result, through a different variable. Count how many of the 450 learnable cells each method ever touches. The undirected random walk, Curiosity V2, and all three critics reach essentially all 450, as does Random Network Distillation fed the cell address rather than the pixels (one of two RND variants, both explained below). Visitation count reaches only 333 on average, leaving over a hundred cells the world model is graded on but has never once trained on. Across the nine methods, the number of learnable cells reached correlates with final error at −0.94, while the share of time spent in the learnable half correlates at only −0.64. Visitation count's per-seed behaviour shows the same thing from the side: its late-run learnable-half share is 95.1, 1.2, 100.0, 97.3 and 0.0 percent across seeds, and even the seed that spends 99.8% of its entire life on learnable cells only reaches 4.578, because it spends that life circling a subset of them.

The tabular critic is the clean counterexample to reading occupancy as the driver. It spends 26.5% of its steps in the learnable half, half of what the random walk manages, and still finishes at 1.912 against the random walk's 2.348, because its visits are the most evenly spread of any method in the comparison. What the final number tracks is coverage of the learnable region, cell by cell, rather than gross time inside it.

A count bonus, of all things, fails to cover the grid, and the reason is the policy it was dropped into. The bonus points toward the least-visited cell in the world, but this agent has a zero discount, so it only ever compares the four cells it is touching and can never navigate toward a distant unvisited region. Count-based exploration in its original form put the bonus inside a planner that re-solved the value function every step, and that planner is what carried the agent to the frontier. Strip it out, hand the bonus to a one-step greedy walker, and the walker grinds down counts wherever it happens to be standing. A prediction-error signal survives the same treatment because error is high where the model is bad, and the model is bad in patches the agent can feel from an adjacent cell.

The two RND variants are a matched pair. Random Network Distillation scores how novel an input looks by how badly a trained predictor matches a frozen random network on it, and that mismatch shrinks with repeated exposure to the same input. Fed the row and column of the cell, it behaves like a soft visit count and is the strongest non-critic method at 2.220 ± 0.109. Fed the 200 noisy pixels instead, every visit to the noisy half is a genuinely new input drawn from a space of $2^{200}$, the mismatch never shrinks, and it fails the same way V1 does at 6.842 ± 0.212. Same algorithm, same reward equation, different input channel, and that alone moved the final error by a factor of three.

What the grid world cannot show

The paper is candid that the evidence comes from one small environment, so it is worth being specific about what that leaves open: the reward derivation is general and the experiment is not. There is no transition model, no action conditioning, no multi-step rollout, and therefore no test of the compounding-error problem the introduction opens with. Nothing here has been run at Atari scale or on continuous control. Atari is where rewards of this family have had to prove themselves (see DQN for that benchmark).

The environment is also a favourable one for the neural critic specifically. "This cell is noisy" is the predicate column ≥ 15, a single linear function of one block of the critic's one-hot input, uniform down all 30 rows. A shared-weight network over that encoding picks up most of that rule from a handful of visits, which is a large part of why the estimate is calibrated by step 400, and nothing says a real environment's learnability boundary has that shape. The paper's description of the grid also oversells the structure in it: the fixed patterns are cyclic shifts of one random binary vector, which leaves adjacent cells disagreeing on 46% of their pixels and correlated at about 0.07, and the shift wraps, so the 450 cells carry only 200 distinct patterns between them. What lets the world model generalize across cells is the shared row and column weights of the one-hot encoding, not any resemblance between the patterns themselves.

The baseline the paper names most prominently is also the one it does not run. It presents ICM, the inverse-dynamics feature-space variant, as an instance of Curiosity V1 that its framework subsumes. ICM attacks the same failure from a different side: it measures error in a feature space trained only to decode which action was taken, so parts of the observation the agent cannot influence get filtered out before the error is computed. The noise in this grid is action-independent by construction, which is the case ICM was designed for, so an ICM run here is the comparison a reader most wants, and the one that is not there.

The closest published relative also goes uncited. Mavor-Parker and colleagues (ICML 2022) predict the mean and variance of the next state with separate heads and reduce the intrinsic reward on transitions with high predicted variance, which is prediction error minus a learned aleatoric noise estimate. Curiosity-Critic arrives at a similar reward from an entirely different direction, by asking what the cumulative objective collapses to, and by regressing a single scalar instead of a per-dimension variance. Those two moves are its own; subtracting a learned noise floor from a prediction error is by now a small family.

What the paper does establish, on the evidence it collected, is narrow and real. A learned scalar estimate of how wrong a model will remain is cheap enough to train alongside the model, converges early enough to be useful, and a reward built on it beats every reward in this comparison that lacks one. Subtracting it turns a reward that gets stuck on static into one that runs out when there is nothing left to learn.