--- title: "The latent heat of insight" author: "Miraia S. Chiou" creator: "Miraia Chiou" published: "2026-09-18" identifier: "epics-bsr-006" source: "https://epistemicphysics.com/articles/the-latent-heat-of-insight/" citation: "https://epistemicphysics.com/cite/?work=epics-bsr-006" --- # The latent heat of insight Watch competing explanations trade places, discover the price of flexibility, and explore how memory changes the insights we can sustain. [Read the illustrated article](https://epistemicphysics.com/articles/the-latent-heat-of-insight/) Imagine watching a coin that keeps landing heads just a little too often. At first you shrug. Coins do that. After a while, the same small anomaly begins to look like a pattern. Eventually the explanation changes: perhaps this coin has a bias. The next head now feels less surprising. Something has clicked, even though the coin has been doing the same thing all along. That is the puzzle of insight: evidence arrives one trial at a time, yet explanations can seem to reorganize in a flash. Can we put a number on the relief of finally seeing the pattern? And could an observer with a short memory miss a pattern that an archive makes obvious? We can build a small mathematical theater for these questions. Its actors are two competing models. Its currency is surprise. Its resemblance to melting ice is intriguing enough to explore carefully. ## Two explanations at the same laboratory bench Let each trial be a coin toss, independent of the others, with a constant but unknown heads probability $p$. These are our sampling assumptions. We compare: | | Fair model $M_0$ | Flexible model $M_1$ | | ---------------------------------------- | ---------------------------------------- | ---------------------------------------- | | Claim | $p=1/2$ | $p$ can lie anywhere between 0 and 1 | | Initial uncertainty about $p$ | None inside this model | Uniform $\operatorname{Beta}(1,1)$ prior | | Prediction after $Y$ heads and $N$ tails | $1/2$ | $(Y+1)/(Y+N+2)$ | | Main advantage | Concentrates its predictions immediately | Can learn a persistent bias | There are two different uncertainties here. Inside $M_1$, we wonder which bias the coin has. Above both models, we wonder whether flexibility is needed at all. The probability of $M_1$ is a separate quantity from its estimated heads rate. Suppose the ordered record contains $n=Y+N$ trials. The **marginal likelihood**, also called model evidence, is the probability that a model assigns to that record before fitting a single preferred parameter. For our two models it is exactly $$ Z_0=2^{-n},\qquad Z_1=\int_0^1 p^Y(1-p)^N\,dp =B(Y+1,N+1) =\frac{Y!N!}{(n+1)!}. $$ Here $B(a,b)$ is the Beta function, the integral that normalizes a Beta density. We are scoring an ordered sequence. Scoring only the counts would multiply both expressions by the same binomial coefficient, which cancels in their ratio. That ratio, $Z_1/Z_0$, is the **Bayes factor**. It multiplies the prior odds of the models to produce their posterior odds. This is the standard model-comparison rule described by [Kass and Raftery](https://www.stat.cmu.edu/~kass/papers/bayesfactors.pdf). Write $\pi=P(M_1)$ for the flexible model’s prior probability and $w_n=P(M_1\mid D_n)$ for its probability after the record $D_n$. Then $$ \frac{w_n}{1-w_n} =\frac{\pi}{1-\pi}\frac{Z_1}{Z_0}. $$ The prior on a model and the prior inside a model play different roles. Choosing $\pi=0.1$ favors the fair explanation initially; it does not change the uniform distribution of biases considered inside $M_1$. ## Why the right explanation can lose early Start with equal model probabilities. One head gives both models likelihood $1/2$, so nothing happens to their relative standing. Add a tail. The fair model assigns probability $1/4$ to this two-trial record; the flexible model assigns $1/6$. Its posterior probability falls to $$ w_2=\frac{1/6}{1/6+1/4}=\frac25. $$ The flexible model spreads its initial bets across many coins, including nearly all-heads and nearly all-tails coins. Those extreme possibilities predict the mixed record poorly. Averaging over them costs probability. This is an **Occam penalty**: unused flexibility lowers a model’s marginal likelihood. It arises from averaging predictions, without an extra rule that complexity is bad. A second head instead would give $Z_1=1/3$ and $w_2=4/7$, so the early dip is a common pattern rather than an inevitable ritual. Now imagine the actual generating rate is $p_*=0.6$. The asterisk marks the rate used by the world in our experiment. The fair model can never adjust, while the flexible model can gradually learn it. Flexibility starts collecting a return. Before revealing trials below, predict which observer will sustain a preference for the flexible model. Try the 60% coin with 50-trial memory, then change the memory to 200. Finally try the fair coin: how often does a temporary streak look like an insight? [Which explanation will the next packet favor?](https://epistemicphysics.com/articles/the-latent-heat-of-insight/#bsr-insight-lab) The teal and gold curves share every revealed trial. Their difference comes entirely from what they retain. Changing the model prior lets you compare a welcoming observer with a skeptical one without changing the coin. ## The price of an explanation Turn probabilities into additive costs by defining $$ F_j=-\log Z_j. $$ Logs are natural logarithms, so the unit is a **nat**; dividing by $\log 2$ converts nats to bits. This quantity is called Bayesian free energy. Lower $F_j$ means the model assigned more probability to the observed sequence. Our exact formulas give $F_0=n\log 2$ and $F_1=-\log B(Y+1,N+1)$. A large-sample approximation reveals the competition. Put $\widehat p=Y/n$, the observed heads fraction. Away from the endpoints 0 and 1, Stirling’s factorial approximation gives $$ F_1\approx nH(\widehat p)+\frac12\log n -\frac12\log\!\bigl(2\pi\widehat p(1-\widehat p)\bigr), $$ where $$ H(p)=-p\log p-(1-p)\log(1-p) $$ is binary entropy, the average surprise when predictions match a coin’s actual rate. The first term pays for the data. The logarithmic term pays for learning the bias. Because that second term grows slowly, a persistent predictive advantage can eventually outweigh it. The fair coin’s excess cost at a fixed rate $p_*$ is $$ D=\operatorname{KL}\!\left(\operatorname{Bern}(p_*)\,\Vert\,\operatorname{Bern}(1/2)\right) =\log2-H(p_*). $$ The notation $\operatorname{Bern}(p)$ means the binary distribution with heads probability $p$. The **Kullback-Leibler divergence**, abbreviated KL, measures an expected excess log loss when one distribution is used to predict data from another. Replacing the observed frequency by its typical value $p_*$ and dropping constant terms gives a rough crossover equation, $$ nD\approx\frac12\log n. $$ For $p_*=0.6$, $D\approx0.02014$ nats per trial, and the large root is about 119 trials. This is a scale estimate. The omitted constant, prior odds, and random streaks all move the actual crossing. There is no universal 119th-trial revelation. ## Where the heat analogy becomes useful An exact identity connects the cumulative cost with incoming surprise: $$ F_j(D_n)-F_j(D_{n-1}) =-\log P_j(y_n\mid D_{n-1}). $$ Here $y_n$ is the latest outcome. A model’s increase in cost is precisely how surprising it found that outcome. This identity uses a growing record; deleting old trials, as the gold notebook does, adds a separate change to the cost. Suppose a predictor uses heads probability $q$. Its expected next-trial cost under the true rate $p_*$ is $$ L(p_*,q)=-p_*\log q-(1-p_*)\log(1-q) =H(p_*)+\operatorname{KL}(p_*\Vert q). $$ This gives a tidy side-by-side comparison: | Before learning the bias | After learning it accurately | | ------------------------------------ | ------------------------------------- | | Predict $q=1/2$ | Predict $q\approx p_*$ | | Expected surprise $\log2$ | Expected surprise approaches $H(p_*)$ | | Pay the full mismatch $D$ each trial | Mismatch tends toward zero | The asymptotic savings is exactly $D$. This is the mathematical heart of the “latent heat of insight” metaphor: an explanation that fits releases us from a recurring surprise cost. Earlier costs remain paid; the savings concerns future predictions. For a 60% coin, the improvement is modest: about 0.02014 nats per trial. That small amount accumulates. An explanation can matter greatly over a long future even when the next prediction changes only a little. A physical latent heat is energy transferred during a phase change. Our quantity has units of information per trial. Converting it into joules, brain activity, or a person’s reported relief would require a physical or psychological model. The identity itself tells us exactly what a prediction experiment should measure: the difference in expected log loss. ## Does an explanation actually undergo a phase transition? The two branches can cross. If we always choose whichever model has larger posterior probability, the chosen label changes abruptly at the crossing. That hard choice can look like a miniature change of phase. The full Bayesian observer carries both models. Its marginal likelihood is their weighted sum, $$ Z=(1-\pi)Z_0+\pi Z_1, \qquad F=-\log\!\left((1-\pi)e^{-F_0}+\pi e^{-F_1}\right). $$ This is a smooth, softened minimum of the prior-adjusted branches. Finite positive likelihoods give a smooth dependence on their relative scores. Trials arrive discretely, and a very diagnostic trial can cause a large update, but a branch crossing alone establishes no thermodynamic singularity or first-order transition. That distinction makes the analogy more interesting. Apparent suddenness can arise at several levels: a steep posterior change, an action threshold, a conscious report, or a genuine collective instability. They need different explanations. The model-averaged prediction illustrates the separation: $$ P(\text{next head}\mid D_n) =(1-w_n)\frac12+w_n\frac{Y+1}{n+2}. $$ Even when the model probability swings strongly, the next-head prediction may move only between roughly 0.5 and 0.6. A large change in which story we tell can accompany a small change in what we predict next. ### Put explanations to work with a shared budget A preference between models becomes useful when it improves a decision. Spend a limited observation budget across three coins, then choose fair, flexible, or model-averaged predictions. Each coin has a sealed future block. Its outcomes are kept separate from the inspected training trials, so the final bill tests prediction beyond the evidence used to choose the recipe. [Spend six tokens on three hidden coins](https://epistemicphysics.com/articles/the-latent-heat-of-insight/#play-explanation-desk) The token budget limits access to observations; it is separate from the predictive bill in bits. A model can have the higher posterior probability and still lose on a short held-out block. Replaying the same blocks helps compare allocations, while repeated tuning on those blocks would eventually make them part of the training process. ## Memory sets a scale for subtlety Give an observer a rolling notebook containing only the latest $W$ trials. Each new trial arrives; the oldest leaves. We recompute both models from that retained record with their priors counted once. This is the gold observer’s explicit memory rule. Its typical accumulated advantage is of order $WD$, while the flexible model’s complexity cost is of order $\tfrac12\log W$. A useful detectability scale is therefore $$ D_{\mathrm{scale}}\approx\frac{\log W}{2W}. $$ For an exponential discount rule, old trials instead carry weights $r^k$, where $0