The latent heat of insight

Imagine watching a coin that keeps landing heads just a little too often. At first you shrug. Coins do that. After a while, the same small anomaly begins to look like a pattern. Eventually the explanation changes: perhaps this coin has a bias.

The next head now feels less surprising. Something has clicked, even though the coin has been doing the same thing all along.

That is the puzzle of insight: evidence arrives one trial at a time, yet explanations can seem to reorganize in a flash. Can we put a number on the relief of finally seeing the pattern? And could an observer with a short memory miss a pattern that an archive makes obvious?

We can build a small mathematical theater for these questions. Its actors are two competing models. Its currency is surprise. Its resemblance to melting ice is intriguing enough to explore carefully.

Two explanations at the same laboratory bench

Let each trial be a coin toss, independent of the others, with a constant but unknown heads probability $p$. These are our sampling assumptions. We compare:

There are two different uncertainties here. Inside $M_1$, we wonder which bias the coin has. Above both models, we wonder whether flexibility is needed at all. The probability of $M_1$ is a separate quantity from its estimated heads rate.

Suppose the ordered record contains $n=Y+N$ trials. The marginal likelihood, also called model evidence, is the probability that a model assigns to that record before fitting a single preferred parameter. For our two models it is exactly

$$Z_0=2^{-n},Z_1={\int }_0^1{p}^Y(1-p{)}^{N}dp=B(Y+1,N+1)=\frac{Y!N!}{(n+1)!}.$$

Here $B(a,b)$ is the Beta function, the integral that normalizes a Beta density. We are scoring an ordered sequence. Scoring only the counts would multiply both expressions by the same binomial coefficient, which cancels in their ratio.

That ratio, $Z_1/Z_0$, is the Bayes factor. It multiplies the prior odds of the models to produce their posterior odds. This is the standard model-comparison rule described by Kass and Raftery.

Write $\pi =P(M_1)$ for the flexible model’s prior probability and $w_n=P(M_1\mid D_n)$ for its probability after the record $D_n$. Then

$$\frac{w_n}{1-w_n}=\frac{\pi }{1-\pi }\frac{Z_1}{Z_0}.$$

The prior on a model and the prior inside a model play different roles. Choosing $\pi =0.1$ favors the fair explanation initially; it does not change the uniform distribution of biases considered inside $M_1$.

Why the right explanation can lose early

Start with equal model probabilities. One head gives both models likelihood $1/2$, so nothing happens to their relative standing. Add a tail. The fair model assigns probability $1/4$ to this two-trial record; the flexible model assigns $1/6$. Its posterior probability falls to

$$w_2=\frac{1/6}{1/6+1/4}=\frac{2}{5}.$$

The flexible model spreads its initial bets across many coins, including nearly all-heads and nearly all-tails coins. Those extreme possibilities predict the mixed record poorly. Averaging over them costs probability.

This is an Occam penalty: unused flexibility lowers a model’s marginal likelihood. It arises from averaging predictions, without an extra rule that complexity is bad. A second head instead would give $Z_1=1/3$ and $w_2=4/7$, so the early dip is a common pattern rather than an inevitable ritual.

Now imagine the actual generating rate is $p_{\ast }=0.6$. The asterisk marks the rate used by the world in our experiment. The fair model can never adjust, while the flexible model can gradually learn it. Flexibility starts collecting a return.

Before revealing trials below, predict which observer will sustain a preference for the flexible model. Try the 60% coin with 50-trial memory, then change the memory to 200. Finally try the fair coin: how often does a temporary streak look like an insight?

The evidence desk

Which explanation will the next packet favor?

Reset desk

Place an explanation on the desk before opening the next 50 trials. The archive and the smaller notebook read the same packet.

Packet 1 of 12 · 0 trials opened · generating heads rate 60%

Fair coinPredict 50% every time Flexible coinLearn the heads rate

Open the next packet

Full archive

50.0%

probability of flexible model

Last 50 trials

50.0%

probability of flexible model

Choose which model you expect to lead after the first packet.

Change the coin, memory, or model prior

Changing an assumption restarts the desk, so a verdict always belongs to the displayed experiment.

Generating heads probability

50% heads60% heads75% heads

Notebook capacity

20 trials50 trials200 trials

Prior probability of flexible model

50%10%

Full-archive costs: fair 0.00 nats, flexible 0.00 nats. Notebook costs: fair 0.00, flexible 0.00. These are negative log marginal likelihoods; priors apply separately. The flexible model uses Beta(1,1). A seeded constant-rate independent Bernoulli sequence makes replays comparable. Preference for an explanation is different from proof that it is true.

The teal and gold curves share every revealed trial. Their difference comes entirely from what they retain. Changing the model prior lets you compare a welcoming observer with a skeptical one without changing the coin.

The price of an explanation

Turn probabilities into additive costs by defining

$$F_j=-\log Z_j.$$

Logs are natural logarithms, so the unit is a nat; dividing by $\log 2$ converts nats to bits. This quantity is called Bayesian free energy. Lower $F_j$ means the model assigned more probability to the observed sequence.

Our exact formulas give $F_0=n\log 2$ and $F_1=-\log B(Y+1,N+1)$. A large-sample approximation reveals the competition. Put $\hat{p}=Y/n$, the observed heads fraction. Away from the endpoints 0 and 1, Stirling’s factorial approximation gives

$$F_1\approx nH(\hat{p})+\frac{1}{2}\log n-\frac{1}{2}\log (2\pi \hat{p}(1-\hat{p})),$$

where

$$H(p)=-p\log p-(1-p)\log (1-p)$$

is binary entropy, the average surprise when predictions match a coin’s actual rate.

The first term pays for the data. The logarithmic term pays for learning the bias. Because that second term grows slowly, a persistent predictive advantage can eventually outweigh it.

The fair coin’s excess cost at a fixed rate $p_{\ast }$ is

$$D=\mathrm{KL}(\mathrm{Bern}(p_{\ast })\Vert \mathrm{Bern}(1/2))=\log 2-H(p_{\ast }).$$

The notation $\mathrm{Bern}(p)$ means the binary distribution with heads probability $p$. The Kullback-Leibler divergence, abbreviated KL, measures an expected excess log loss when one distribution is used to predict data from another.

Replacing the observed frequency by its typical value $p_{\ast }$ and dropping constant terms gives a rough crossover equation,

$$nD\approx \frac{1}{2}\log n.$$

For $p_{\ast }=0.6$, $D\approx 0.02014$ nats per trial, and the large root is about 119 trials. This is a scale estimate. The omitted constant, prior odds, and random streaks all move the actual crossing. There is no universal 119th-trial revelation.

Where the heat analogy becomes useful

An exact identity connects the cumulative cost with incoming surprise:

$$F_j(D_n)-F_j(D_{n-1})=-\log P_j(y_n\mid D_{n-1}).$$

Here $y_n$ is the latest outcome. A model’s increase in cost is precisely how surprising it found that outcome. This identity uses a growing record; deleting old trials, as the gold notebook does, adds a separate change to the cost.

Suppose a predictor uses heads probability $q$. Its expected next-trial cost under the true rate $p_{\ast }$ is

$$L(p_{\ast },q)=-p_{\ast }\log q-(1-p_{\ast })\log (1-q)=H(p_{\ast })+\mathrm{KL}(p_{\ast }\Vert q).$$

This gives a tidy side-by-side comparison:

The asymptotic savings is exactly $D$. This is the mathematical heart of the “latent heat of insight” metaphor: an explanation that fits releases us from a recurring surprise cost. Earlier costs remain paid; the savings concerns future predictions.

For a 60% coin, the improvement is modest: about 0.02014 nats per trial. That small amount accumulates. An explanation can matter greatly over a long future even when the next prediction changes only a little.

A physical latent heat is energy transferred during a phase change. Our quantity has units of information per trial. Converting it into joules, brain activity, or a person’s reported relief would require a physical or psychological model. The identity itself tells us exactly what a prediction experiment should measure: the difference in expected log loss.

Does an explanation actually undergo a phase transition?

The two branches can cross. If we always choose whichever model has larger posterior probability, the chosen label changes abruptly at the crossing. That hard choice can look like a miniature change of phase.

The full Bayesian observer carries both models. Its marginal likelihood is their weighted sum,

$$Z=(1-\pi )Z_0+\pi Z_1,F=-\log ((1-\pi )e^{-F_0}+\pi e^{-F_1}).$$

This is a smooth, softened minimum of the prior-adjusted branches. Finite positive likelihoods give a smooth dependence on their relative scores. Trials arrive discretely, and a very diagnostic trial can cause a large update, but a branch crossing alone establishes no thermodynamic singularity or first-order transition.

That distinction makes the analogy more interesting. Apparent suddenness can arise at several levels: a steep posterior change, an action threshold, a conscious report, or a genuine collective instability. They need different explanations.

The model-averaged prediction illustrates the separation:

$$P(\text{next head}\mid D_n)=(1-w_n)\frac{1}{2}+w_n\frac{Y+1}{n+2}.$$

Even when the model probability swings strongly, the next-head prediction may move only between roughly 0.5 and 0.6. A large change in which story we tell can accompany a small change in what we predict next.

Put explanations to work with a shared budget

A preference between models becomes useful when it improves a decision. Spend a limited observation budget across three coins, then choose fair, flexible, or model-averaged predictions. Each coin has a sealed future block. Its outcomes are kept separate from the inspected training trials, so the final bill tests prediction beyond the evidence used to choose the recipe.

The explanation workshop

Spend six tokens on three hidden coins

Reopen the workshop

Three coins share six observation tokens. Each token opens four training trials. Allocate them, choose a forecasting recipe for each coin, then open the sealed future.

6 tokens left16 future trials await each coin

Coin A

0 heads · 0 tails observed

Unopened

Inspect coin A · 1 token Forecast recipeFair modelFlexible modelAverage both models

Forecast: 50.0% heads

Coin B

0 heads · 0 tails observed

Unopened

Inspect coin B · 1 token Forecast recipeFair modelFlexible modelAverage both models

Forecast: 50.0% heads

Coin C

0 heads · 0 tails observed

Unopened

Inspect coin C · 1 token Forecast recipeFair modelFlexible modelAverage both models

Forecast: 50.0% heads

Seal recipes and open the future

What the budget and the bill mean

The fair recipe predicts 50%. The flexible recipe uses a Beta(1,1) prior and the inspected counts. The mixture weights those recipes by model probabilities with equal model priors. Each recipe is frozen before its 16 held-out trials; the bill sums −log₂ of their assigned probabilities. Inspection tokens constrain access to training data and have no conversion into bits. All three coins have fixed generating rates; samples are repeatable seeded Bernoulli sequences. The held-out block is generated separately and stays fixed regardless of which training data you inspect. A model's posterior probability and a forecast's held-out score answer different questions.

Coin A: current probability of the flexible model 50.0%.

Coin B: current probability of the flexible model 50.0%.

Coin C: current probability of the flexible model 50.0%.

The token budget limits access to observations; it is separate from the predictive bill in bits. A model can have the higher posterior probability and still lose on a short held-out block. Replaying the same blocks helps compare allocations, while repeated tuning on those blocks would eventually make them part of the training process.

Memory sets a scale for subtlety

Give an observer a rolling notebook containing only the latest $W$ trials. Each new trial arrives; the oldest leaves. We recompute both models from that retained record with their priors counted once. This is the gold observer’s explicit memory rule.

Its typical accumulated advantage is of order $WD$, while the flexible model’s complexity cost is of order $\frac{1}{2}\log W$. A useful detectability scale is therefore

$$D_{\mathrm{scale}}\approx \frac{\log W}{2W}.$$

For an exponential discount rule, old trials instead carry weights $r^k$, where and $k$ is their age. Their limiting total weight is $1/(1-r)$. Writing $\lambda =1-r$ and using $W\approx 1/\lambda$ suggests

$$D_{\mathrm{scale}}\approx \frac{\lambda }{2}\log (1/\lambda ).$$

That substitution is a heuristic comparison between memory models. Fractional likelihood weights define another inference rule, and their total weight need not equal an effective sample size for every statistic.

Neither expression creates an absolute insight horizon. Even a short window can contain an unusually one-sided run and favor the flexible model. In fact, with an indefinitely running independent coin having , any particular finite run eventually occurs with probability one. The difficulty is sustaining a reliable preference amid repeated forgetting and fluctuation.

The practical idea survives beautifully. Archives can make subtle regularities visible across timescales that exceed an individual’s memory. Longitudinal studies and replay buffers extend the record over which a weak predictive advantage can accumulate. Copying the same testimony many times does not create that advantage: its lineage still points to the same underlying trial. Storage preserves evidence; duplication does not manufacture independence.

Supercooling, stubbornness, and the missing dynamics

What if the familiar explanation receives 90% prior probability? Then $\pi =0.1$, and the flexible model must earn a log Bayes factor greater than $\log 9$ before becoming more probable. Extra prior support delays a switch.

That resembles a paradigm persisting through anomalies, but physical supercooling involves metastability: a system remains trapped because its dynamics encounter a barrier. Static Bayesian comparison contains no such barrier. Reorder the same retained independent trials and its final answer remains unchanged.

To model hysteresis, where the route into a state affects the route out, we could add switching costs, different thresholds for adopting and abandoning an explanation, social reinforcement, or limited search. These are fertile extensions. Each supplies a concrete mechanism that can be varied and tested.

An “aha calorimetry” experiment could ask people for probabilistic predictions before and after a reported insight, then score later independent outcomes. The observed change in log loss would test the predictive benefit. Comparing that benefit with subjective surprise or physiological measurements would investigate the psychological bridge rather than assume it.

Beyond a single adjustable coin

For a regular model with $d$ identifiable parameters, the leading complexity term is often $\frac{d}{2}\log n$. Neural networks and latent-variable models can be singular: different parameters can represent the same distribution, and the usual local quadratic approximation can fail.

Watanabe’s singular learning theory replaces $d/2$ with a geometric quantity called the real log canonical threshold, or RLCT. Under the theorem’s conditions, the free energy has leading form

$$F_n=n{L}_n({\theta }_{\ast })+{\lambda }_{\mathrm{RLCT}}\log n+O_p(\log \log n).$$

Here $L_n$ is average observed log loss, ${\theta }_{\ast }$ is a best-fitting population parameter, and $O_p$ describes the stochastic remainder’s scale. The RLCT depends on the model, true distribution, and prior. It is distinct from the forgetting rate used above. This established theory provides a richer complexity coefficient; identifying it with physical specific heat requires additional structure.

We can imagine three competing explanations, each learning a different pattern and paying a different complexity cost. Their posterior weights might take turns leading. Whether a particular neural system actually passes through such stages is a question for a specified model and measurement.

Epistemic Physics adds a useful map of the two levels. Within the flexible coin model,

$$\alpha =Y+1,\beta =N+1,\tau =\alpha +\beta =n+2,x=\alpha -\beta =Y-N,$$ $$\mu =\frac{\alpha }{\tau },v=\frac{x}{\tau },m=2\sqrt{\alpha \beta },{\tau }^2-x^2=m^2.$$

Each independent trial extends the evidence lineage by a tracelet. The Beta parameters include the prior, and $\mu$ is the mean. Above that cone sits the separate model probability $w_n$. Carrying both requires a richer state than one Beta distribution: a mixture with a point mass at $p=1/2$ and a continuous Beta component.

That extra layer is where the adventure begins. A slow-moving estimate inside one explanation can feed a much larger change in which explanation organizes the world. The coin gives us a place to explore that possibility with every assumption, trial, and cost visible.

Further reading

• Kass and Raftery, Bayes Factors: model evidence, prior odds, and comparison of competing explanations.
• Watanabe, A Widely Applicable Bayesian Information Criterion: Bayesian free energy and complexity in regular and singular models.
• A coin, a magnet, and a learning machine: the site’s earlier journey through entropy and binary thermodynamic analogies.