How much can one trial teach?

Three expedition notebooks predict a coin will land heads half the time.

The first belongs to someone who has barely met the coin. The second records a modest collection of trials. The third has watched it for ages. Tomorrow’s flip still catches all three off guard: heads and tails remain equally likely.

Would that flip teach each notebook the same amount?

The question sends us across a useful border. Predicting the next outcome and learning the mechanism that produces outcomes are different jobs. Epistemic Physics keeps a coordinate for each side of this distinction: the mean tells us where a belief points, while its total trial amount helps determine how much it can change.

We will find two equations waiting at that border. In the first, normalized mass cancels Fisher information, leaving a simple scale for learning from one trial. In the second, the Fisher length $\pi$ reappears in the extra bits needed to encode a coin whose bias we must learn.

A coin inside a notebook

Call the coin’s unknown heads rate $B$. Put a Beta distribution over it:

$$B\sim \mathrm{Beta}(\alpha ,\beta ).$$

The positive shapes $\alpha ,\beta$ include the prior and whatever supporting and opposing trial amounts the notebook has collected. Its mean and total amount are

$$\mu =\frac{\alpha }{\alpha +\beta },{\tau }_{\mathrm{E}}=\alpha +\beta .$$

A fresh outcome $S$ is 1 for heads and 0 for tails. Given $B=b$, heads has probability $b$. Averaging over the notebook’s possible rates gives $Pr(S=1)=\mu$.

We use natural logarithms first. Surprisal is $-\ln$ of an outcome’s predicted probability. Its average is entropy:

$$h(p)=-p\ln p-(1-p)\ln (1-p).$$

Natural logarithms measure information in nats. Divide by $\ln 2$ to obtain bits. At $p=1/2$, either outcome has one bit of surprisal and the outcome entropy is one bit.

Now imagine a friendly laboratory sprite whispers the exact value of $B$ before the flip. You still cannot generally predict the outcome perfectly. A coin with known rate $b$ retains entropy $h(b)$.

The average advantage of hearing the sprite’s answer is therefore

$$I(B;S)=h(\mu )-\mathbb{E}[h(B)].$$

This quantity is mutual information: how much learning one variable reveals about the other, on average. Read the equation in either direction. Knowing the rate helps predict the flip; seeing the flip helps learn the rate. The same joint distribution supplies both readings.

The three notebooks can now submit different answers:

The last notebook has nearly settled the rate while leaving the individual flip unpredictable. Balanced support and opposition can make a belief resistant to change even when its next answer remains a toss-up.

A trial's information budget

One outcome, two amounts of learning

Reset notebooks

Both notebooks predict the same chance of heads. A starts with total Beta amount 1; B can carry much more. Before revealing a trial, predict which notebook will learn more about the coin's hidden heads rate.

Shared predicted chance of heads 50%

50% heads75% heads90% heads

Notebook B's total amount: 100

Divide amount by 10 Multiply amount by 10

Each bar totals the entropy of the next outcome. Gold is the part learnable about the rate; sky is the average unpredictability that remains even if the rate were revealed. Bar scale: 0 to 1 bit.

Notebook A: amount 1

Beta(0.50, 0.50)

Next-outcome entropy

1.0000 bits

Expected learning about the rate

0.4427 bits

Variance of the rate

0.125000

Geometric approximation

0.3607 bits

Notebook B: amount 100

Beta(50.00, 50.00)

Next-outcome entropy

1.0000 bits

Expected learning about the rate

0.0072 bits

Variance of the rate

0.002475

Geometric approximation

0.0071 bits

$$I(B;S)=h(\mu )-\mathbb{E}[h(B)]$$ $$\frac{1}{2}J(\mu )\mathrm{Var}(B)=\frac{1}{2({\tau }_{\mathrm{E}}+1)}$$

The approximation is in nats before dividing by ln 2. It improves as both Beta shapes grow. Here both notebooks have velocity 0.00 and normalized mass 1.000; their total amounts differ.

Now open one outcome

Reveal heads Reveal tails Hide outcome

Try a 90% prediction, then reveal tails. Compare its surprisal with the posterior KL in each notebook. The gold values above are averages over both possible outcomes.

Revealing heads or tails compares one specified observation against each starting notebook. It does not accumulate a history. Changing an amount preserves the starting mean.

Open the trial-information experiment in the playground.

Surprise opens one door; learning measures the step

After heads, the notebook becomes $\mathrm{Beta}(\alpha +1,\beta )$. After tails, it becomes $\mathrm{Beta}(\alpha ,\beta +1)$. These are the familiar supporting and opposing trial updates.

To measure how far an update moves the whole probability distribution, use Kullback-Leibler divergence, abbreviated KL. For a posterior density $f_s(b)$ and starting density $f(b)$, define

$$G_s={\int }_0^1{f}_s(b)\ln \frac{f_s(b)}{f(b)}db.$$

The subscript $s$ identifies the observed outcome. $G_s$ is the realized information gain for that outcome, with the posterior on the left side of the density ratio.

Bayes’ rule makes the ratio beautifully simple. If the outcome is heads,

$$\frac{f_1(b)}{f(b)}=\frac{b}{\mu }.$$

The posterior favors rates that gave the observed heads greater probability. Averaging the logarithm of this ratio over possible heads and tails gives

$${\mathbb{E}}_S[G_S]=I(B;S).$$

So the gold quantity in the demo can be calculated two ways: subtract the entropy left after knowing the rate, or average the KL movement caused by each possible observation. The expected-information viewpoint is central to Lindley’s measure of what an experiment provides.

Try the demo at a predicted heads probability of $90\mathrm{\%}$. Tails has surprisal $-{\log }_2(0.1)\approx 3.322$ bits in both notebooks. Its posterior KL depends strongly on their trial amounts. An outcome can be a rare visitor without rearranging much furniture.

An exact Beta calculation

The Beta density’s normalizing integral is

$$\mathcal{B}(\alpha ,\beta )={\int }_0^1{b}^{\alpha -1}(1-b{)}^{\beta -1}db.$$

Differentiating a power brings down a logarithm. For example, differentiating the same integral with $\alpha$ replaced by $\alpha +1$ gives the average of $B\ln B$:

$$\mathbb{E}[B\ln B]=\mu [\psi (\alpha +1)-\psi ({\tau }_{\mathrm{E}}+1)].$$

Here $\psi (x)=d\ln \mathrm{\Gamma }(x)/dx$ is the digamma function, the derivative of the logarithm of the Gamma function. The Gamma function extends factorials; $\mathrm{\Gamma }(n+1)=n!$ for whole $n$. Digamma packages the logarithmic averages our calculation needs.

Repeating the step for $(1-B)\ln (1-B)$ yields

For one revealed heads, the exact information gain is

$$G_1=\psi (\alpha +1)-\psi ({\tau }_{\mathrm{E}}+1)-\ln \mu .$$

Replace $\alpha$ with $\beta$ and $\mu$ with $1-\mu$ for tails. These formulas let the demo separate an outcome’s surprisal from its actual posterior movement.

Watch mass cancel curvature

Now bring the notebook into the evidence cone. Its centered mean is the velocity-like coordinate

$$v_{\mathrm{E}}=2\mu -1.$$

Write normalized mass as $z$ and the Lorentz factor as ${\gamma }_{\mathrm{E}}$:

The Beta variance then becomes

$$\mathrm{Var}(B)=\frac{\mu (1-\mu )}{{\tau }_{\mathrm{E}}+1}=\frac{z^2}{4({\tau }_{\mathrm{E}}+1)}.$$

This measures the spread of possible rates. A second ingredient measures how strongly neighboring rates can be distinguished by a trial. For a Bernoulli outcome, its Fisher information is

$$J(\mu )=\frac{1}{\mu (1-\mu )}=4{\gamma }_{\mathrm{E}}^2.$$

The same quantity appears as negative entropy curvature: $h^{″}(\mu )=-J(\mu )$. Expand $h(B)$ around the mean. The average linear term vanishes because $\mathbb{E}[B-\mu ]=0$. Keeping the quadratic term gives

$$I(B;S)\approx \frac{1}{2}J(\mu )\mathrm{Var}(B).$$

Put the two cone formulas side by side:

Multiply them and the directional factors disappear:

$$\frac{1}{2}J(\mu )\mathrm{Var}(B)=\frac{1}{2({\tau }_{\mathrm{E}}+1)}.$$

The cancellation is exact. Using that product for mutual information is a large-concentration approximation, with the mean held inside $(0,1)$ or bounded away from its endpoints.

Moving toward an extreme mean narrows the Beta spread, while increasing how sharply a trial distinguishes nearby rates. At leading order, those changes balance. The total amount sets the remaining learning scale:

$$I(B;S)\sim \frac{1}{2{\tau }_{\mathrm{E}}}\text{nats}.$$

This invites an alternative clock for a learning system. A clock that counts trials advances once for every incoming tracelet. A clock that accumulates expected information slows as repeated observations settle the same question. New contexts or new mechanisms could make it hurry again. The displayed equations give us a concrete starting point for exploring such a clock.

A message arrives with no instruction manual

Imagine sending a long record of coin flips through a storage channel. An encoder assigns a short description to likely sequences and a longer description to unlikely ones. Its ideal logarithmic bill is $-{\log }_2$ of the sequence probability.

If both sender and receiver know the true heads rate $p$, an ordered sequence with $k$ heads in $n$ trials has probability

$$P_p^n(s^n)=p^k(1-p{)}^{n-k}.$$

Here $s^n$ means the whole ordered message. The expected bill is $nh(p)/\ln 2$ bits.

An encoder without $p$ can learn as it goes. Start with $\mathrm{Beta}(1/2,1/2)$, predict each next trial from the earlier ones, and multiply those predictions. The resulting sequence probability is

$$Q_n(s^n)=\frac{\mathcal{B}(k+1/2,n-k+1/2)}{\mathcal{B}(1/2,1/2)}.$$

There is no binomial coefficient here: we are pricing one ordered message. When averaging over all possible messages with $k$ heads, their multiplicity $(\genfrac{}{}{0ex}{}{n}{k})$ enters the average.

The sequential Beta code is explored in When the numbers refuse to fit. Our new question is its redundancy: the expected extra cost of learning compared with already knowing the true rate.

In nats, that cost is

$$R_n(p)={\mathbb{E}}_{S^n\sim P_p^n}[\ln \frac{P_p^n(S^n)}{Q_n(S^n)}].$$

This is another KL divergence, now between two distributions over complete messages. It is nonnegative, although a particular unusual message can receive a shorter bill from the learning encoder.

The geometric price of learning a code

What does an unknown rate add to the bill?

Reset code comparison

A coin sends a message. One encoder knows its fixed heads rate; a second learns using Beta(½, ½) predictions. Compare their expected ideal code lengths, then double the message. Predict the extra cost before pressing the button.

True heads rate 50%

Fair coin 10% heads Always tails

Halve message Double message 128 trials

Exact finite expectationFixed-interior approximation

Encoder knows the true rate

128.000 bits

Encoder learns the rate

131.104 bits

Expected extra cost

3.1044 bits

Extra cost per trial

0.02425 bits

$$R_n\approx \frac{1}{2}\ln \frac{n}{2\pi e}+\ln \pi$$

Doubling a long message adds about half a bit to the total learning cost. The cost per trial falls as the encoder learns. At this length the approximation gives 3.1044 bits.

Inspect one message instead of the average

Heads in the chosen message 64 of 128

Any ordering with these counts has the same bill under these models. The hindsight encoder chooses its rate after reading the message; its displayed bill omits the cost of telling the decoder that fitted rate.

Learning encoder

131.829 bits

Known-rate encoder

128.000 bits

Hindsight fit, rate 50.0%

128.000 bits

Regret versus hindsight

3.829 bits

For this message, learning minus knowing is 3.829 bits. An individual message can favor the learner; the expected extra cost is nonnegative. All bills are ideal values −log₂(probability), before any whole-bit or file-format overhead.

Open the learning-code-cost experiment in the playground.

Pi sends an invoice

Why did the learner begin with half a trial on each side?

Recall the Fisher geometry of the coin:

$$\sqrt{J(p)}dp=\frac{dp}{\sqrt{p(1-p)}}.$$

Substitute $p={\sin }^2(\theta /2)$. The right-hand side becomes $d\theta$, and the interval becomes . The entire family of coin rates has Fisher length

$$V_J={\int }_0^1\sqrt{J(p)}dp=\pi .$$

Uniform weight along this geometric length gives the Jeffreys density

$$w(p)=\frac{\sqrt{J(p)}}{\pi }=\frac{1}{\pi \sqrt{p(1-p)}}.$$

That is precisely $\mathrm{Beta}(1/2,1/2)$. You can walk along the probability circle in A little pi between belief and relativity.

For a regular one-parameter model, a smooth positive prior, and a fixed interior true parameter, the expected coding redundancy has the large-sample form

The term $o(1)$ tends to zero as $n$ grows. This is Clarke and Barron’s asymptotic formula, equation (1.4).

Insert the Jeffreys density. Its local Fisher factor cancels, leaving the total Fisher length:

$$R_n(p)=\frac{1}{2}\ln \frac{n}{2\pi e}+\ln \pi +o(1).$$

The same $\pi$ that measures the probability family’s length contributes a constant to its learning bill. Geometry has followed us into compression.

Try a fair coin at 128 trials in the demo, then keep doubling the message. The asymptotic difference is

$$R_{2n}(p)-R_n(p)⟶\frac{1}{2}\ln 2.$$

That is half a bit. An ever-longer message costs more in total, while its learning overhead per trial approaches zero.

Change the competitor, change the constant

There is another tempting encoder to race: one that reads the entire message first and selects the best-fitting rate $\hat{p}=k/n$.

Its bill for that same message is $nh(\hat{p})$ nats. This is a hindsight benchmark; a working code must also give the receiver enough information to reconstruct its fitted model.

The learning encoder’s excess over that benchmark is called regret:

$$\ln \frac{P_{\hat{p}}^n(s^n)}{Q_n(s^n)}.$$

Using the displayed Beta formula and Stirling’s large-argument approximation for $\mathrm{\Gamma }$, an empirical rate staying away from the endpoints gives

$$\text{regret}=\frac{1}{2}\ln \frac{n}{2\pi }+\ln \pi +o(1).$$

The $e$ has disappeared. Fitting the observed fluctuations buys an extra advantage of about half a nat on average compared with knowing the true rate. The demo lets you inspect one message and name the competitor explicitly.

Endpoint sources offer another experiment. Set the true rate to zero: only the all-tails message can occur. Its exact learning bill follows from the same Beta formula, while the fixed-interior approximation loses its correct constant. The finite calculation remains available throughout the slider’s range.

Return to our expedition notebooks. A new trial adds one unit of amount, yet its expected information can shrink like the inverse of that amount. A longer message offers more chances to learn, yet the accumulated price grows only logarithmically. Between those two scales sit the mass, curvature, and length of a remarkably familiar cone.

Reading map

• A belief is more than a number develops the distinction between a mean, trial amount, spread, and malleability.
• A coin, a magnet, and a learning machine connects entropy and KL to thermodynamics and learning.
• Lindley, On a measure of the information provided by an experiment develops expected information as a way to evaluate experiments.
• Clarke and Barron, Information-theoretic asymptotics of Bayes methods supplies the general asymptotic redundancy formula.