When a channel carries mass

A spacecraft sends a single bit: 0 means “keep exploring”; 1 means “come home.” Between transmitter and receiver, the message passes through a cloud of interference.

One radio occasionally flips the bit. Another leaves surviving bits untouched and prints ? whenever a transmission disappears. Which radio would you choose?

We can put their messages inside the same light-cone geometry used by Epistemic Physics. Then a familiar mass becomes channel overlap, a Lorentz factor becomes an error exponent, and forgetting an output label becomes addition of vectors. The geometry gives us a way to ask what a receiver knows, what a summary preserves, and how many more transmissions might help.

Put a received message inside a cone

Call the transmitted bit $X$ and the observed output $Y$. A channel specifies a conditional probability $W(y\mid x)$ for every possible input $x$ and output $y$. Each input’s output probabilities sum to one.

Assume the spacecraft is equally likely to send 0 or 1. For each received symbol $y$, form a pair of likelihood weights:

$${\alpha }_y=W(y\mid 0),{\beta }_y=W(y\mid 1).$$

The pair asks two questions about the same message: how often would it appear if the input were 0, and how often if it were 1? Package it in epistemic coordinates:

They obey ${\tau }_y^2-x_y^2=m_y^2$. Every nonnegative likelihood pair therefore lands on or inside the future light cone.

Bayes’ rule supplies the operational dictionary. The output occurs with probability ${\tau }_y/2$, and the posterior probability of input 0 is

$$p_y=Pr(X=0\mid Y=y)=\frac{{\alpha }_y}{{\tau }_y}.$$

Normalize the cone vector by its height. Its velocity, normalized mass, and rapidity become

An output that determines the input has $v_y=\pm 1$ and $z_y=0$: its vector follows a cone edge. A completely ambiguous output has $p_y=1/2$, $v_y=0$, and $z_y=1$: it points straight up the middle.

Here the likelihood weights give the cone’s scale through output frequency. In a Beta model, the same algebra can instead describe support and opposition accumulated through trials, with their sum measuring concentration. This channel experiment follows likelihood states produced by a specified communication process.

Average mass has a name in coding theory

The Bhattacharyya parameter, written $Z(W)$, measures overlap between the two output distributions:

$$Z(W)=\sum _y\sqrt{W(y\mid 0)W(y\mid 1)}.$$

Substitute the cone mass and the probability of receiving each output:

Channel overlap is the average normalized mass of its received likelihood states. Outputs that never occur contribute zero and can be omitted. The channel quantity and its reliability role appear in Arıkan’s original channel-polarization paper, Section I-A; the displayed substitution turns it into a cone equation.

Now return to our radios. A binary symmetric channel, or BSC, flips either input with probability $ϵ$. On receiving 0, the likelihood pair is $(1-ϵ,ϵ)$. On receiving 1, the two entries switch places. The two vectors have opposite directions but the same normalized mass:

$$Z_{\mathrm{B}\mathrm{S}\mathrm{C}}=z=2\sqrt{ϵ(1-ϵ)}.$$

A binary erasure channel, or BEC, loses a transmission with probability $e$. A surviving 0 or 1 resolves the input exactly and contributes zero normalized mass. The question mark leaves equal odds and contributes one. Its average is simply

$$Z_{\mathrm{B}\mathrm{E}\mathrm{C}}=e.$$

That gives us a fair comparison knob: set both average masses to the same value. At $Z=0.6$, the flip channel has $ϵ=0.1$, while the erasure channel has $e=0.6$.

To compare how much they communicate, use binary entropy in bits:

Entropy measures average uncertainty. It is one bit for an unresolved fair choice and zero when the answer is known. The mutual information between input and output is the reduction in that uncertainty:

$$I(X;Y)=1-{\mathbb{E}}_Y[h_2(p_Y)].$$

For these symmetric channels, equally likely inputs achieve capacity, the greatest rate of reliable communication achievable with suitable codes over many independent uses. Their capacities are

At the shared mass $0.6$, they carry approximately $0.531$ and $0.400$ bits per use. Try predicting the shapes of the two capacity curves before moving the points.

A channel comparison

Same mass, different messages.

Reset the channels

One channel flips bits; the other marks lost bits with a question mark. Drag either colored point along its curve to give both channels the same average normalized mass, $Z$. Predict which carries more information.

Z = 0.0Z = 0.3Z = 0.6Z = 1.0

Flip channelErasure channelShared overlap Z

Drag or use arrow keys. Z = 0.60 for both channels. Inputs 0 and 1 are equally likely.

Sometimes wrong

Flip probability: 10.00%

v = −10+1

Each arrow has height τ = 1. Larger dots occur more often.

0 Occurs 50.00%
p(0 | output) = 0.90, z = 0.60

1 Occurs 50.00%
p(0 | output) = 0.10, z = 0.60

Average mass Z

0.600

Capacity

0.531 bits/use

Sometimes missing

Erasure probability: 60.00%

v = −10+1

Each arrow has height τ = 1. Larger dots occur more often.

0 Occurs 20.00%
p(0 | output) = 1.00, z = 0.00

1 Occurs 20.00%
p(0 | output) = 0.00, z = 0.00

? Occurs 60.00%
p(0 | output) = 0.50, z = 1.00

Average mass Z

0.600

Capacity

0.400 bits/use

The flip channel carries 0.131 extra bits per use at the same average mass. The erasure channel mixes perfectly readable outputs with completely unreadable ones.

Send the same bit again.

Choose independent transmissions. The flip decoder takes a majority vote. The erasure decoder succeeds whenever any copy arrives.

1 copy3 copies7 copies15 copies31 copies

Flip channel: exact error

0.27%

Erasure channel: exact error

1.40%

Shared upper bound

1.40%

Error exponent −ln Z

0.511 nats/use

$$P_{\mathrm{e}\mathrm{r}\mathrm{r}\mathrm{o}\mathrm{r}}\le \frac{1}{2}{Z}^n$$

For the flip channel, −ln Z = ln γ. The bound is exact for the erasure channel; the flip channel's error approaches the same exponential rate as the number of copies grows.

Open the channel comparison in the playground.

The average mass compresses a whole distribution of conditional states into one number. The flip channel always gives a partially readable answer. The erasure channel alternates between fully readable and fully unresolved answers. Entropy averages a different function of those states, so it can distinguish channels whose average masses agree.

For a BSC alone, eliminating the flip probability gives an explicit mass-to-capacity equation:

$$C_{\mathrm{B}\mathrm{S}\mathrm{C}}=1-h_2\left(\frac{1-\sqrt{1-z^2}}{2}\right).$$

For a general channel, retain the distribution of output states when you want more than one average can tell you.

A Lorentz factor measures the disappearance of errors

Suppose the spacecraft repeats the same unknown bit $n$ times through independent uses of the flip channel, with . A decoder compares the probabilities of the received sequence under input 0 and input 1. Here that amounts to a majority vote, with a fair tie-break when necessary.

There is a short route from mass to an error bound. For any received sequence, the decoder’s contribution to its average error is half the smaller of the two likelihoods. The smaller of two nonnegative numbers is at most their geometric mean. Summing over sequences, and factoring the independent uses, yields

$$P_{\mathrm{e}\mathrm{r}\mathrm{r}\mathrm{o}\mathrm{r}}(n)\le \frac{1}{2}Z(W{)}^n.$$

Every extra transmission multiplies this bound by the same normalized mass. Write the BSC’s Lorentz factor as $\gamma =1/z=\cosh \varphi$. The exponential rate becomes

$$-\ln z=\ln \gamma =\ln \cosh \varphi .$$

For this channel it is also the exact limiting error exponent:

$$-\underset{n\to \mathrm{\infty }}{lim}\frac{\ln P_{\mathrm{e}\mathrm{r}\mathrm{r}\mathrm{o}\mathrm{r}}(n)}{n}=\ln \gamma .$$

An error exponent describes how quickly the logarithm of error falls per additional transmission. The finite error can sit below the simple bound, as the demo shows.

The limiting equality follows from the Chernoff discrimination exponent. Its calculation minimizes

over $0\le s\le 1$. Symmetry and convexity place the minimum at $s=1/2$, giving $c(1/2)=z$. The hyperbolic potential $\ln \cosh \varphi$ has acquired a communication task: it measures the asymptotic rate at which repeated observations separate two possible inputs.

This also invites a geometric reading of persistence. A likelihood direction farther toward a cone edge provides more separation per independent observation. The remaining mass quantifies the overlap that the receiver still has to work through.

Forgetting a label adds its vector

Imagine a receiver with a tiny display. The radio produces four distinct output symbols, but the display has room for only two names. Which symbols should share a name?

Let $T=g(Y)$ be the retained label. The function $g$ assigns fine outputs $y$ to folders $t$. To obtain the chance of a folder, add the chances of its contents:

Thus forgetting distinctions adds cone vectors. The reverse triangle inequality in Physical Statistics tells us that the mass of their sum is at least the sum of their masses:

Apply that result within each folder and sum. The factor $1/2$ converts total mass back into channel overlap:

$$Z(W_{\mathrm{c}\mathrm{o}\mathrm{a}\mathrm{r}\mathrm{s}\mathrm{e}})\ge Z(W_{\mathrm{f}\mathrm{i}\mathrm{n}\mathrm{e}}).$$

This is a data-processing inequality: processing that discards output distinctions cannot reduce their overall overlap. The familiar collision geometry now describes a receiver losing access to distinctions between messages.

The equality condition is particularly useful. Positive cone vectors add without extra mass precisely when they share a direction, or rapidity. Their likelihood ratios are then equal, so they already give the same posterior about the input.

Try this puzzle with the four cards below. A and B give different chances of appearing, but both favor input 0 by the same odds. C and D point the other way. Find the merges that shrink the display while preserving its ability to answer the spacecraft’s question.

Geometry of a summary

Which labels can you forget?

Restore four labels

A receiver sees one of four output cards. Select cards, then replace their names with a shared folder label. Their likelihood vectors add. Can you throw away a name without losing any information about the input bit?

ASelected ✓ α = 0.54, β = 0.06 p(0 | A) = 0.90 φ = 1.099BSelected ✓ α = 0.36, β = 0.04 p(0 | B) = 0.90 φ = 1.099CSelect card α = 0.06, β = 0.54 p(0 | C) = 0.10 φ = -1.099DSelect card α = 0.04, β = 0.36 p(0 | D) = 0.10 φ = -1.099

Merge A+B

Each custom merge starts with the original four cards. α and β are output probabilities under inputs 0 and 1; φ is half their log ratio.

Merge equal directions Merge opposite directions Forget every label

A Occurs 30% Posterior p = 0.90 Mass m = 0.360

B Occurs 20% Posterior p = 0.90 Mass m = 0.240

C Occurs 30% Posterior p = 0.10 Mass m = 0.360

D Occurs 20% Posterior p = 0.10 Mass m = 0.240

Original output vectors (dashed)Merged output vectors

Channel overlap Z0.600 → 0.600

Information, bits per use0.531 → 0.531

Conditional entropy, bits0.469 → 0.469

All output names survive. Try merging A with B: compare their rapidities first.

$$Z_{\mathrm{c}\mathrm{o}\mathrm{a}\mathrm{r}\mathrm{s}\mathrm{e}}\ge Z_{\mathrm{f}\mathrm{i}\mathrm{n}\mathrm{e}}$$ $$I(X;T)\le I(X;Y)$$

The whole collection always sums to τ = 2, x = 0. What changes is which individual vectors the receiver can still distinguish. The plot rescales to fit the retained vectors.

Open the label-merging experiment in the playground.

In the initial channel, A has likelihoods $(0.54,0.06)$ and B has $(0.36,0.04)$. Both imply posterior probability $0.9$. Replacing A and B with “A or B” gives $(0.9,0.1)$ and the same posterior. Their different names carry no further information about $X$.

A and C instead have opposite likelihood directions. Merge them and the folder has weights $(0.6,0.6)$: it leaves equal odds. Combining different directions has manufactured ambiguity that the original labels would have resolved.

A summary $T$ is sufficient for predicting $X$ from $Y$ when knowing the full output cannot improve the posterior:

$$Pr(X\mid Y)=Pr(X\mid T).$$

For these positive likelihood states, sufficiency is exactly common rapidity within every merged folder. A useful geometric rule emerges: compress along rays before compressing across them.

Entropy supplies a second potential on the cone

Mass gives one way to measure the cost of merging. Shannon entropy gives another. Let $h(p)=h_2(p)\ln 2$ denote entropy in nats, the units produced by natural logarithms. Define an extensive entropy potential

$$\mathcal{S}(\tau ,x)=\tau h\left(\frac{1+x/\tau }{2}\right).$$

“Extensive” means that scaling a vector by a positive factor scales this potential by the same factor. For our equally likely inputs,

$$H(X\mid Y)=\frac{1}{2}\sum _y\mathcal{S}({\tau }_y,x_y)$$

in nats. The potential is concave and extensive, which together imply

$$\mathcal{S}(e_1+e_2)\ge \mathcal{S}(e_1)+\mathcal{S}(e_2).$$

Consequently, merging increases conditional entropy and decreases mutual information. The two geometric potentials run side by side:

There is one more connection hiding in the derivative. Inside the cone, direct differentiation gives

$$d\mathcal{S}=\ln (2\gamma )d\tau -\varphi dx.$$

Follow a small supporting or opposing tracelet increment, $(d\tau ,dx)=(dn,sdn)$, where $s=+1$ or $-1$. With $p_s=(1+sv)/2$ for the corresponding binary outcome,

$$\frac{d\mathcal{S}}{dn}=\ln (2\gamma )-s\varphi =-\ln p_s.$$

The surprisal of an outcome, $-\ln p_s$, is the marginal growth of this entropy potential along its null direction. A rare outcome has a steep slope. A finite trial follows the integral of that slope as the state changes.

The same cone now carries a mass potential for discrimination, an entropy potential for uncertainty, and null-direction derivatives for surprise. These relationships let us move between physical-looking geometry and concrete questions about messages.

What should a representation remember?

Every compression choice carries a question. In the four-card experiment, the question is which input bit was sent. Equal posterior odds make two labels interchangeable for that task. If the task changes to identifying the radio hardware that generated the labels, those discarded names might matter again.

This suggests an adventurous design principle for epistemic systems: choose a representation by the future distinctions it should preserve. A lineage might retain many separate tracelets; a task-specific summary might merge the ones that point along the same predictive ray. More aggressive compression could trade a measured information loss for a smaller representation.

That tradeoff is the subject of Tishby, Pereira, and Bialek’s information bottleneck method. The cone adds a visual question to the optimization: which directions are being identified, and what ambiguity appears when they meet?

To continue the coding journey, A decoder that computes in rapidity turns these likelihood states into parity-check messages and polar channels. For the information gained while learning the source itself, visit How much can one trial teach?.

Reading map

• Arıkan, Channel polarization: channel overlap, symmetric capacity, and the construction that turns ordinary channels into polarized ones.
• Chernoff, A measure of asymptotic efficiency for tests of a hypothesis: the limiting discrimination rate behind the error-exponent calculation.
• Tishby, Pereira, and Bialek, The information bottleneck method: compression organized around information relevant to another variable.
• Physical Statistics: the cone mass inequality and its collision interpretation.
• A coin, a magnet, and a learning machine: the binary entropy and hyperbolic potentials encountered along this route.