How words emerge without spaces

anicecream

You might read an ice cream. In a cosmetics shop, a nice cream becomes plausible too. The letters have stayed put. The spaces have moved, carrying a different world with them.

Now imagine meeting a language in which you recognize none of the words. Familiar English no longer does the work for you. You have a river of signs and a notebook. What could tell you where one useful piece ends and another begins?

One clue comes from prediction. Inside a recurring piece, the next sign often has few choices. At its edge, possibilities open out. Read enough examples and the river begins to reveal joints.

That is our first expedition. Then we will ask a stranger question: when two divisions both make sense, why insist that the river have only one set of banks?

Put the spaces back

The experiment below uses a tiny invented language. Its words were chosen so that repeated patterns are easy to inspect. You have three ribbons to decipher and a separate stream from which to learn.

Try placing spaces before asking for clues. Then open Inspect clues, read more letters, and tap the gaps. Look for a contrast: a place with one repeated continuation beside a place with several.

A little laboratory for discovering units

Find the spaces in an unfamiliar language

Start the puzzle again

These invented words have lost their spaces. Try placing them, then inspect the patterns in a separate stream of the same language. The clue finder sees letters only.

Ribbon 1Ribbon 2Ribbon 3

1. Place spaces 2. Inspect clues

Tap a letter to add or remove a space after it. Teal marks your spaces. Tab and Enter work too. Scroll the ribbon on a small screen.

m · i · p · l · u · m · o · d · a · x · k · e · f · l · u · m · o

Your readingmiplumodaxkeflumo

3. Reveal the original spaces Clear my spaces

Open the word-boundary experiment in the playground.

The saved spaces belong to the generator’s answer key. The clue finder receives only the unspaced training stream. This separation lets us ask two different questions: What patterns can it see? and How well do its proposed gaps match the units that generated this particular world?

The training stream and the puzzle are separate. Seeing a word in training helps with a new arrangement of the same words. That is the small discovery at the heart of the game: a reusable piece travels beyond the first place you met it.

A boundary is a place where prediction opens up

Suppose the letters da have always been followed by x. After da, your notebook has a simple forecast.

After ax, the next letter could begin any of several words. You might have seen d, m, l, and k. The possibilities branch.

We can count that branching. Let $c$ denote a context, such as the last two letters. Let $n_c(a)$ be the number of times letter $a$ followed it in the training stream, and let $N_c$ be the total number of those examples. When the context has appeared at least once, its empirical share is

$$p(a\mid c)=\frac{n_c(a)}{N_c}.$$

Here $p(a\mid c)$ reads “the share for letter $a$, given context $c$.” This particular demo uses the observed shares directly. With only a few examples, unseen continuations can still surprise it.

Shannon entropy measures how spread out those shares are:

$$H(c)=-\sum _{a}p(a\mid c){\log }_{2}p(a\mid c).$$

The sum runs over possible letters. The base-two logarithm makes the unit a bit. A letter with share zero contributes zero to the sum.

Now read backward too. At a candidate gap, ask what usually follows the letters on its left, and what usually precedes the letters on its right. The demo adds those two entropies. A tall gold bar means the neighborhood offers many alternatives in one or both directions.

The two count panels let you inspect the reason. A low bar supported by twenty repetitions says something different from a low bar supported by a single occurrence. An empty table says there was no matching example at all.

Our suggested spaces use a deliberately simple rule: select a positive local peak, with two letters of context and at least three examples on each side. That is a convenient way to turn the picture into a guess. The gold height remains an entropy score, not a probability that a space belongs there.

Try a prediction. Return to the first ribbon and inspect the very first gap. There is only one letter of left context. That shorter view can mix different situations together. Does giving the learner more text repair the missing context, or only give it a better count of the mixed situations?

More examples improve the count. They cannot supply a letter that lies beyond the edge of the ribbon. This is why the suggestion rule leaves those edge gaps alone.

A repeated chunk becomes a question you can ask

Finding a promising gap is the beginning. Next comes a new kind of unit.

Once dax is a candidate chunk, a learner can ask whether it appears after mip, before lumo, or in several different places. The whole chunk gets its own record of occurrences. Its letters remain available too.

In Epistemic Physics, a useful way to describe this move is pooling trials. Several occurrences contribute to a shared category, letting experience from one place inform a prediction in another. Their lineage still matters: copying one passage ten times should not create ten independent observations. The category pays its way when that sharing helps.

Pooling also has a cost. If two different things are given the same name, the combined record can hide the difference. A good learner needs both moves: gather recurring pieces into a category, and split a category when its surroundings reveal distinct behaviors.

Imagine discovering red and blue in many of the same positions, then noticing that blue also appears in descriptions of someone’s mood. The shared pattern suggests a color category; the exceptional context suggests another use. Categories become a growing set of questions, each aimed at a regularity in experience.

That opens a loop:

1. Letter patterns suggest candidate chunks.
2. Recurring chunks support categories and predictions about their neighbors.
3. Those predictions help decide where the chunks belong in a new stream.

The laboratory above implements the first step. A full word model can carry the other steps, including uncertainty about the divisions themselves. Bayesian word-segmentation research studies how assumptions about words and their context change the units that a learner discovers. Goldwater, Griffiths, and Johnson (2009)

Keep the paths, then add them

Return to our opening puzzle:

A segmentation is a complete division of the string into chunks. The two paths disagree about the first boundary, while agreeing about the boundary before cream.

Suppose an illustrative two-path model gives path A a posterior share of $0.6$ and path B a share of $0.4$. “Posterior” means after taking the observed text and context into account. These are example weights, not measured frequencies from the word game.

Then the probability of a gap after the first a is $0.4$, while the probability of a gap after an is $0.6$. Those two gaps are linked: in this model, exactly one of them appears. Flipping an independent coin at each gap would invent extra readings that neither path contains.

This gives us a reason to keep a belief over whole paths. We can obtain a gap’s probability by adding the shares of all paths that pass through it.

Let $x$ denote the visible string, and $s$ a segmentation. A generative model assigns a joint probability $P(x,s)$ to the string together with that path. Since the path is hidden, the string’s probability is

$$P(x)=\sum _{s}P(x,s).$$

Its posterior path share is

$$P(s\mid x)=\frac{P(x,s)}{\sum _{s^{\prime }}P(x,s^{\prime })}.$$

The symbol $s^{\prime }$ simply runs over the possible paths again. Context can be included in every probability; we have left it out of the notation to keep the map readable.

Here is a precise reward for keeping alternatives. If $s_{\ast }$ is the most probable path, its joint probability is one term in the sum, so

$$P(x)\ge P(x,s_{\ast }).$$

A probability $p$ corresponds to a logarithmic score of $-{\log }_{2}p$ bits, with lower scores better. The difference between scoring the best path jointly with the string and scoring the string summed over paths is

For our $0.6/0.4$ example, that difference is about $0.737$ bits for this string. When the leading path has almost all the share, the difference approaches zero.

The comparison holds inside the same joint model: one score keeps a single path’s mass, and the other sums all compatible path masses. It does not by itself establish that every soft model predicts new language better than every model that selects a segmentation. Predictive performance still depends on the model and its treatment of context.

There is also a practical surprise. For models whose path score factors into local chunk contributions, dynamic programming can add the paths by reusing sums for shared prefixes. We do not have to list every complete reading. A branching river can be navigated by remembering what has already flowed to each junction.

A word and its parts can both belong

Now consider walking.

It is useful as a whole word. It is also useful as walk plus ing. The ending connects it with talking, looking, and many other words; the stem connects it with walk, walked, and walks.

A morpheme is a unit of grammatical function or meaning, such as a stem or an ending. Word boundaries and morpheme boundaries answer different questions. A learner that discovers walk | ing may have found a useful internal joint even when a written-space answer key counts that cut as an error.

We can keep records at several levels. A familiar whole word can draw on its own occurrences. A rare form can also draw on related stems and endings. How much to pool across those relationships becomes another learnable question.

This separates two kinds of open question. We might need more examples to choose between two readings at the same level. Or we might already have good reasons to preserve several levels because they serve different predictions.

The distinction matters when checking a result. A hidden-space key can evaluate recovery of those spaces. It cannot, on its own, judge every useful unit that a learner might discover inside or across them.

The same expedition in a picture

An image also arrives without labels saying “this patch is a window” or “this edge belongs to a leaf.” We can ask where neighboring measurements predict one another well, where that relation changes, and which pieces recur elsewhere.

The correspondence is easiest to see side by side:

The geometry changes. Letters give us an ordered line; an image offers several directions and overlapping regions. The simple entropy rule in the word game would need a new model for that setting. But the guiding question travels well: what should count as one thing, for the prediction we want to make?

The idea of finding linguistic units through their changing continuations has a long history, including Zellig Harris’s 1955 investigation of phoneme and morpheme boundaries. EP invites us to carry that discovery problem across domains: let patterns suggest units, let units share trials, and let their usefulness be questioned again when the world changes. Harris (1955)

Try the first ribbon once more. This time, pick a gap where the clue is convincing and explain it using the two count panels. Then pick a gap where more examples would help, and one where a richer context would help. They are three different requests you can make of a learner.