The short version. In physics, emergence means new collective behaviour at scale, and a phase transition is a sharp change in that behaviour at a critical point. Some of the “sudden” abilities reported in large language models turn out to be smooth improvements seen through an all-or-nothing yardstick. Others appear to track a real threshold in training loss. Physics is a good source of questions here, and a poor source of slogans.
What physicists mean by emergence
In 1972, Philip Anderson published a short essay in Science called “More is different” [1]. His argument was that each level of organization in nature has its own laws, consistent with the level below but not obvious from it. Knowing everything about a single water molecule does not hand you the boiling point of water. Collective behaviour of this kind, which appears only when many parts interact, is what physicists mean by emergence.
Phase transitions are the most dramatic examples. The textbook case is the Ising model of a magnet: spins on a lattice, each pointing up or down, with neighbours preferring to align. In 1944, Lars Onsager solved the two-dimensional version exactly and showed that it has a sharp order-disorder transition at a critical temperature [2]. Above that temperature, the spins point every which way and the magnet has no net magnetization. Below it, a net magnetization appears, although nothing about the individual spins has changed. The magnetization is an order parameter: a quantity that is zero in one phase and nonzero in the other.
One technical point matters a great deal for what follows. In 1952, C. N. Yang and T. D. Lee showed, in effect, that a truly sharp transition, a genuine mathematical kink in the free energy, can appear only in the limit of an infinitely large system [3]. Every finite system changes smoothly, even if very steeply. So when a physicist says a finite system shows a phase transition, the real claim is that a smooth crossover sharpens, in a systematic and predictable way, as the system grows. That is a testable statement, not an impression from a plot.
The claim about language models
In 2022, Jason Wei and colleagues defined an ability of a large language model (LLM) as emergent if it is absent in smaller models, present in larger ones, and cannot be predicted by extrapolating the performance of the smaller models [4]. They documented examples across few-shot prompted tasks and prompting strategies such as chain-of-thought, in which a model is asked to write out its intermediate steps [4]. Plots of accuracy against model scale showed long flat stretches near chance and then steep climbs.
Meanwhile, the training loss of language models, their average error in predicting the next token, was known to fall smoothly as a power law in model size, dataset size and compute, with some trends spanning more than seven orders of magnitude [5]. A token is a word or a fragment of a word. So the field faced a puzzle that any physicist would recognize: a smooth microscopic quantity alongside an apparently abrupt macroscopic one.
How a yardstick can manufacture a jump
In 2023, Rylan Schaeffer, Brando Miranda and Sanmi Koyejo offered a simple resolution for many of the cases [6]. They found that two metrics, multiple-choice grade and exact string match, account for more than 92% of the claimed emergent abilities in BIG-Bench, a large collaborative benchmark of language tasks [6]. Exact match gives credit only when every token of the answer is right. If a model gets each token right with probability p, and the answer is L tokens long, then under an independence approximation the chance of a fully correct answer is
P(correct) ≈ p^L
Here p is the per-token accuracy, a number between 0 and 1, and L is the number of tokens the answer must contain. The exponent does the damage. For a 20-token answer, a per-token accuracy of 0.90 gives a fully correct answer about 12% of the time. Raise p to 0.95 and it is about 36%. At 0.99 it is about 82%. A steady, unremarkable improvement in p looks like a sudden leap in P. Schaeffer and colleagues showed that apparent emergence often fades when the same model outputs are scored with continuous metrics, and that they could produce similar-looking “emergence” in vision models simply by choosing the metric [6].
They were careful about what they did not claim. In their words, “nothing in this paper should be interpreted as claiming that large language models cannot display emergent abilities” [6]. Their point is narrower and more useful: before calling a curve a transition, check the yardstick.
A threshold that survives a better yardstick
The story did not end there. In 2024, Zhengxiao Du, Aohan Zeng, Yuxiao Dong and Jie Tang put pre-training loss, rather than model size, on the horizontal axis [7]. They reported that on certain tasks, performance stays near random guessing until the loss falls below a specific threshold and then rises, and that this holds even with continuous metrics [7]. They also found that models with the same pre-training loss performed similarly on downstream tasks, whatever their size [7].
This is closer to the shape of a physical transition. There is a control parameter, the loss, playing a role somewhat like temperature. There is a quantity that stays flat and then rises. But a physicist would want more before using the word. Is there an order parameter defined independently of any one benchmark? Does the rise sharpen in a predictable way as models grow, as the Yang and Lee result says a true transition must? Do different tasks share the same behaviour near the threshold, the property physicists call universality? As far as I know, none of these has been established for language models.
| Concept | In a magnet | In a language model |
|---|---|---|
| Control parameter | Temperature | Scale, or pre-training loss |
| Order parameter | Magnetization | Task score (contested) |
| Sharpness | Grows with system size | Depends on the metric |
| Universality | Shared exponents | Not established |
A lesson from the brain
My own research has looked at phase structure in brain dynamics, which sits next to a long-running debate about whether neural activity operates near a critical point. That debate gained momentum in 2003, when John Beggs and Dietmar Plenz reported cascades of activity in cortical tissue, which they called neuronal avalanches, whose sizes followed power laws [8]. The lesson I take from that literature transfers directly to AI: a power law, or a steep curve, is a hypothesis to be tested against alternatives, not a conclusion to be announced.
Established, plausible, speculative
Here is where I would draw the lines. Established: language-model loss improves smoothly with scale [5], and the choice of metric can create or remove an apparent jump [6]. Plausible, with evidence: some abilities switch on once pre-training loss crosses a threshold, whatever the metric [7]. Speculative: that training passes through genuine phase transitions with critical exponents and universality classes, as magnets do. I find that last idea attractive as a research program, and I would not yet build a forecast on it.
For decision-makers the practical advice is plain. When someone reports that a capability “emerged”, ask three questions. What metric was used, and does the jump survive a continuous one? How many model sizes or training checkpoints sit on the curve? Was the threshold predicted in advance, or found afterwards? Physics earned its confidence in phase transitions by predicting where they occur and how they sharpen with size. Claims about AI deserve the same test.
References
- P. W. Anderson, “More is different,” Science, vol. 177, no. 4047, pp. 393-396, 1972.
- L. Onsager, “Crystal statistics. I. A two-dimensional model with an order-disorder transition,” Physical Review, vol. 65, no. 3-4, pp. 117-149, 1944.
- C. N. Yang and T. D. Lee, “Statistical theory of equations of state and phase transitions. I. Theory of condensation,” Physical Review, vol. 87, no. 3, pp. 404-409, 1952.
- J. Wei et al., “Emergent abilities of large language models,” Transactions on Machine Learning Research, 2022.
- J. Kaplan et al., “Scaling laws for neural language models,” arXiv:2001.08361, 2020.
- R. Schaeffer, B. Miranda and S. Koyejo, “Are emergent abilities of large language models a mirage?” in Advances in Neural Information Processing Systems 36 (NeurIPS), 2023.
- Z. Du, A. Zeng, Y. Dong and J. Tang, “Understanding emergent abilities of language models from the loss perspective,” in Advances in Neural Information Processing Systems 37 (NeurIPS), 2024.
- J. M. Beggs and D. Plenz, “Neuronal avalanches in neocortical circuits,” The Journal of Neuroscience, vol. 23, no. 35, pp. 11167-11177, 2003.