From Magnets to Machine Learning: What Statistical Mechanics Gave AI

The short version. The 2024 Nobel Prize in Physics went to John Hopfield and Geoffrey Hinton for discoveries that enable machine learning with artificial neural networks. The award makes sense once you follow the lineage: a model of a magnet became a model of memory, then of learning, and eventually of generating images. Statistical mechanics supplied ideas that modern artificial intelligence (AI) still uses: energy, temperature and sampling.

A toy magnet with a long reach

Statistical mechanics is the branch of physics that explains how the behaviour of large collections, such as gases, liquids and magnets, follows from the behaviour of their parts. Its favourite toy is the Ising model. In 1925, Ernst Ising analyzed a row of tiny magnetic moments, called spins, each pointing up or down, with each spin preferring to align with its neighbours [1].

The central idea of the subject fits in one line. At temperature T, a system in equilibrium visits each configuration s with probability

P(s) = exp(-E(s) / T) / Z

Here E(s) is the energy of configuration s, T is the temperature measured in units of energy, and Z is the sum of exp(-E(s) / T) over every possible configuration, which makes the probabilities add up to one. This is the Boltzmann distribution. At low temperature, the system almost always sits in its lowest-energy configurations. At high temperature, every configuration becomes nearly equally likely. Each idea in the rest of this article is a variation on that line.

Memory as a landscape

In 1982, John Hopfield showed that a network of simple on-off units could work as a memory [2]. Give each pair of units a connection strength, define an energy from those connections exactly as for interacting spins, and let each unit change state whenever doing so lowers the energy. The network then rolls downhill on an energy landscape until it settles in a valley. To store a set of patterns, shape the connections so that each pattern sits at the bottom of its own valley. Present a fragment or a corrupted version, and the network slides into the nearest valley and recovers the whole. Hopfield called this content-addressable memory, and he stressed that its useful properties were collective ones, only weakly sensitive to the details of individual units [2].

Physics then answered an engineering question precisely. How many patterns can such a network hold? In 1985, Daniel Amit, Hanoch Gutfreund and Haim Sompolinsky used the mathematics of spin glasses, magnets with disordered and competing interactions, to show that retrieval works only while the number of stored patterns stays below about 0.14 times the number of units [3]. A network of 1,000 units holds roughly 140 patterns. Beyond that, the memory does not degrade gently. It fails, collapsing into a glassy state in which the stored patterns can no longer be retrieved. That is a phase transition in a memory, and it was derived, not guessed.

Adding temperature: the Boltzmann machine

Hopfield’s network always moves downhill, so it can settle in shallow valleys that match no stored memory [2], [3]. In 1985, David Ackley, Geoffrey Hinton and Terrence Sejnowski made the units random: each unit switches on with a probability that depends on its inputs and on a temperature, so the network as a whole samples configurations from a Boltzmann distribution [4]. They added hidden units, which correspond to no part of the data, and a learning rule that adjusts the connections until the network’s own equilibrium statistics match the statistics of the training data [4]. They called it the Boltzmann machine.

The price was speed. Learning required letting the network reach thermal equilibrium again and again [4]. In 2006, Hinton, Simon Osindero and Yee-Whye Teh described a fast, greedy way to train a network with many layers, one layer at a time, using a restricted form of the Boltzmann machine [5].

Generation as thermodynamics run backwards

The best-known descendants today are diffusion models for generating images. In 2015, Jascha Sohl-Dickstein and colleagues took their inspiration explicitly from non-equilibrium statistical physics [6]. Take data, such as images, and destroy their structure gradually by adding noise step by step, the way ink spreads in water. Then train a network to reverse each small step. Run the reversal from pure noise, and structure reappears. Jonathan Ho, Ajay Jain and Pieter Abbeel made the approach practical for high-quality images in 2020 [7]. In 2021, Yang Song and colleagues described these models with stochastic differential equations, whose reverse direction needs only the gradient of the logarithm of the data’s probability density, a quantity known as the score [8].

Here the Boltzmann line pays off again. If a probability has the form exp(-E / T) / Z, the score is minus the slope of the energy, divided by the temperature. In other words, it is a force. Generating a sample by following the score while adding a little noise is, mathematically, a particle drifting through an energy landscape while being jostled by heat, the process physicists call Langevin dynamics, which Song and colleagues use as one of their sampling steps [8].

Hopfield returns

The lineage has a recent twist. In 2021, Hubert Ramsauer and colleagues showed that a modern Hopfield network, with continuous states and a sharper energy function, has an update rule that is mathematically equivalent to the attention mechanism used in transformer networks [9]. The equivalence is a theorem. Whether the energy view explains why transformers work as well as they do is an open question, and I would not claim more than that.

The prize, and the lineage

On 8 October 2024, the Royal Swedish Academy of Sciences awarded the Nobel Prize in Physics to Hopfield, of Princeton University, and Hinton, of the University of Toronto, “for foundational discoveries and inventions that enable machine learning with artificial neural networks” [10], [11]. Some readers were surprised to see a physics prize go to work now associated with computer science. The table above is my answer. These methods did not merely borrow physical vocabulary. They used the mathematics of energy, temperature and equilibrium to derive results, such as the memory capacity, that could be checked.

What is established, and what is not

It is worth separating three kinds of claim. Established: the mathematical correspondences described here, from the Hopfield energy to the diffusion score. Under active research: whether statistical mechanics can explain why large networks generalize, that is, why they perform well on data they have never seen. Speculative: that physics will hand us design rules for the next generation of AI systems.

Much of my own work sits where statistical mechanics, network science and Markovian models meet. The part of this story I find most valuable in practice is the habit it teaches: prefer quantities you can compute and check, such as capacities, temperatures and transition points, to metaphors that only sound like physics.

References

1. E. Ising, “Beitrag zur Theorie des Ferromagnetismus,” Zeitschrift für Physik, vol. 31, pp. 253-258, 1925.
2. J. J. Hopfield, “Neural networks and physical systems with emergent collective computational abilities,” Proceedings of the National Academy of Sciences, vol. 79, no. 8, pp. 2554-2558, 1982.
3. D. J. Amit, H. Gutfreund and H. Sompolinsky, “Storing infinite numbers of patterns in a spin-glass model of neural networks,” Physical Review Letters, vol. 55, no. 14, pp. 1530-1533, 1985.
4. D. H. Ackley, G. E. Hinton and T. J. Sejnowski, “A learning algorithm for Boltzmann machines,” Cognitive Science, vol. 9, no. 1, pp. 147-169, 1985.
5. G. E. Hinton, S. Osindero and Y.-W. Teh, “A fast learning algorithm for deep belief nets,” Neural Computation, vol. 18, no. 7, pp. 1527-1554, 2006.
6. J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” in Proc. 32nd International Conference on Machine Learning (ICML), PMLR vol. 37, pp. 2256-2265, 2015.
7. J. Ho, A. Jain and P. Abbeel, “Denoising diffusion probabilistic models,” in Advances in Neural Information Processing Systems 33 (NeurIPS), 2020.
8. Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon and B. Poole, “Score-based generative modeling through stochastic differential equations,” in Proc. International Conference on Learning Representations (ICLR), 2021.
9. H. Ramsauer et al., “Hopfield networks is all you need,” in Proc. International Conference on Learning Representations (ICLR), 2021.
10. The Royal Swedish Academy of Sciences, “The Nobel Prize in Physics 2024,” press release, Oct. 8, 2024.
11. The Royal Swedish Academy of Sciences, “Scientific background to the Nobel Prize in Physics 2024: For foundational discoveries and inventions that enable machine learning with artificial neural networks,” Oct. 2024.