Energy Is the Real Budget for AI: What Physics Says About the Cost of Computation

Bottom line. Computation has a physical minimum energy cost, and today’s hardware operates far above it. Within that gap, the largest cost is usually moving data rather than calculating with it. For AI systems, where electricity and cooling are becoming the binding constraint, designing around data movement and numerical precision is a first-order decision, not a tuning detail.

Information has a physical cost

In 1961, Rolf Landauer showed that erasing one bit of information must release a minimum amount of heat, equal to kT ln 2 [1]. Here k is Boltzmann’s constant, T is the temperature in kelvin, and ln 2 is the natural logarithm of 2. At room temperature (300 kelvin) the minimum is about 2.9 × 10⁻²¹ joules per bit. The principle was confirmed experimentally in 2012 [2].

The number is tiny. Its importance is conceptual: computation is a physical process, governed by thermodynamics like any other. Energy is not an overhead on top of computing. It is part of what computing is.

How far above the limit we operate

Mark Horowitz’s widely cited figures for a 45-nanometre chip process give the energy of common operations [3]. One picojoule (pJ) is 10⁻¹² joules.

DRAM is dynamic random-access memory, the main memory outside the processor. Newer manufacturing processes have lowered every figure in the table. The main pattern, however, comes from distance and wiring: moving a value across a chip or off it means charging longer wires, and that costs more than switching a few transistors nearby.

Two observations follow. First, a 32-bit floating-point multiply uses about a billion times the Landauer energy for one bit (3.7 × 10⁻¹² divided by 2.9 × 10⁻²¹ is about 1.3 × 10⁹). Even allowing for the many bit-level operations inside one multiplication, practical hardware sits several orders of magnitude above the physical floor. Physics leaves plenty of room to improve.

Second, reading a value from off-chip memory costs about 170 times as much as multiplying it (640 divided by 3.7). In many AI workloads, the dominant energy cost is not arithmetic. It is moving model weights and intermediate results between memory and processor.

The automatic improvements have slowed

For decades, efficiency improved without anyone designing for it. Jonathan Koomey and colleagues found that the number of computations per unit of energy doubled roughly every 1.57 years over more than six decades up to 2009 [4]. Much of that came from Dennard scaling: as transistors shrank, their power density stayed roughly constant [5]. That scaling broke down in the mid-2000s, when leakage current stopped supply voltages from falling further. Since then, chip designs have been limited by how much heat they can remove [6].

The consequence is that efficiency gains now have to be designed in: through specialized hardware, lower numerical precision and architectures that move less data.

The system-level picture

At the scale of whole economies, this arithmetic becomes an energy question. The International Energy Agency (IEA) estimates that data centres consumed about 415 terawatt-hours (TWh) of electricity in 2024, about 1.5% of global electricity use. It projects around 945 TWh by 2030, with AI as the largest driver of growth [7]. For a single organization, the same constraint appears as a cloud bill, a power allocation at a facility or a cooling limit in a server room.

What this means for design

Reasoning from the physics produces design rules that hold regardless of platform:

1. Move computation to the data, not data to the computation. Every transfer across a chip, a board or a network costs energy. Filtering and summarizing data close to where it is produced reduces what must move.
2. Use the lowest precision the task allows. In the figures above, an 8-bit integer add costs about 30 times less than a 32-bit floating-point add. Large language models can run inference at 8-bit precision with little or no loss of accuracy [8].
3. Right-size the model. For a dense language model, the computation per generated token is roughly two floating-point operations per parameter [9]. A model ten times larger costs roughly ten times more per token. Use the large model where its accuracy changes the decision, and a smaller one where it does not.
4. Measure energy per useful result. Tokens per second and operations per second describe the machine. Energy per correct answer, or per decision supported, describes the value.
5. Treat power and cooling as design inputs. A facility’s power allocation is a fixed boundary condition. Design the workload to fit it, rather than discovering the limit after deployment.

Conclusion

Physics does not tell us which model to deploy. It tells us what every design will cost, and where the largest savings are. The gap between practical hardware and the physical limit is wide, so progress will continue. That progress will come from design choices that respect the physics, above all moving less data, rather than from waiting for the next chip generation.

References

1. R. Landauer, “Irreversibility and heat generation in the computing process,” IBM Journal of Research and Development, vol. 5, no. 3, pp. 183-191, 1961.
2. A. Bérut, A. Arakelyan, A. Petrosyan, S. Ciliberto, R. Dillenschneider, and E. Lutz, “Experimental verification of Landauer’s principle linking information and thermodynamics,” Nature, vol. 483, pp. 187-189, 2012.
3. M. Horowitz, “Computing’s energy problem (and what we can do about it),” in Proc. IEEE Int. Solid-State Circuits Conf. (ISSCC), 2014, pp. 10-14.
4. J. Koomey, S. Berard, M. Sanchez, and H. Wong, “Implications of historical trends in the electrical efficiency of computing,” IEEE Annals of the History of Computing, vol. 33, no. 3, pp. 46-54, 2011.
5. R. H. Dennard, F. H. Gaensslen, H.-N. Yu, V. L. Rideout, E. Bassous, and A. R. LeBlanc, “Design of ion-implanted MOSFET’s with very small physical dimensions,” IEEE Journal of Solid-State Circuits, vol. 9, no. 5, pp. 256-268, 1974.
6. H. Esmaeilzadeh, E. Blem, R. St. Amant, K. Sankaralingam, and D. Burger, “Dark silicon and the end of multicore scaling,” in Proc. 38th Annu. Int. Symp. Computer Architecture (ISCA), 2011.
7. International Energy Agency, “Energy and AI,” IEA, Paris, Apr. 2025.
8. T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer, “LLM.int8(): 8-bit matrix multiplication for transformers at scale,” in Advances in Neural Information Processing Systems (NeurIPS), 2022.
9. J. Kaplan et al., “Scaling laws for neural language models,” arXiv:2001.08361, 2020.