This post is about the fundamental link between thermodynamics and computations. This is not on the practical level, about the heat dissipation of circuitry - though of course that is a very important concern when building computer compontents. This is about a fundamental and unavoidable mathematical link between the very concept of computation and our understanding of thermodynamics.

There is a profound difference between how the real world behaves, and how computers simulate processes. This was identified in a keynote speech in 1981 by Richard Feynman, making one simple observation:


Physics is reversible, but a classical computer is irreversible
Richard Feynmann

A process is physically reversible if the initial state can be recovered from the initial state without changing the system. That is so say, if there is no preferred arrow of time.

This is a concept introduced by Arthur Eddington, an astronomer and mathematician. Eddington was also a Quaker, and declared himself a pacifist during World War 1. His religious beliefs were expressed in his writings on the philosophy of science in his work Science and the unseen world, published 1929. Eddington wrote that the unseen world could not be discovered from science alone but must also be sought through the understanding of spiritual reality.

The arrow of time is itself a philosophical concept as much as it is a mathematical one.

Consider the dynamics of a collection of j particles with mass m governed by Newtonian equations of motion

\[m\frac{d^2}{dt^2}q_{j}(t) = F_{j}(q_{1}(t),...,q_{N}(t))\]


This equation is invariant under a time reversal. If we substitute t for -t, this equation remains the same.

If we were to use a very tiny video recorder to record the motion of these particles and then play it back, we would have no idea whether the video was being played forward or in reverse. There is no experiment we could do or calculation we could perform to deduce the arrow of time from the video recording of the motions of these particles.

However, let's now consider the case of an expanding gas. At the microscopic level, the motion of the atoms of this gas satisfies the equation above. However, if we were to zoom out and film the expanding gas it would be obvious whether the video recording were being played forwards or backwards. When being played forwards, we see the gas expanding. When being played backwards, it would appear the gas is spontaneously condensing.

The physical properties of low density gases were described by Ludwig Boltzmann, Austrian physicist and philosopher. Boltzmann developed his theory of gases on the assumption that matter was made of atoms and molecules, a theory that was heavily disputed at the end of the 19th century and dismissed by German philosophers such as Mach and Ostwald. Boltzmann managed to develop a model that would satisfy both camps, discussing atoms as 'models' of reality that the anti-atomists could consider as a useful - though unrealistic - mathematical fabrication. A similar discussion would arise a century later over the concept of Feynmann diagrams, outlining the interactions of subatomic particles.

Bolztmann introduced the distribution function \(f(\vec{r},\vec{v},t)\) to count the number of molecules within the volume element \(d^3r\) at a position \(\vec{r}\) and the velocity volume element \(d^3v\) at \(\vec{v}\) at a time t. This gives us the Bolztmann equation

\[\frac{d}{dt}f(t) = -\vec{v} \cdot \Delta_{r}f(t) + Q(f(t),f(t))\]


The first term in this equation describes the free motion of particles, while the second term describes the binary collisions between pairs of particles.

In this equation, if we substitute t for -t, the equation does not remain unchanged. As we also have to change the sign of the velocity v, the second term ends up with a negative sign.

This is our first indication of an irreversible process arising entirely out of reversible fundamental physics

The Excorcism of Maxwell's Demon

This concept of irreversibility is inseparable from the Second Law of Thermodynamics, which may well be the only fundamental law of physics that exhibits an intrinsic arrow of time. Historically, this law has been stated in many different but equivalent ways, combining heat and work to the conceptual meaning of time. The variable entropy was introduced to quantify this relation.

One statement of the Second Law of Thermodynamics is it is never possible for a cyclic process to convert heat entirely into work, so that in a closed system the overall entropy must always increase.

The possibility of violating this law is encapsulated by Maxwell's Demon thought experiment. This formulation shows how thermodynamic irreversibility is linked to the act of measurement and introduces the link between entropy and information.

The image above shows a single molecule Slizard engine.. A Slizard engine is the simplest possible steam engine, consisting of a cylinder with a single molecule inside it. The cylinder is is contact with a heat reservoir to maintain constant thermal conditions. There is, however, a demon outside this cylinder that controls a partition that can either open or close the partition at will.

The phase space representing what the demon 'knows' about the position of the molecule. This phase space has three states:

  • S meaning the demon knows nothing
  • L if the demon knows the particle is at the left side
  • R if the demon knows it is on the right side.

The engine goes through six steps in the procedure:

  1. The molecule is in the thermal path and moves randomly throughout this box
  2. The demon inserts a diving wall into the box to hold the molecule on the left side or the right side
  3. The demon carries out a reversible measurement to determine whether the molecule is on the left or the right
  4. The demon inserts a piston the side not containing the molecule and uses this to extract work from the molecule bouncing against the partition.
    The amount of work extracted can be calculated using the ideal gas equation \(pV = NkT\) where in this case N=1. The maximum work extracted is \(W = kT ln(2)\)
  5. The piston has moved to fill the box again
  6. The system is reset and both the demon and the box are at the same state as when it started

The molecule is always in thermal contact with the reservoir so the kinetic energy of the molecule is always \(\frac{3}{2} kT\). This process makes it appear that the demon can extract heat from the closed system and convert it into userful work. The demon could repeat this process cyclically, in clear violation of the Second Law of Thermodynamics.

The only way to avoid this violation is if the acquisition of knowledge causes a corresponding entropy cost.

But where does this cost occur?

The entropy cost occurs from steps 5 to 6, when the demon resets its mind. It is this process of resetting that compresses the phase state. This phase state compression corresponds to an inavoidable entropy increase

\[E_{l} = kT ln(2)\]

This limit is the Landauer limit, first discovered in 1961 by Rolf Landauer of IBM. Landauer's principle states that any logically irreversible operation that manipulates information, such as erasing information (or resetting the demon's mind), requires an unavoidable entropy increase. Consequently, the corresponding amount of energy is dissipated as heat.

This is approximately \(3 \times 10^{-21}\) Joules at room temperature. A 100 MHz processor with 100,000 gates erasing one bit at every cycle will dissipate about 28 nanowatts, due to the Landauer limit

In 2012, physicists from the Ecole Normale Supérieure de Lyon, University of Augsburg and the University of Kaiserslautern published an article in Nature describing that they have indeed measured the minute amount of energy dissipated when an individual bit is increased

In reality, this processor dissipates approximately 100 million times this amount, so this limit may seem unimportant. However, this limit is a fundamental consequence of the Second Law of Thermodynamics, so cannot be avoided by engineering current reversible architechture.