Two wrong numbers and a right difference

The uncomfortable fact about applied quantum chemistry is that the methods in daily use get total energies badly wrong and are trusted anyway. A calculation on a medium organic molecule can be tens of thousands of kilojoules a mole from the exact answer and be relied on to a few kilojoules for a reaction energy.

The defence is error cancellation, and it is correct. This essay measures it, on a pair of systems where both the approximate and the exact answers are available, and finds two things the defence leaves out.

The first is that the cancellation gets better as the approximation gets worse, and the answer gets less accurate at the same time.

The second is that cancellation is not a property of the method at all.

The pair #

Four sites with four electrons, against two separate two-site systems with two electrons each: the same number of electrons, the same number of orbitals, the same repulsion on every site. The difference between them is a reaction energy of the kind quantum chemistry is asked for constantly — a bond-forming step in which the connectivity changes and nothing else does.

Both sides can be diagonalised exactly, and both sides can be solved in a spin-paired mean field, which is what this collection’s Hartree–Fock analogue is.

The two numbers in that figure #

The cancellation is real and large. At the strongest repulsion here the mean field is wrong by 12.11 on one side and 12.49 on the other, and the difference between those errors is 0.38 — three per cent of either. Nothing in the method knows that the two calculations are related; the cancellation happens because the two systems have the same number of electrons in similar local environments, so the mean field misses similar amounts on both sides.

And the answer is useless anyway. The exact reaction energy at that repulsion is 0.09, and a residue of 0.38 is four times it. A calculation that reproduces 96.9 per cent of a 12-unit error and then reports a reaction energy wrong by a factor of five is doing exactly what the defence says it does, and is not usable.

The two statements are not in tension. They are two ratios with different denominators, and the argument for error cancellation quietly uses the first while the user needs the second.

The condition is not that the errors are similar. It is that what survives is small compared with the answer — and whether it is depends on how big the answer is, which is a fact about the chemistry rather than about the method.

Why the cancellation improves #

The rise from 83 to 97 per cent has a reason worth following, because it is the opposite of reassuring.

At weak repulsion the mean field is nearly right on both sides and the errors are small; the residue is a small difference between two small numbers, and its ratio to them is not especially favourable. At strong repulsion the mean field is catastrophically wrong on both sides — the paired solution’s energy rises without limit while the exact energy falls towards zero — and the two catastrophes are nearly identical, because they are both dominated by the same term: the double occupancy a spin-paired description cannot avoid.

So the cancellation improves because the error becomes more systematic, not because it becomes smaller. A systematic error cancels well and is still an error, and every quantity that does not happen to be a difference between two systems with the same systematic error inherits all of it.

Where that error comes from is worth two sentences here. The energy a spin-paired mean field misses grows linearly with the repulsion, with a slope that can be written down from the electron count before anything is diagonalised — so two systems with the same electron count have nearly the same slope, which is the whole of the cancellation above. And the quantity doing it is the double occupancy: the exact state reduces it as the repulsion rises and a spin-paired mean field cannot, so the error is the repulsion times the gap between those two curves and it is the same gap on both sides of the reaction.

The pair where nothing cancels #

The second finding is the one that makes the first useful, because it says when to expect the defence to hold.

Take one system — the four-site chain — and compare two of its own states: the spin-paired ground state and the state with two spins flipped. The singlet–triplet gap is a quantity of exactly the kind error cancellation is supposed to protect: same molecule, same geometry, same number of electrons, same basis, everything identical but the spin.

The errors are 12.11 for the singlet and 0.17 for the triplet at the strongest repulsion — a factor of seventy — and the reason is not subtle.

The triplet is nearly a single determinant. Parallel spins cannot occupy the same site, so most of what the repulsion would have done is already forbidden by the Pauli principle, and a determinant that gets the Pauli principle right gets most of the answer right.

The singlet is not. Its exact wavefunction at large repulsion is a superposition of many arrangements with the spins alternating, and no single determinant resembles it. That is the multireference case, and most of what is hard about correlation is about it.

So the two states are not the same kind of wavefunction, the errors have nothing in common, and the difference between them inherits the whole of the larger one.

The rule that comes out #

Putting the two halves together gives a statement that is more useful than “the errors cancel”, and it is a statement about pairs.

Errors cancel between two calculations to the extent that the two wavefunctions are the same kind of object. Same number of electrons, same number of pairs, similar local environments, similarly single-determinantal — those are the conditions, and every one of them is a statement about the two systems rather than about the method.

That is why the reactions quantum chemistry is reliable for have a name: an isodesmic reaction is one in which the number and type of each kind of bond is the same on both sides, and the definition is exactly the condition above written in chemical language. It is why bond energies are additive where they are, and why the standard advice for a difficult calculation is to find a balanced reaction to compute it as.

And it is why spin gaps, bond dissociations and transition states are the hard cases. In each of them the two states being compared differ in precisely the way that makes the cancellation fail: one of the pair is a good single determinant and the other is not.

The rule has a second use, which is to say when a correction computed on one system may be carried to another — the same condition read as a licence rather than as a warning.

The four-ring is where that failure is starkest, and the smallest many-electron calculation plots its levels against the repulsion exactly. At zero repulsion the singlet and the triplet are degenerate — the one-electron model’s honest answer for two electrons in two degenerate orbitals — and the singlet drops below for every repulsion above it. A mean field cannot follow that curve, because following it means being more than one determinant.

The same failure, three places in this collection #

The pattern is general enough that three other results turn out to be instances of it, and reading them together is what makes it a rule rather than an observation.

Koopmans’ theorem reads an ionisation energy off an orbital energy, neglecting relaxation and correlation, which have opposite signs. It works because they partly cancel and it is exact for nothing, because the cancellation is between two errors that depend differently on the system.

A method that is not additive fails on a pair of separated subsystems for the same reason in reverse: two calculations that ought to add do not, because the error is not proportional to the number of electrons. That is size consistency, and it is error cancellation’s requirement stated as a theorem.

And the correlation energy itself is a difference between an exact energy and a reference, so every statement about it inherits whatever the reference was — which is the argument a comparison of two references makes and finds a factor of 259 between them.

All three are the same complaint. A quantity computed as a difference is as good as the correlation between its two errors, and nothing in the calculation reports that correlation.

What a chemist does about it #

The practices that follow are all standard and all are versions of the same idea.

Compute a balanced reaction. Arrange the comparison so that both sides have the same bonds, then subtract.

Never quote an absolute energy. This is stronger than it sounds: a total energy from a correlated method is not a number about the molecule, and a better energy is not a better answer about anything else either. A total energy from a correlated method is a number with several digits of systematic error in it, and its only use is inside a difference.

Treat any comparison across a change in spin or bonding as unprotected. A spin gap is the standard hard case, and so is any process where a bond is broken — where the description itself changes partway along the coordinate. The gap between a closed-shell and an open-shell state gets no cancellation, and the method has to be right about both.

And check the reference. A calculation whose reference determinant is a poor description of one of the two states is the case above, and there are diagnostics for it — the weight of the leading determinant, the occupation of the natural orbitals — that can be computed directly.

Two of those diagnostics are computed in this collection and both are single numbers. The weight of the leading determinant says how much of the exact state one determinant accounts for — where molecular orbital theory dissociates follows it down — and where it falls, the cancellation measured above stops being available. The occupations of the natural orbitals say the same thing from the other side: a single determinant has occupations of exactly two and zero, anything else is a measure of the distance from being one, and the departure rises through exactly the range where the singlet–triplet sign goes wrong.

Where the model stops #

This is one model with one parameter, and the repulsion is turned up far past anything a molecule has. The point of the range is to show a trend rather than to claim a magnitude.

The mean field here is spin-restricted. Allowing the two spins different orbitals recovers much of the singlet’s error and buys it with a reference that is not a spin eigenfunction, which is a subject of its own and does not remove the difficulty — it moves it into the spin.

Only one kind of reaction was diagonalised. A comparison in which the electron count changes — an ionisation, or an addition — has an error that does not cancel even in principle, because the two sides have different numbers of pairs for the mean field to be wrong about. The section below takes that case as far as the closed form allows, which is far enough for a number and not far enough for a curve. Two kinds of correlation are in play there, and only one of them is proportional to the electron count.

And the reaction is a model reaction. Nothing in it corresponds to a real bond being formed; what corresponds is the arithmetic of two calculations with the same electron count and different connectivity.

A third comparison of the same kind comes from photoelectron spectroscopy. An ionisation energy compares two states differing by an electron rather than by a spin, and there relaxation and correlation are errors of opposite sign that partly cancel — what a photoelectron spectrum measures plots both against the repulsion. The size of that accidental cancellation is the whole reason a theorem with no correlation in it is quoted next to a measured spectrum, and the section below is what happens when the electron count is what changes.

The reaction that cannot cancel, and by how much #

The comparison above keeps the electron count fixed on both sides, which is the condition the whole defence rests on. It is worth taking the other case far enough to put a number on it, because the closed form that generates the errors makes that possible without another diagonalisation.

A spin-paired mean field spreads each spin uniformly, so at a site density $n$ its double occupancy per site is $(n/2)^2$, and at strong repulsion the exact state’s is near zero. The error is therefore

$$\Delta E \approx U N \frac{n^2}{4},$$

which is the linear growth the slope figure shows, with the electron count sitting in it as $n^2$ rather than as $n$. That square is the whole of what follows.

At half filling $n = 1$ and four sites, the expression gives $\Delta E \approx U$ — and the measured error at the strongest repulsion here is 12.11, so the model is being run near $U = 12$ and the closed form reproduces the diagonalised number without being told it.

Now remove one electron. The density falls to $n = 3/4$, and the error falls with the square, to $0.5625\,U$. The two errors do not cancel: what survives is

$$\left(1 - \tfrac{9}{16}\right) U \approx 0.44\,U \approx 5.3,$$

against an ionisation energy of order the bandwidth, which for these parameters is a few units. The residue is larger than the answer by roughly a factor of two, and it is fourteen times the 0.38 that survived the balanced reaction on the same system at the same repulsion.

That is the quantitative form of the warning, and the shape of it is worth keeping. The error is not proportional to the electron count; it is proportional to the square of the density, so two calculations differing by one electron in four differ in error by nearly half. Balancing a reaction is not a stylistic preference about how to write it down — it is the difference between a residue of three per cent of an error and a residue of forty-four per cent of the same error.

It also explains why ionisation energies and electron affinities are the quantities computational chemistry is worst at, and why they are usually obtained from a method built to compute the difference directly rather than by subtracting two total energies. A method that solves for the gap never forms the two large numbers, so it never needs their errors to be similar.

And it sharpens the isodesmic rule into something checkable before the calculation runs. Count the electrons on each side; if they differ, no amount of similarity elsewhere recovers the cancellation, because the leading term of the error is not the same on the two sides. If they agree, the remaining question is the one the rest of this essay is about — whether the two wavefunctions are the same kind of object — and that one cannot be settled by counting.

The number that would say when to trust it #

An obvious question follows from all of this: is there a quantity that says, before a calculation is believed, how much cancellation it is getting?

There is one, and it is not usually computed. The residue is the difference of two errors, and each error is the correlation energy of one side. So the residue is the change in correlation energy across the reaction — and a reaction whose two sides have the same correlation energy has a residue of zero, however wrong each side is.

That reframing is useful because the change in correlation energy is estimable without the exact answer. It goes with the change in the number of electron pairs, with the change in how far the state is from a single determinant, and with the change in the number of near-degenerate orbitals. Every one of those can be read off the approximate calculation itself.

So the practice the arithmetic recommends is: compute the reaction, and then compute how much the character of the wavefunction changed across it. A reaction in which the leading determinant’s weight moves by a per cent is protected; one in which it moves by twenty is not, whatever the total energies look like.

What is quoted, and what is computed #

Nothing is quoted. Every exact energy is a full diagonalisation of the configuration space; every approximate one is a self-consistent solution of the same model’s mean field; the residues, the cancellation fractions and the relative errors are differences between them.

The word isodesmic and the practices in the section above are quoted, as a description of what the field does.

What the comparison requires #

More than eighty per cent of the error cancels at every repulsion tested, or there would be nothing to explain.

The cancellation improves along the range and the relative accuracy of the answer gets worse, both, on the same pair of systems.

The residue exceeds the quantity being computed at the strongest repulsion, which is the sharp form of the second half.

For a pair whose two members are not the same kind of state, the mean field gets the sign wrong, and the two errors differ by more than a factor of ten. Both are needed, because the sign change alone could be a coincidence and the ratio alone is not the finding.

Still open: cancellation between two methods #

Everything above measures cancellation between two calculations with the same method. The other half of the practice is cancellation between two methods, and it is what every composite scheme in computational chemistry is built on: compute a small correction at a high level and a large one at a low level, and assume the errors of the two are separable.

That assumption is testable in exactly the way this essay tests the first one — take a system where the exact answer is available, compute the pieces, and ask whether the sum of the corrections is the correction of the sum. It is not, in general, and the extent to which it fails is a quantity that could be measured on four sites rather than argued about.