The reference decides the correlation

The correlation energy is defined by subtraction: take the exact energy and take away the Hartree–Fock energy, and what is left is what a mean field misses.

The first term of that subtraction is a property of the system. The second is the output of a procedure, and this essay is about what happens when the procedure has more than one answer.

It has more than one answer for every system in which a symmetry-broken solution exists, which is most of them past a point, and the two answers do not differ by a little.

Two procedures with the same name #

A restricted mean field puts both spins in the same orbitals. It is the natural choice for a closed-shell system, it is a spin eigenfunction by construction, and it is what “Hartree–Fock” means in most sentences that use the phrase.

An unrestricted one lets the two spins have different orbitals. Below a certain repulsion the search returns to the restricted answer, because there is nothing to gain; above it there is, and the solution polarises — one spin accumulating on one sublattice and the other on the other.

Both are stationary points of the same energy expression. Both are variational, so both are above the exact energy. Both are called Hartree–Fock.

The exact energy is a single smooth curve through the whole of that figure. There is no repulsion at which anything happens to the system.

The numbers #

At U = 4 the restricted reference reports a correlation energy of −1.481 and the unrestricted one −0.519, a factor of 2.9.

At U = 8, −4.645 and −0.375: a factor of 12.4.

At U = 32, −27.82 and −0.107: a factor of 259.

And the sign of the slope differs. The restricted correlation energy grows without limit, as it must, because the restricted energy rises linearly with the repulsion while the exact energy falls towards zero. The unrestricted correlation energy peaks at about U = 4 and falls back — because a polarised solution with one electron on each site pays almost nothing to the repulsion, which is what the exact state does too.

So the question how strongly correlated is this system at U = 32 has the answers “extremely” and “hardly at all”, and both are computed from the same exact wavefunction.

What the second answer costs #

An unrestricted solution is lower in energy and is not a state of the system.

Its spin can be computed. For a determinant built from two sets of orbitals, ⟨S²⟩ is the value the spin projection forces plus a term measuring how far the two sets have drifted apart; for a restricted determinant the two sets are identical and the value is exactly zero.

That is the trade. The unrestricted reference describes the energy of the exact state much better and describes the state much worse: it has the right energy and the wrong spin, where the restricted one has the wrong energy and the right spin.

The exact state, throughout, is a singlet. Every property that depends on the spin — the magnetic susceptibility, the hyperfine couplings, the selection rules of any spectroscopy — is a property the unrestricted reference gets wrong at exactly the repulsions where its energy is best.

The kink that is not in the system #

There is a second oddity in the figure and it is easy to miss.

The unrestricted correlation energy has a kink at the instability. Below it the curve is the restricted one; at U = 2 it departs; above it, it turns over. A plot of the correlation energy against the repulsion is not smooth there, and nothing whatever is happening to the molecule.

Non-analytic behaviour in a computed quantity is usually a signal. Here it is a signal about the definition: the point where a second stationary point of the mean-field energy comes into existence is a property of the mean-field procedure, and the quantity defined by subtracting that procedure’s answer inherits it.

That shape has turned up before. A resonance energy is measured from a reference state and jumps when the reference is changed; a mode’s percentage composition depends on the coordinate set; a structure’s weight depends on the orthogonalisation. Each of them is a difference or a projection against something chosen, and each carries features that belong to the choice.

Where the instability comes from #

The point at which the second solution appears is computable and has a name in every corner of physics where a mean field is used.

Below it, the symmetric solution is a minimum of the mean-field energy. At it, one of the curvatures of that energy with respect to a symmetry-breaking distortion of the orbitals passes through zero. Above it, the symmetric solution is a saddle point and the minima are two polarised solutions related by exchanging the spins.

That is the same mathematics as a Peierls distortion, a Jahn–Teller distortion and a magnetic ordering transition — and in the first two of those, the symmetry that breaks is a symmetry of the molecule and the breaking is real, measurable and structural. In each case a symmetric solution stops being stable at a computable point and a lower one appears with less symmetry than the problem has.

The difference here is that the symmetry being broken is not a symmetry of the molecule. It is a symmetry of the description, and the exact wavefunction never breaks it. So the whole phenomenon is an artefact of insisting on a single determinant — and the artefact is what the correlation energy is measured against.

What each reference is right about #

It is worth setting out what the two descriptions get right, because the choice between them is a genuine one and this essay is not recommending either.

The restricted reference is a singlet, has the right symmetry, and gives orbitals that transform as the irreducible representations of the molecule’s group — so every symmetry argument built on it is sound, and a degeneracy predicted by the group is present in its levels. Its energy is badly wrong at strong repulsion and the wrongness is systematic.

The unrestricted reference has an energy far closer to the truth and describes the actual spin arrangement of the exact state — alternating up and down around the ring — much better. What it cannot do is be a singlet while doing so, because a single determinant with alternating spins is a mixture of spin states.

That is not a defect of the calculation. It is a theorem: the exact singlet at strong repulsion is a superposition of the two alternating arrangements, and a single determinant is one of them. Choosing a determinant means choosing one arrangement, which breaks a symmetry the exact state has.

The exact state’s own description of the same physics is computed elsewhere: the spins are correlated without either of them being anywhere in particular, and the quantity that says so is a correlation function rather than an orbital.

Which one is quoted #

Both, in practice, and the literature is not always careful about which.

A method whose energy is quoted as a correlation energy — second-order perturbation theory, coupled cluster, configuration interaction — is quoting a difference from its own reference. Comparing two such numbers computed from different references is comparing two different quantities.

There is a rule of thumb in use and it is worth stating: quote the total energy and the reference used, and let the reader do the subtraction they want. That is the same advice that holds for a fitted moment — report the curve, not the summary — and for the same reason.

What it does to a calculation nobody would call unusual #

The two references are not a pathology of a model with a large parameter in it. The instability appears in ordinary chemistry, at ordinary geometries, and the place it is met most often is a bond being stretched.

A closed-shell molecule at its equilibrium geometry is below the instability: the restricted and unrestricted solutions coincide, both are singlets, and the correlation energy has one value. Stretch the bond and at some separation the unrestricted solution appears — the Coulson–Fischer point — and from there to dissociation the two references give different energies, different correlation energies, and different spins.

So a potential energy curve computed at the unrestricted level is smooth in energy and has a kink in every quantity defined against the reference; and one computed at the restricted level dissociates to the wrong products while remaining a proper singlet all the way.

That is the practical form of this essay’s finding. Neither curve is wrong; they are two different subtractions, and a paper that plots a correlation energy along a dissociation coordinate is plotting something whose definition changes partway along.

There is a quantity that does not depend on a reference at all: how the exact state’s own natural occupations move with the repulsion. That is a measurement of the wavefunction rather than a difference from a determinant, and it is the kind of quantity that survives this essay’s objection.

The third answer, which is the one a chemist actually uses #

Two procedures with one name is the essay’s finding, and it undercounts. The broken-symmetry solution’s defect — a reference with $\langle S^2 \rangle$ near two where a singlet requires zero — has a standard repair, the repair is in daily use, and it produces a third energy for the same system.

The repair is projection. A determinant with the wrong spin is a mixture of states of several spins, and the unwanted components can be projected out, leaving a function with the right symmetry and a lower energy than the broken determinant had. That function is no longer a single determinant, so what it returns is not a Hartree–Fock energy by the definition the subtraction assumes — and the correlation energy measured against it is a third number, between the two this essay compares.

Which is not a curiosity. It is how exchange couplings in magnetic molecules are computed. A pair of transition-metal centres with one unpaired electron each has a singlet and a triplet separated by a few tens of wavenumbers; the triplet is describable by a single determinant and the singlet is not; and the standard procedure is to compute the broken-symmetry solution, compute the triplet, and extract the coupling from the difference with a projection formula relating the two. The whole field of computed magnetic couplings rests on a reference that is deliberately not a spin eigenfunction, and on a correction that puts the spin back afterwards.

Two forks open up as soon as that is admitted, and both are choices somebody has to make.

Project before or after optimising. Optimising a determinant and then projecting is not the same as optimising within the space of projected functions, and the second is both better and far more expensive. Papers do both, and the difference is a real energy.

And which quantity the projection formula assumes. The relation between a broken-symmetry energy and a spin-state splitting depends on how strongly the two centres interact, and there are competing expressions with different limits built into them. Choosing one is choosing an answer.

So the count of things called the Hartree–Fock energy for one system, in one basis, is at least three: restricted, unrestricted, and projected — with two versions of the last. Each supports a different correlation energy, and the factor of 259 measured here is the spread between only two of them.

The essay’s conclusion is unchanged and its scope is wider. A correlation energy is a distance from a reference, and the reference is a decision. What the practical case adds is that the decision is often made for a good reason — a broken-symmetry reference is the only affordable way to describe an open-shell singlet — so the answer is not to insist on the symmetric one. It is to say which was used, in the same breath as the number, exactly as a delocalisation energy has to name what it was measured from.

Where the model stops #

One system, one parameter. Four electrons on four sites, with the repulsion swept over a range no molecule reaches. What survives the model is the structure of the argument rather than the numbers.

The unrestricted solution here is a spin density wave, the simplest kind of symmetry breaking available. Real unrestricted calculations break other symmetries — spatial ones, and complex conjugation — and each of them produces its own instability with its own threshold and its own kink.

Nothing here projects the spin back out. There are methods that do, and they recover a reference that is both low in energy and a spin eigenfunction at the cost of no longer being a single determinant. That is the sensible resolution and it removes the possibility of quoting a correlation energy at all, because there is no determinant left to subtract.

The polarised solution here is found by nudging the density and letting it settle. A search that started from the symmetric density would stay there for ever, because a symmetric density is a fixed point of a symmetric problem — which is a numerical fact with a physical shadow: an instability has to be looked for to be found, and a calculation that never looks reports the symmetric answer with no warning.

And the exact energy is available, which is why every statement here can be made. In a real calculation it is not, and the correlation energy is estimated by a method whose own error is what is being estimated.

The same instability seen from photoelectron spectroscopy is what a symmetry-broken solution does to an ionisation energy. The quantity computed there is a difference of two mean-field energies, so it inherits the convention twice over.

And the consequence for a quantity a chemist wants: the singlet–triplet gap from a restricted mean field changes sign. A reference that is wrong about the singlet by one amount and about the triplet by another does not merely give a poor gap — it gives one on the wrong side of zero.

The exact states have no kink, no instability and no convention anywhere in them: every level of a two-site system against the repulsion, with the closed form checked against the diagonalisation at every point. Everything this essay is about is a property of the reference, and the exact answer has none.

What is quoted, and what is computed #

Nothing is quoted. Both mean fields are solved self-consistently on the same model; the exact energies are full diagonalisations; the spins of the references are computed from the overlaps of their occupied orbitals; the instability is located by watching where the polarised search stops returning the symmetric answer.

A word about the word #

“Correlation energy” was defined by Löwdin in 1955 as the difference between the exact non-relativistic energy and the Hartree–Fock energy in a complete basis, and the definition was careful: it named the reference, and it named the basis.

Both qualifications have worn off in use. A correlation energy is now routinely quoted from a finite basis, against whichever mean-field solution the program converged to, and compared with another one computed differently. The quantity that results is a difference between two numbers of which only one is a property of the molecule.

That is not an argument for abandoning it — it is the organising quantity of an entire field and it works — but it is an argument for two habits that keep proving their worth from different directions. Name the reference. And where a quantity can be computed without a subtraction, prefer the one that can.

What was checked #

At zero repulsion both correlation energies are exactly zero. Without this the divergence above could be arithmetic drift rather than a definition parting company with itself.

Below the instability the two agree to a part in a billion, and the unrestricted reference is still a singlet there. Both halves: the agreement makes the kink real, and the spin makes it a fair comparison.

Above it the restricted correlation energy grows and the unrestricted one falls — opposite signs of slope, not merely different magnitudes — until they differ by more than an order of magnitude.

The restricted reference is a spin eigenfunction at every repulsion, exactly, and the unrestricted one at the largest repulsion is not, at ⟨S²⟩ above 1.5.

One more reading closes the argument, because it prices the two conventions against each other rather than each against the exact answer.

Still open: the pair distribution #

The correlation energy has now been taken apart by its energy, its wavefunction, its scaling and its reference. What has not been computed is the thing all four are proxies for.

Correlation is a statement about where the electrons are relative to each other, and there is a function that says so directly: the pair distribution, the probability of finding one electron here given another there. It needs no reference, no determinant and no subtraction; it is a property of the exact state alone, and it is computable exactly for every system here.

Two systems with the same correlation energy can have quite different pair distributions, and two references that disagree by a factor of 259 about the energy would produce visibly different ones. Computing it would replace a quantity defined by subtraction with a quantity defined by measurement, which is the move worth making wherever it can be made.