The lookup was the problem, not the references
Worth reading first: A reference at the worst miss made the worst miss worse · Two readings separate the square only where they do not fold.
A composite method adds a correlation correction computed somewhere else to a mean field computed here, and on the transferability square — a ring of four at half filling, swept in its repulsion and a staggered site energy — the plainest version is a lookup. Read two cheap diagnostics off the mean field, the double occupancy and the highest occupied level; find the reference system whose two readings are nearest; borrow its correlation energy. Two readings separate the square, so the lookup is not fooled by a fold, and on eighty-one coarse references it misses the other 1,008 systems by a median of 0.013 hopping units.
Its worst miss, 0.166, sits on the line where the site energy equals the repulsion, and placing more references there helped less than hoped. The best rule a calculation could run reached 0.060 after 128 references; an oracle that knew every answer needed 480 to reach the median; and the oracle’s very first reference made the worst miss larger, because a nearest-neighbour lookup is piecewise constant and a new reference redraws the cells. That essay ended by asking about the lookup itself — whether an interpolating form would remove the cells, and whether the disagreement among a system’s nearest references could at least say which answers not to trust.
The lookup was most of the problem. Changing it halves the worst miss with no new references. It does not make references harmless, and the form that does best has a failure of its own that the nearest reference could never have.
Three ways to use the same references
The three forms read the same two scaled readings and the same references. The nearest reference is the published lookup. The weighted form averages the three nearest, each weighted by the inverse square of its distance, which is the smoothing most interpolation schemes start from. The local plane fits by least squares through the six nearest references, in the two readings and measured from the system being asked about, and returns : the plainest form that can follow a slope rather than average across it.
On the eighty-one coarse references the plane misses by a median of 0.0050 and at worst by 0.088. That is a median 2.6 times smaller than the nearest reference’s and a worst 1.9 times smaller, and it is better at the worst than the nearest reference on the whole 289-system grid, 0.130, and as good as the best runnable placement managed for the nearest reference after adding sixty-four references along the crossover line, 0.089. It is better in the median than the nearest reference on that grid too. On that grid the plane’s own median is 0.0015 and its worst 0.068.
The weighted average is the surprise. Its median is 0.0073, close to twice as good as the nearest reference’s, and its worst is 0.232, worse than doing nothing. The worst is at , , on the crest of the correlation energy along the crossover line, and the reason is visible in the readings: the system’s three nearest references lie on both sides of the steep stretch, and an average of a high value and two low ones is wrong in a way neither neighbour alone would be. A smoother is not a better lookup where the thing being looked up is steep. The plane can tilt to follow the slope; the average can only blend across it.
Beyond the edge of the references
The crest system shows what separates the three forms, and it is not smoothness. Its correlation energy is −0.596, and the most negative of all eighty-one coarse references is −0.432: eleven systems of the fine grid lie beyond the coarse square’s range on this crest, where no reference has an answer as large. Its nearest reference in the readings, at and , lends −0.432 and misses by 0.164. The weighted form adds the next two nearest, at −0.154 and −0.328, and averages to −0.364, further from the truth than the nearest reference alone; an average of references can only land inside their range, and here the right answer is outside it on the far side from the second and third neighbours. The plane through the six nearest tilts with the trend from the crossover line’s lower reaches up to this one, and returns −0.561, a miss of 0.035.
So the plane’s advantage is that it can extrapolate, and the weighted average’s disadvantage is that it cannot. On a square whose worst misses are on a crest the coarse grid never sampled, a lookup confined to the range of its references is confined to being wrong there. The same property that lets the plane reach −0.561 is what will let it reach 3.99 a few sections below, and the two cannot be had separately: a form that may leave the references’ range may leave it in the wrong direction.
The ground state’s character explains why a trend exists to follow. Along the line the cost of putting both electrons of a pair on a low site balances the cost of keeping one electron on every site, and the mean field, which can describe either insulator but not their mixture, misses more correlation energy the larger both costs are. The crest is that growth carried past the coarse grid’s largest repulsion on the line, and a slope fitted lower down points at it.
Where the plane still misses
The plane’s misses are where the nearest reference’s were, along the crossover line, but there are far fewer of them. Ten systems are missed by more than 0.06, against thirty-one for the nearest reference, and two by more than 0.08. They sit in a narrow strip just below the line, at site energies a little smaller than the repulsion, from to . The column at small site energy where the highest occupied level turns over, which supplied eight of the nearest reference’s fifty worst, is gone: there the correlation energy changes smoothly along a reading that has stopped moving, and a plane in two readings handles a slope in one of them without difficulty.
What remains is the crossover itself. Across the line the ground state changes from one with a single electron on each site to one with the low sites doubly occupied, and the correlation energy the mean field misses has a ridge there that a plane through six points can only approximate by a slope. The plane’s residual is the curvature of that ridge.
Placing references with a better lookup
Placing references along the crossover line helps the plane quickly and then stops helping. Its worst miss goes from 0.088 to 0.062 with sixteen references and 0.057 with thirty-two, and then stays between 0.052 and 0.061 to 208. The nearest reference and the weighted average fall much further over the same placements, to 0.060 and 0.079 at 128, because they had further to fall; both rise again at 208 when the band of systems near the line runs out and the placements move off it. The plane rises there too, from 0.052 to 0.061. A better lookup changes how much the references are worth; it does not change the fact that the right references are a finite set and the wrong ones can hurt.
A better lookup is not a monotone one
The essay that found a reference could make the worst miss worse traced it to the nearest reference’s cells, and asked whether an interpolating form would remove them. It does not. Placing each new reference at the form’s own current worst miss — the oracle placement, which knows every answer and bounds what any rule could do — the nearest reference’s worst rises 22 times in 128 placements, the weighted average’s 37 times and the plane’s 30.
The mechanism is different for each and the effect is the same. The weighted average’s neighbourhoods change discontinuously when a new reference enters a system’s three nearest, so it has cells of its own, finer than the nearest reference’s. The plane’s fit changes when a new reference enters its six, and its first oracle placement, the crest system at , , tilts the plane for a neighbour and raises the worst miss from 0.088 to 0.117. And the plane has a failure the others cannot: at the thirty-ninth and forty-third placements the worst miss is 3.99 and 2.05, twenty to forty times larger than anything the nearest reference ever produced. At those steps some system’s six nearest references lie almost on a line in the two readings, the plane through them is barely determined in the perpendicular direction, and its value at the system is an extrapolation along that direction. A nearest-reference lookup can never be further off than the range of the references’ answers; a plane can be off by anything.
Between those spikes the plane’s oracle is far better than the others: 0.017 at 64 placements and 0.010 at 128, against 0.045 and 0.031 for the nearest reference, and it reaches the nearest reference’s coarse median, 0.0131, after 93 placements rather than 480.
The warning, tested
The second question was whether the lookup can say which of its answers not to trust. The candidate is the spread of correlation energy among a system’s three nearest references, which is known as soon as the references are, before anything about the system except its two readings.
It is a fair warning, not a perfect one. The rank correlation between spread and miss across the 1,008 systems is 0.58. The fifty systems with the largest spreads, five per cent of the square, contain seventeen of the twenty-one worst misses; the twenty-one largest spreads contain ten. All twenty-one worst misses lie near the crossover line, and so do most of the flagged systems. Of the four the flag misses, two have three nearest references that agree with one another and disagree with the system: on the ridge at , and , , three references from one side of it agree to 0.035. The other two have spreads of 0.090 and 0.126, at , and , , below the fiftieth-largest spread of 0.152. The worst miss among unflagged systems is 0.129.
Combined with the plane, the warning means something sharper. Among the fifty flagged systems the plane’s worst miss is 0.088; among the 958 unflagged, 0.062. Nine of the plane’s ten misses above 0.06 are among the flagged fifty, and the tenth is at 0.062. A composite that used the plane and reported the spread alongside its answer would be wrong by more than 0.06 almost only where it had already said it might be.
How the forms were built and checked
Every system’s correlation energy is its exact ground-state energy on the ring of four, less its lowest unrestricted mean-field energy from the search that fixed the solver’s worst system, as throughout these essays; the two readings are the mean field’s double occupancy and the energy of its highest occupied level, each scaled by its range on the coarse square, and distances are Euclidean in the scaled pair. The weighted form averages the three nearest references with weights . The plane is the least-squares solution of a three-by-three normal equation through the six nearest, centred on the system so that its constant term is the answer. Every form is scored on the 1,008 systems of the 33×33 grid that the 9×9 coarse square does not contain, less any that have been added as references.
The checks run where these figures are drawn. The nearest form reproduces the published lookup, worst 0.166 and median 0.013, and the published 0.060 after 128 placements along the line. The two interpolating forms return a reference’s own correlation energy when asked about it, to ; the plane does not, being a least-squares smoother rather than an interpolant. Given references whose correlation energy is an exact linear function of the two readings, the plane returns that function at two hundred held-out systems to . And the refusal: the weighted average is an interpolant too, and it must do worse than the nearest reference at the worst on the coarse square, which it does, 0.232 against 0.166. Had it done better, the plane’s gain would have been interpolation as such; it is following the slope.
What a better lookup on a ring of four cannot say
Six neighbours and a plane are one choice among many. A quadratic surface through more neighbours, a kernel with a fitted width, or a plane whose neighbour count adapts to how well the neighbours span the readings would each behave differently, and the last would remove the near-collinear failures. The comparison here is between the plainest forms of three ideas, not a search for the best lookup.
The square has two parameters and two readings. The plane can follow the correlation energy because two readings separate a two-parameter family and the answer is smooth in them away from the ridge. On a benchmark with more axes than readings no lookup in the readings can be exact, and the slope a plane follows would be partly an artefact of what the readings do not see.
The warning is computed from the coarse references. As references are added the spread among a system’s neighbours shrinks, and a threshold chosen on eighty-one references would flag fewer systems on two hundred. Whether the flag stays calibrated as the references grow is not tested.
The lookup is part of the composite
The sequence of these essays has asked what a cheap calculation can warn of and found, one diagnostic at a time, failures in five different places, a constant that ties the best single reading and a pair of readings that separates the square. Every one of those questions was about the inputs to the composite. This one is about the step that turns inputs into an answer, and it turns out to matter as much as any diagnostic: the same two readings and the same references give a worst miss of 0.232, 0.166 or 0.088 according to how the answer is read off them.
The difference between the forms also reframes where on the square the difficulty lives. The arms of the square were kind to every diagnostic because the correlation energy there changes slowly along one reading; the crossover line is unkind because it changes quickly along both. A form that can follow a slope in two readings at once turns the unkind region into a region of curvature, which is a smaller problem than a region of slope, and the plane’s residual misses are what is left of it.
That is a familiar lesson in a different setting. A correction computed somewhere else is only as good as the rule that transfers it, and the transfer rule usually goes unexamined because the nearest reference is the obvious one. On this square the obvious rule is neither the best nor the safest: the plane is better almost everywhere and unbounded at a few points, and a composite worth trusting would use one and guard against the other.
Still open: a plane that knows when it is guessing, and a warning that scales
The obvious open question is the plane’s failure. It extrapolates only when its six neighbours are nearly collinear in the readings, and that is detectable from the neighbours alone: the smaller eigenvalue of their spread in the two readings. A plane that falls back to the nearest reference, or widens its neighbourhood, whenever that eigenvalue is small would keep the plane’s 0.088 and remove the spikes of 3.99 and 2.05, and whether its own oracle is then monotone would settle what the cells of a lookup really cost.
The nearer question is the warning’s calibration. The spread among the three nearest references flags seventeen of twenty-one on eighty-one references. Repeated on the 17×17 grid as references, with the same five-per-cent threshold, it would say whether the warning keeps finding the worst misses as they shrink, or whether it was finding the crossover line, which is a fixed region whatever the references are.
What links here
Computed from the collection rather than written here: the essays that point at this one.
Reads more easily once this is understood
Essays that name this one as worth reading first.
Shares its objects with
Essays naming at least two of the same things, that neither author linked.
- The second number is the error, rearranged — both name composite method, correlation energy, hubbard model, mean-field approximation, transferability
- A blank is not a pass — both name composite method, correlation energy, mean-field approximation, transferability
- The worst case measured the reference — both name composite method, correlation energy, mean-field approximation, transferability
- A cancellation between two separations — both name correlation energy, hubbard model
- A mean field cannot get out of the way — both name correlation energy, hubbard model
- A root that moves when nothing else does — both name correlation energy, hubbard model
Named objects
A dashed tag is an object no other essay names yet.
Composite methodCorrelation energyHubbard modelMean-field approximationTransferability