What is taught wrongly

Leverage catches the spikes and none of the rises

A least-squares plane through a system's six nearest references was the best lookup the transferability square had found, and its own greedy placement twice extrapolated to misses of 3.99 and 2.05. Both are visible before any answer is known: the plane's leverage at the system, the variance factor of the value it returns, is 6,706 and 2,172 there and never above 1.88 on any held-out system of the coarse references. A guard anywhere between 2 and 100 removes both spikes and changes nothing else. It does not make the plane monotone. The guarded plane's worst miss still rises thirty-two times in 128 placements, and every one of those rises — and every one of the unguarded plane's thirty — happens at a system whose six neighbours have just gained the reference placed.

Worth reading first: The lookup was the problem, not the references · A reference at the worst miss made the worst miss worse.

A composite method borrows correlation energy from reference systems that a calculation can afford, and gives it to systems that it cannot — a correction computed somewhere else, carried to where it is needed. On the Hubbard transferability square — a ring of four sites at half filling, its repulsion U and staggered site energy ε each varied, every system’s exact correlation energy known — the plainest version gives each held-out system the correlation energy of whichever of the 81 coarse references is nearest to it in two cheap readings, its double occupancy and its highest occupied level, the pair that separates the square where it does not fold. The lookup was the problem, not the references found that a least-squares plane through the six nearest references does much better with the same 81: a worst miss over the 1,008 held-out systems of 0.088 and a median of 0.0050, against the nearest reference’s 0.166 and 0.013.

It also found the plane’s one failure. Placing references one at a time wherever the plane currently misses worst — the oracle placement, which knows every answer and so bounds what any rule could achieve — the worst miss at the thirty-ninth placement was 3.99 and at the forty-third 2.05, in units where every correlation energy on the square lies between −0.6 and zero. At those steps some system’s six nearest references lay almost on a line in the two readings, and a plane through points on a line is free to tilt about it. The essay ended by asking for a plane that sees that from its neighbours alone and declines to answer — and whether, with the spikes gone, adding a reference would stop making the worst miss worse.

The first half has a clean answer. The second half does not, and the reason it does not is the more useful finding.

Where the plane was guessing

The thirty-ninth placement’s spike is worth looking at directly, because its geometry says what any detector has to detect.

The spike's six neighbours are one row of the square, nearly collinear in the readings, and the system sits off their line. The system the unguarded plane missed by 3.99 after its 39th placement, at U = 14.89, ε = 8.5, correlation energy -0.3098, and its six nearest references in the two scaled readings, all at ε = 8: U = 13.86, -0.3240; U = 16.00, -0.2781; U = 12.89, -0.3472; U = 20.00, -0.2166; U = 12.00, -0.3686; U = 10.84, -0.3917. The neighbours' spread across their line is 1.5e-5 of its spread along it; the plane evaluated at the system gives -4.302, with leverage 6706.
Fig. 1 The system the unguarded plane missed by 3.99, and its six nearest references in the two scaled readings. All six are references at one site energy; the system lies off their line.

The system sits at U = 14.9, ε = 8.5, and its six nearest references are all at ε = 8, at repulsions from 10.8 to 20. In the two readings those six lie along a line: their highest-occupied-level coordinates agree to two parts in a thousand while their double occupancies spread across a tenth of the scale. Along that line their correlation energies change steadily, from −0.217 to −0.392, and the plane fits that change well. Across the line it has almost nothing to go on — six points whose variance across the line is a hundred-thousandth of their variance along it — and the system is not on the line. It lies 0.027 above it, which is as far away as its nearest reference.

The plane’s slope across the line is whatever the tiny scatter of the six about their line dictates, and evaluated 0.027 out it returns −4.30 for a system whose correlation energy is −0.310. The forty-third placement’s spike is the same picture one row down: six references at ε = 6, a system at ε = 6.5, a variance across the line seven hundred-thousandths of the variance along it, and an answer of −2.39 for −0.339.

That geometry does not need the answer to be detected. It needs only the six neighbours’ readings and the system’s own, which are what the plane is built from anyway.

The number the plane already computes

The question as it was posed suggested the most direct detector: the smaller eigenvalue of the neighbours’ spread in the two readings, relative to the larger. It is near zero exactly when they lie on a line. At the two spikes it is 1.5 × 10⁻⁵ and 7.4 × 10⁻⁵, and over every held-out system on the coarse references it never falls below 0.023.

There is a better number, and the plane computes it anyway. A least-squares plane centred on the system returns its intercept, and the intercept’s sensitivity to the data is a diagonal element of the inverse normal matrix, h=[(XTX)−1]00h = [(X^{\mathsf T}X)^{-1}]_{00}, for the design XX whose rows are (1,Δx,Δy)(1, \Delta x, \Delta y) — the neighbour’s offset from the system in the two readings. That element is the factor by which the value’s variance exceeds the variance of a single reference, and it is called the leverage of the evaluation point. For six neighbours spread symmetrically round the system it is one sixth, the variance factor of a plain average; for six neighbours on a line with the system beside them it is enormous. It also grows when the neighbours are well spread but the system lies off to one side of all of them — the other way a plane extrapolates — which the eigenvalue ratio cannot see.

Leverage puts the two spikes three decades beyond every coarse system; the distance to the nearest reference puts them in the crowd. Top: the leverage of the six-neighbour plane at each of the 1,008 held-out systems on the coarse references, median 0.25, largest 1.88, and at the two systems the unguarded plane's placement missed by 3.99 and 2.05: 6706 and 2172. Bottom: the distance in the scaled readings to the nearest reference, median 0.0233, largest 0.1245, and at the two spike systems 0.0278 and 0.0278.
Fig. 2 Above: the plane’s leverage at every held-out system on the coarse references, with the two spike systems marked. Below: each system’s distance to its nearest reference, with the same two marked.

On the coarse references the leverage of the six-neighbour plane at the 1,008 held-out systems has a median of 0.25 and a largest value of 1.88. At the two spikes it is 2,172 and 6,706. Three decades separate the spikes from every ordinary evaluation, with nothing in between.

The obvious alternative, and the one most often reached for, is distance: a lookup is surely guessing when the system is far from its references. It is not what went wrong here. Both spike systems are 0.028 from their nearest reference in the scaled readings, where the median held-out system is 0.023 away and the furthest 0.125. They sit in the crowd. The refusal makes that a test rather than a picture: a guard that falls back whenever the nearest reference is further than any coarse system’s nearest reference — the tightest distance threshold that leaves the coarse plane untouched — lets both spikes through, and its own placement reaches 3.99 just as the unguarded plane’s does. Distance measures how far the system is from its references; leverage measures how far it is from where its references can say anything.

Two detectors, one gap

The eigenvalue ratio and the leverage both put the spikes three decades beyond the coarse systems, and either would remove them. They part company on the systems in between, and the difference is the reason to prefer the second.

A neighbour set can lie nearly on a line with the system on that line. Then the plane is being asked to interpolate along the one direction it knows, its value is well determined, and its leverage is modest; the ratio of the two eigenvalues is just as small as at a spike. On the 17×17 grid used as references, the eigenvalue guard fires on ten held-out systems and leverage on none, and the ten it fires on were being answered well — falling back to the nearest reference there moves the median miss from 0.0015 to 0.0016. Under its own placement the eigenvalue guard fires on up to thirty-five held-out systems at once, the leverage guard on at most fifteen. The ratio measures the neighbours; the leverage measures the neighbours as seen from the system, which is the only thing the answer depends on.

The same distinction separates a warning that reads the cheap calculation from one that reads only where the references sit. A warning is useful in proportion to how few good answers it condemns, and on that count the leverage guard condemns none here.

A guard that changes only the spikes

A guarded plane computes the six-neighbour plane and its leverage, and if the leverage exceeds a threshold it does something else: either returns the nearest reference’s correlation energy, or widens the neighbourhood to nine and then twelve references and fits again. The eigenvalue guard is run the same way, with a threshold of 10⁻³, for comparison.

On the coarse references none of the guards fires on any held-out system, so all three return the unguarded plane’s 0.088 and 0.0050 exactly. The interesting question is the oracle placement.

A plane that reads its own leverage loses both spikes and keeps every one of the ordinary rises. The worst held-out miss as each lookup places references one at a time at its own current worst miss, on a logarithmic scale, to 128: plane, unguarded: 30 rises, largest value 3.992, 0.0101 at 128; leverage guard, wider plane: 32 rises, largest value 0.117, 0.0101 at 128; leverage guard, nearest fallback: 32 rises, largest value 0.117, 0.0275 at 128. The guard falls back when the six-neighbour plane's leverage exceeds 2.
Fig. 3 The worst held-out miss as each lookup places references at its own current worst miss, on a logarithmic scale: the unguarded plane, and the plane guarded by its leverage with a wider-plane fallback and with a nearest-reference fallback.

All three remove both spikes. The guarded planes’ worst miss over 128 placements never exceeds 0.117, which is the plane’s first placement — the crest system at U = 8.85, ε = 10, whose reference tilts the plane for a neighbour — and the largest single rise falls from 3.97 to between 0.03 and 0.05. After 128 placements the plane with a wider fallback ends at 0.0101, the unguarded plane’s own value; the plane that falls back to the nearest reference ends at 0.0275, because by then five held-out systems trip the guard and the nearest reference is a worse answer for them than a wider plane.

Any leverage threshold from 2 to 100 does the same thing; only the ends of the range do anything else. For a plane that falls back to the nearest reference when its leverage exceeds a threshold, against the threshold: oracle's largest worst 0.129, 0.129, 0.117, 0.117, 0.117, 0.117, 3.992; worst, coarse references 0.164, 0.164, 0.088, 0.088, 0.088, 0.088, 0.088; worst after 128 0.014, 0.026, 0.028, 0.012, 0.012, 0.012, 0.010 at thresholds 1.2, 1.5, 2, 4, 10, 100, 10000. Coarse systems on which the guard fires: 6, 3, 0, 0, 0, 0, 0.
Fig. 4 For the plane with a nearest-reference fallback: the worst miss on the coarse references, the largest worst miss of its own placement, and the worst after 128, against the leverage threshold.

The threshold barely matters. From 2 to 100 every number in the figure is the same, because there is nothing between 1.88 and 2,172 for a threshold to separate. Below 2 the guard begins to fire on coarse systems — six of them at 1.2, three at 1.5 — and since those include systems the plane was answering well, the coarse worst miss rises to 0.164. At 10⁴ the guard misses the larger spike and 3.99 returns. A guard on a fitted lookup is usually presented as a trade between accuracy and safety with a threshold to tune. Here, on this square, it is neither: the plane’s own leverage puts the failures three decades away from the successes.

That is a statement about this square and these 81 references, and it is a measured one rather than a general one. A set of references laid out on a coarser or a more irregular grid would put more systems at intermediate leverage, and the gap could close.

The rises were never the spikes

The second half of the question was whether, with the spikes gone, the plane’s own placement becomes monotone. A reference at the worst miss made the worst miss worse found the nearest-reference lookup’s worst miss rising twenty-two times in 128 placements, even along the line where the steep side of the square concentrates every large miss, and traced it to cells: a piecewise-constant lookup reassigns a whole region when a reference arrives. The plane was expected to be smoother. It was not — its worst miss rose thirty times — and the spikes made it easy to suppose that the plane’s rises were its extrapolations.

Every rise is a new reference joining the worst system's six neighbours. Each placement after which the worst miss rose, the size of the rise on a logarithmic scale, for the unguarded and the guarded plane: plane, unguarded: 30 rises, median 0.0054, largest 3.968; in 30 of them the reference just placed is one of the new worst system's six nearest; leverage guard, wider plane: 32 rises, median 0.0044, largest 0.029; in 32 of them the reference just placed is one of the new worst system's six nearest.
Fig. 5 Each placement after which the worst miss rose, and by how much, for the unguarded and the guarded plane. The two spikes are off the scale; the rest are common to both.

They were not. Two of the unguarded plane’s thirty rises are the spikes. The other twenty-eight have a median of 0.005, and the guarded plane has almost exactly the same rises — thirty-two of them, median 0.004 — at almost exactly the same placements. In every one of the sixty-two rises across the two, the reference just placed is one of the new worst system’s six nearest references. The rise happens where the neighbourhood changes, not somewhere else on the square.

The mechanism is the one the nearest reference had, one level up. A plane through the six nearest neighbours is a different plane for every set of six. As references are added, a held-out system’s six change, and when the new reference enters them the plane through them changes too — sometimes by tilting toward an answer that is right at the new reference and wrong a short distance away. The greedy placement puts its reference at the worst current miss, which is where the correlation energy is changing fastest, so the new reference is precisely the kind that tilts a neighbour’s plane. A lookup built from nearest neighbours of any kind is piecewise in the neighbour set, and cells of some kind are its price.

The leverage cannot see this. A neighbour set that has just changed is usually a perfectly good one, with modest leverage: the plane through it is well determined and simply different from the plane through the previous six. Leverage reports how uncertain the plane is about its own value, and these planes are certain and wrong in a new way.

Four planes on the same references: the guard costs nothing on the coarse square and removes the oracle's spikes. Worst and median held-out miss on the 81 coarse references, worst on the 17×17 grid, and the own-placement oracle's largest worst, number of rises and worst after 128, for each lookup: plane, unguarded: 0.088, 0.0050, 0.068, 3.992, 30, 0.0101; shape guard, nearest fallback: 0.088, 0.0050, 0.068, 0.117, 31, 0.0104; leverage guard, nearest fallback: 0.088, 0.0050, 0.068, 0.117, 32, 0.0275; leverage guard, wider plane: 0.088, 0.0050, 0.068, 0.117, 32, 0.0101.
Fig. 6 Four planes on the same references: their misses on the coarse references and on the 17×17 grid, and their own placement’s largest worst miss, number of rises and worst after 128.

How the guards were built and checked

The scaled readings, the references and the held-out pool are those of the essays on the square: the 81 coarse systems, the 1,008 systems of the finest grid that the coarse square does not contain, and the two readings each divided by its range over the coarse references. The plane is a least-squares fit of correlation energy to an intercept and two slopes, centred on the system, solved by elimination with partial pivoting; its value is the intercept, and its leverage the first diagonal element of the inverse of the 3 × 3 normal matrix, found by solving with the unit vector on the right.

The unguarded plane reproduces the published one to 10⁻⁹ at every system checked, and its own placement reaches 3.99 at the thirty-ninth step. The leverage of six points evenly spaced on a circle round the system was checked to be one sixth to 10⁻¹². Each guard leaves the coarse plane untouched, each guarded placement stays below the nearest reference’s coarse worst miss of 0.166, and the distance guard of the refusal — a threshold equal to the largest nearest-reference distance among coarse held-out systems, 0.125 in the scaled readings — fails to stop the spike. The oracle placements are greedy and deterministic, and the comparison between forms is between their own placements, each chasing its own worst miss.

What a guard on a ring of four cannot say

Every number here belongs to one model, a ring of four sites — the one system on the square with no threshold of its own to find — and to one reference layout, a nine-by-nine grid in U and ε. The gap of three decades between the spikes and the ordinary systems is a property of that layout: the spikes arise only when the greedy placement has put six references in one row of the grid near a system off the row, and a layout without rows — references scattered at random, or placed by a rule that avoids alignment — would put fewer systems at the extremes and more in between. Whether leverage would still separate failure from success on such a layout is not tested.

The readings are also exact here. A real composite reads its cheap quantities from an approximate calculation, with errors of its own, and leverage multiplies those errors as surely as it multiplies the scatter of the references. A system whose readings are uncertain sits at a range of leverages rather than at one.

Nor is the transferred quantity the only thing that can go wrong. Every score on the square carries one correlation correction fitted at a single reference system, and carried from elsewhere it changes the worst case by an order of magnitude; a guard on the lookup says nothing about whether the references themselves were the right ones to trust.

And the oracle placement is not a placement anyone can run. It knows every system’s correlation energy and spends that knowledge on choosing the next reference. What it shows is the worst a lookup’s own structure can do; a placement by physics or by a warning has different rises, and possibly fewer.

A lookup has two ways to be wrong

The guard separates them. One is the plane being asked a question its neighbours cannot answer, and that is announced in advance by a number the plane already computes: the guard costs nothing, needs no tuning on this square, and removes the only catastrophic misses the oracle ever found. The other is the neighbourhood itself changing under a system, and that is not announced at all, because a fresh neighbourhood looks exactly like a good one. The warning a cheap calculation gives and the spread among nearest references both read the answers of the neighbours, and neither was built to see that the neighbours have just changed.

The second kind is the smaller of the two by two orders of magnitude, and on this square no single rise of it exceeds about 0.05. It is also the only kind left.

Still open: a lookup that does not jump, and a guard that reads the readings’ errors

The obvious open question is whether any lookup built from references can be monotone under its own placement. A lookup whose answer changes continuously as a reference is added — weights that fall smoothly to zero at a radius, so that a new reference enters gradually rather than at once — would have no cells, and the question is whether its oracle then rises at all. If it does not, monotonicity is a matter of smoothness in the weights, and the plane’s rises are the price of its hard cutoff at six. If it still rises, the rises belong to greedy placement itself, which puts every reference where the surface is steepest, and no form of lookup removes them.

The nearer question is the readings’ own uncertainty. On this square the readings are exact. A composite that reads them from a mean-field calculation carries their error into the plane’s slopes, and leverage multiplies it. Adding the readings’ error to the guard — falling back when leverage times the readings’ variance exceeds the scatter of correlation energy among the neighbours — would turn the threshold from a number chosen above a gap into a number derived from the calculation, and say whether the gap survives once the readings are noisy.

Shares its objects with

Essays naming at least two of the same things, that neither author linked.

Named objects

A dashed tag is an object no other essay names yet.

Composite methodCorrelation energyHubbard modelMean-field approximationTransferability