Adding data made it worse

A force field was projected onto the part a spectrum determines and reported what was left: about a tenth of spread in the worst component across four starting points, attributed to the eigenvalue problem rather than to the coordinate set. It said the attribution was not a measurement and named the test:

Refitting with two isotopologues instead of one, and watching whether the projected spread falls, would test it directly.

Run on methane, which has both in the library, it does not fall. The fit runs away.

The constant that was dropped #

Methane’s force field in this collection has four constants and not five, and the reason is recorded beside it. The six angles at a tetrahedral centre are not independent, so the bending and bend–bend constants have a direction along which they can both run without changing a single frequency — and allowed to, they did. The first fit came back with both at −97,596 millidyne per ångström, equal to four figures and opposite in effect, reproducing every frequency to half a per cent.

Dropping one of the two costs almost nothing in residual and returns a C–H constant of 5.47, which is the number methane is supposed to have. So the constant was dropped and the comment explains why.

Restoring it makes the test possible, because the runaway is the flat direction made visible, and the question is whether a second isotopologue closes it.

One converges and two do not #

Fitted to the light molecule alone, with all five constants free, the search converges cleanly. The residual is 2.06 × 10⁻¹³ — every one of the nine frequencies reproduced to the arithmetic’s own precision — and the constants are ordinary: a C–H stretch of 5.4248, a bend of 0.4067, a stretch–stretch of 0.0035, a stretch–bend of 0.3384 and a bend–bend of −0.0891.

Fitted to both isotopologues, it runs to −97,596.4 and −97,596.9.

So adding nine measured frequencies made the fit worse, and not by a little. The library’s own guard catches it: any constant past a hundred millidyne per ångström is outside anything in chemistry, since the stiffest bond there is carbon monoxide’s at about nineteen.

That is the opposite of what the proposed test expected, and it is worth being clear that the direction of the surprise is the finding. More data constraining a fit less is not a thing that usually happens, and when it does the explanation is not in the data.

The direction is flat at every mass #

The explanation is available without any fit, by moving a converged field along the flat direction on purpose and asking what happens.

Take the field the single-isotopologue fit converged to, and add a large multiple of the direction the determined-subspace calculation says no frequency can see. The move here is ninety times the field’s own norm and puts one constant at 289 millidyne per ångström — fifteen times the stiffest bond in chemistry, and an obvious nonsense.

Every frequency stays where it was. The worst relative change across the nine is 1.67 × 10⁻⁶.

And the heavy molecule gives the same number. 1.67 × 10⁻⁶ for CH₄ and 1.67 × 10⁻⁶ for CD₄, to three figures, which is not a coincidence and is the whole point: a redundancy is a combination of force constants that produces no displacement of any atom, and a mass cannot see a displacement that is not there.

The reason it holds at every mass is one line of the eigenvalue problem. A harmonic frequency is an eigenvalue of $GF$, where $F$ is the matrix of force constants in the chosen internal coordinates and $G$ is built from the atomic masses and the coordinates’ geometry. Changing an isotope changes $G$ and nothing else. A redundant coordinate set is one whose coordinates are not independent functions of the atomic positions, so some combination of them is identically zero for every displacement — and a force-constant combination supported on that null direction contributes nothing to the potential energy of any displacement whatever. It therefore contributes nothing to $F$’s action on the space $G$ multiplies, and $GF$ is unchanged for every $G$. The statement is about the coordinates rather than about the molecule, which is why no measurement of the molecule reaches it.

That also says which redundancies are dangerous. A redundancy is harmless if nobody quotes a constant it touches: the field predicts every frequency correctly and the arbitrary part never leaves the fit. It becomes a defect the moment a single constant is reported, because a single constant is a coordinate of the point the search stopped at, and the point was chosen by the simplex rather than by the spectrum.

So no isotopic substitution closes the direction, and the proposed test asks a question the geometry answered before any fit was run. The same is true of every other substitution: a heavier carbon, a mixed isotopologue, a molecule with one deuterium. All of them change masses, and the flat direction is a statement about second derivatives of the potential with respect to coordinates that describe no motion. Deuterium changes the masses in the kinetic energy matrix; the flat direction is a statement about the potential’s second derivatives with respect to coordinates that describe no motion, and the kinetic energy matrix never meets it.

Why the isotope method works everywhere else #

The result needs setting against the fact that isotopic substitution is the standard way of determining a force field, and it works.

Water’s four force constants cannot be got from its three frequencies, and a whole family of quite different fields reproduces all three exactly — which is the ordinary kind of underdetermination, a fit with more parameters than data. Adding D₂O adds three more frequencies and the family collapses to a point. That is the isotope method, it is sound, and nothing here disputes it.

The two situations look identical from inside a fit and are not the same at all. In the first, the objective function has a valley because the data are too few; the valley’s floor is flat because nothing distinguishes the points on it yet, and a new measurement tilts it. In the second, the valley’s floor is flat because the direction along it changes nothing about the molecule — not about its spectrum, not about its structure, not about any observable whatever — and no measurement of anything can tilt it.

The tell is which coordinates the flat direction runs along. Water’s is a direction in a four-dimensional space of independent constants and the data pick it out. Methane’s runs between a bend and a bend–bend constant whose combination describes no displacement, and it stays flat under deuteration, under any other substitution, and under any measurement of the molecule at all.

So the two cures are for two diseases. Adding an isotopologue cures too little data. It does nothing for a redundant coordinate set, and on methane it makes the search’s behaviour worse — because the wider valley of a nine-frequency fit stops the simplex before it has travelled far, and eighteen frequencies give it a sharper valley with the same flat floor and more reason to slide along it.

The residual is round-off #

The 1.67 × 10⁻⁶ deserves one more test, because a small number is not automatically zero and the difference matters here. If it were a small real curvature — the direction nearly flat rather than exactly flat — then more data would grip it, slowly, and the proposed test would be right in principle and merely weak.

Make the move ten times smaller. If the residual is round-off in a difference of large numbers, it falls by ten. If it is a genuine quadratic curvature, it falls by a hundred.

It falls by ten, on both isotopologues: 1.67 × 10⁻⁷ at a move of a tenth the size, and 1.67 × 10⁻⁸ at a hundredth. Exactly linear, which is round-off and is not a curvature.

That is worth computing rather than claiming, because the direction is exactly flat and the direction is very nearly flat have different consequences and the same appearance at one move size.

What the converged fit is worth on its own #

The single-isotopologue fit converged, and it is tempting to treat that as the answer — five constants, a residual at the arithmetic’s precision, a C–H constant of 5.4248 where the four-constant fit gave 5.47.

It is not the answer and the reason is the same reason.

The converged field sits somewhere on the flat line, and where it sits is a property of where the simplex started and how it walked. Nothing selected that point. The residual being 2 × 10⁻¹³ is not evidence that the point is right: every point on the line has the same residual, and the line is infinitely long. A reader shown those five numbers and told the residual would reasonably conclude the field is determined, and the five numbers include two that are not.

Which of the five, the projection says and the fit cannot. The C–H stretch, the stretch–stretch coupling and the stretch–bend coupling are fixed on their own; the bend and the bend–bend are determined only in combination. So three of the five numbers can be quoted and two cannot, and the fit prints all five to the same precision.

That is the practical shape of the whole argument. A converged fit reports constants; a converged fit to a redundant coordinate set reports constants and a position on a line nobody chose, with nothing in the output to say which is which.

The projection does not care #

Project the converged field and project the moved one, and the two agree to 5.8 × 10⁻¹³ in every one of the fifty-five components — against components as large as several millidyne per ångström.

They must, and the reason is what makes the projection the right repair. The projection removes exactly the directions no frequency can see, and the move was along one of them, so the two fields differ only in the part being thrown away.

Which settles what dropping a constant is. It is a repair to the search: it makes the objective function have a unique minimum by removing one of the two constants the flat direction runs between, and it does so by choosing which coordinate to lose. The projection is a repair to the answer: it reports the combinations a spectrum determined, and it gives the same report wherever the fit happened to stop.

The two are not alternatives with a preference between them. Dropping a constant changes what is fitted and therefore what the residual means; projecting leaves the fit alone and changes what is quoted from it.

What was computed, and how #

The two fits are the library’s own simplex search with the bend–bend constant added back and the same starting values. The flat direction is the eigenvector of the determined-subspace matrix belonging to a zero eigenvalue — the same construction used there — and the move is that eigenvector times a stated amount.

Nine things are checked: that the single-isotopologue fit converges; that the two-isotopologue one does not; that the converged fit returns a C–H constant near 5.42, which is the one the molecule is supposed to have; that the move is far larger than the field itself; that neither isotopologue’s frequencies move along it; that the test covers both, which is what says no substitution closes the direction; that the residual scales with the move rather than with its square; and that the moved field projects to the same point as the original.

The scaling check is the refusal. Without it the whole essay would rest on a number being small, and a number being small is what a nearly-flat direction and an exactly-flat one both look like.

Where this stops #

The runaway is a property of a search and this essay does not claim otherwise. A different optimiser, different starting values or different convergence criteria would stop somewhere else along the flat line, and none of that changes what the flat line is. What the two fits demonstrate is that the line is reachable and that adding data does not remove it — not that two isotopologues always run away where one does not.

The flat direction is exact only for a harmonic model in these coordinates. A real molecule is anharmonic, its frequencies depend on the potential beyond second order, and a combination of second derivatives that displaces no atom at the equilibrium geometry can still matter away from it. So the statement is about the model in use here, which is the model the force field belongs to.

And the projection’s own residual is still not explained. That tenth of spread across four starting points is not the flat direction, since the projection removes that exactly; it is whatever is left, and this essay has shown only that a second isotopologue is not the way to reach it. What it is remains open, and the candidates are the search’s convergence and the eigenvalue problem’s conditioning rather than anything about the molecule.

The generalisation #

The habit is to ask, before adding data to an underdetermined fit, whether the data can see the direction the fit is undetermined along.

More data narrowing a fit is so nearly always true that the exception is not looked for, and the exception has a clean signature: a parameter combination that produces no observable at all cannot be constrained by any observation, however many. Here the combination is a force constant direction that displaces no atom; elsewhere it is a gauge, a reparameterisation or an exact symmetry of the model. In every case the test is the same and costs nothing — move along the direction and see whether anything measurable moves.

The corollary is about which repair to choose. Faced with a fit that wanders, the available moves are to constrain the fit and to constrain the report, and they are usually presented as the same move. They are not. Constraining the fit — dropping a parameter, adding a penalty, fixing a value — changes what is being fitted and puts a convention into the residual. Constraining the report leaves the fit alone and states which combinations the data reached. The second is the one that gives the same answer from every starting point, which is the property being asked for when a fit is said to be unstable.

Who found it, and when #

The redundancy of a symmetric internal-coordinate set is Wilson, Decius and Cross, 1955. The isotope method for determining force fields is older and is the standard response to an underdetermined one — and the product rule it rests on is exact, which is why it works where it works. The force fields and the frequencies are fitted here to quoted data. What is computed here is the two fits with the dropped constant restored, the frequency change along the flat direction at three move sizes on both isotopologues, and the two projections.

The number worth carrying is 1.67 × 10⁻⁶, twice: the same figure for the light molecule and the heavy one, after a move that put a force constant fifteen times past the stiffest bond in chemistry.

Still open: a ring, and the residual that is left #

The obvious open question is the molecule where this argument would bite differently. Every redundancy here is among angles, so every flat direction is a combination of bending constants and every stretching constant is determined. A ring’s redundancy involves a bond length, because the ring has to close — so a ring would put a stretching constant into the flat space, and the isotope argument would apply to it unchanged while the practical consequence would be much worse, since a stretching constant is the one thing a spectroscopist actually quotes. Cyclopropane’s frequencies are measured and it has no force field here.

The nearer question is the residual the projection leaves, which this essay has narrowed rather than closed. It is not the flat direction and it is not the data. That leaves the search and the arithmetic, and those are distinguishable: tightening the simplex’s convergence should shrink a search residual and leave a conditioning one alone, and the determined subspace’s smallest non-zero eigenvalue says how badly conditioned the problem is near its edge. Both are one run each on what is already here, and between them they would say whether a tenth of a millidyne per ångström is a number worth reporting or a number worth removing.