The force field is not in the spectrum

A spectrum is a list of numbers and a force field is a list of numbers, and it is natural to assume the first determines the second. Textbooks encourage it: a table of force constants appears, each with a bond beside it, and nothing in the presentation suggests the numbers were anything but read off.

Count them. Water has three vibrational frequencies. Its general valence force field has four constants — one for the O–H stretch, one for the H–O–H bend, one coupling the two stretches to each other, and one coupling a stretch to the bend. Four unknowns and three equations. The system is underdetermined before any experimental uncertainty is mentioned, and no amount of care in the measurement can fix it.

The family, computed #

The demonstration is direct. Fix the interaction constant at some value, refit the remaining three to water’s three observed harmonic frequencies, and see whether the fit succeeds. It does, over a wide range, and each success is a different force field that no measurement on H₂O can distinguish from any other.

Across the sampled range the fitted constants move like this. The O–H stretching constant runs from 8.256 to 8.455 mdyn per ångström. The bending constant runs from 1.042 down to 0.761 — a change of 37 per cent. The stretch–bend interaction runs from 1.011 to 0.236, a factor of four. Every one of those fields puts water’s bend at 1,648.5, its symmetric stretch at 3,832 and its antisymmetric stretch at 3,943, which are the observed values to the last digit quoted.

The bending constant of water is not determined to better than a third by water’s own spectrum. That is the finding, and it is not a statement about experimental error. The frequencies could be known exactly and the conclusion would not change.

The same exercise on a different constant gives the same answer with different numbers. Holding the stretch–bend constant anywhere from 0 to 0.4 mdyn per ångström and refitting the rest also reproduces all three frequencies exactly, with the interaction constant compensating between −0.118 and −0.101 and the bending constant between 0.751 and 0.788. Whichever direction the family is sampled in, the spectrum has nothing to say.

What breaks the tie #

A force constant is a property of the electronic energy surface, and the Born–Oppenheimer separation makes that surface independent of nuclear mass. So an isotopologue is a second molecule with the same force field and different frequencies — three more equations and no more unknowns.

Fitted to H₂O and D₂O together, water’s four constants come out at 8.456, 0.763, −0.100 and 0.248 mdyn per ångström, reproducing all six frequencies with a residual of seven thousandths of a per cent. Six numbers from four constants is now a real constraint, and the constants that come out of it are close to the ones spectroscopists have quoted for water since the 1950s.

The heavy line in the figure at the top of this page is what that constraint looks like from inside the family. Every field in the family fits H₂O; they predict D₂O’s frequencies wrongly by amounts running from 1.13 per cent down to 0.04 per cent, and the minimum sits at one particular member. The measurement that chooses is the one the fit did not see.

The family has an end, and it is informative #

The sampled range stops at −0.10, and it stops for a reason rather than by choice. Beyond about that value no assignment of the other three constants reproduces all three frequencies: the best available fit leaves a residual of a quarter of a per cent at −0.05 and half a per cent at zero.

Why the family ends is worth following. Water’s antisymmetric stretch belongs to a symmetry species of its own, and the only constants that reach it are the O–H stretching constant and the stretch–stretch interaction, in the combination k − k′. Fixing the interaction therefore fixes the stretching constant outright, through that one frequency. The two remaining frequencies must then be produced by the bending and stretch–bend constants alone, and the coupling between a mode at 1,648 and one at 3,832 is weak — the two are far apart — so there is a limit to how far the coupling can push them. Past that limit the required arrangement does not exist.

So the spectrum does constrain the field: it confines it to a curve in a four-dimensional space, and the curve has ends. It simply does not choose a point on the curve, and reporting one point as though the data had chosen it is the move this essay is about.

Two ways this fails silently #

The underdetermination becomes dangerous when the number of constants is large enough that nobody counts.

Methane, and a runaway. Its five-constant field, fitted to eighteen frequencies from CH₄ and CD₄, came back with the bending constant at −97,596 mdyn per ångström and the bend–bend constant at −97,597 — equal to four figures, opposite in effect, reproducing every frequency to half a per cent. The largest force constant in ordinary chemistry is carbon monoxide’s, about 19 mdyn per ångström. These are five thousand times that and they are meaningless.

The cause is a redundancy in the coordinates rather than an error in the fit. A tetrahedral centre has six angles between its four bonds, and they are not independent: five bending coordinates span every deformation there is. So the bending and bend–bend constants have a direction along which both can run without changing any observable at all, and a least-squares search walks along it for free. Dropping one of the two costs almost nothing — the residual goes from 0.379 to 0.401 per cent — and returns a C–H constant of 5.47, which is the number methane is supposed to have.

Formaldehyde, and a swapped assignment. Its seven-constant field reproduced all six observed frequencies to a tenth of a per cent, with a C=O force constant of 22.4 mdyn per ångström — three quarters again the largest bond force constant known — and with the two C–H stretches assigned the wrong way round. A least-squares fit compares two sorted lists of numbers and is blind to which mode is which; the residual was excellent and the answer was nonsense in two independent ways at once. No force field for formaldehyde is used here as a result.

The count the fitting problem is really against is how many distinct frequencies a molecule has against how many free constants its symmetry permits. A molecule with as many constants as frequencies is fitted rather than tested, and water is exactly that case.

Not every constant is equally undetermined #

The family is a curve rather than a cloud, and different directions along it are cramped by different amounts. That is worth measuring, because it turns a blanket warning into a ranking.

Across the sampled family the O–H stretching constant moves from 8.256 to 8.455 — a spread of 2.4 per cent. The bending constant moves from 1.042 to 0.761, a spread of 37 per cent. The stretch–bend constant moves by a factor of four.

The reason is which frequency reaches which constant. Water’s antisymmetric stretch belongs to a symmetry species of its own, so the only constants that touch it are the O–H stretch and the stretch–stretch interaction, in the combination k − k′. One measured frequency therefore pins one combination of two constants almost outright, and the stretching constant inherits most of that precision. The bending constant has to be inferred from a mode that also involves the stretches, through a coupling that is itself unknown, and it inherits the ambiguity of both.

So a stretching constant read off a spectrum is a much better number than a bending constant read off the same spectrum, and the difference is computable rather than a matter of judgement. Which is the practical form of this essay’s finding: not do not trust force constants, but compute how far each one can move before the spectrum notices.

Where the residual does mean something #

Three of the fitted fields here are constrained by more numbers than they have constants, and those are the ones whose residuals are measurements rather than tautologies.

Ammonia’s five constants were fitted to twelve frequencies — six from NH₃ and six from ND₃ — so its 0.46 per cent residual is a measurement of how far a valence force field of that form can be pushed rather than a restatement of the input. That is the difference between the two molecules in this essay.

Methane is the sharpest case: four constants against eighteen frequencies, and a residual of 0.40 per cent. Eighteen numbers reproduced by four is a real claim about the model, and it is the kind of claim a fit can only make when it is overdetermined.

Both sets were fitted together for ammonia, so the agreement between them is not a prediction. That is worth saying plainly: an isotopic substitution constrains a force field only if the substituted molecule’s frequencies were not among the data.

The pattern worth carrying is a ratio rather than a rule. Count the free constants; count the independent measurements; if the first is not smaller than the second, the residual is decoration. Water at four against six is barely constrained and its field is close to the literature’s; ammonia at five against twelve and methane at four against eighteen are genuinely tested; sulfur dioxide at three against three is not tested at all.

Zero residual is not a triumph #

Three of the six molecules with fitted fields here have as many free constants as they have distinct observed frequencies, and all three fit exactly. Sulfur dioxide: three constants, three frequencies, residual zero. Carbon dioxide: three and three, zero. Boron trifluoride: six constants and four distinct frequencies, zero.

Those zeroes carry no information whatever. A model with as many parameters as data points reproduces the data by construction, and the only honest thing to do with such a fit is to say so and to use the constants for what they are — a compact restatement of the spectrum rather than an explanation of it. Where a spectrum is drawn for those molecules, the interest is never in the agreement.

Water, ammonia and methane are the cases where the residual means something, because each is fitted to two isotopologues at once: four constants against six frequencies, five against twelve, four against eighteen. Ammonia’s residual is 0.46 per cent and methane’s 0.40, and those numbers are measurements of how far a simple valence force field can be pushed rather than of anything about the fit.

What does not move as the field moves along the family is the mode composition: water’s stretching modes are fifty per cent in each bond for every member of it, because that is fixed by symmetry rather than by any constant. The undetermined direction is invisible in every quantity symmetry decides.

The same shape of error, elsewhere #

The pattern — a quantity reported to three figures that the data determine to one — is not peculiar to force constants.

Electronegativity is not one quantity is the same argument about a different table: four scales, four definitions, a correlation of 0.99 quoted as though it settled something, and twelve pairs of elements out of a hundred and fifty-three ordered differently by two of them. What a lone pair is worth is the same argument about VSEPR’s second clause: the lone-pair weight fitted to water is wrong for hydrogen sulfide by a factor of two, which means it was never a property of a lone pair. In every case the failure is invisible while only one molecule is looked at.

The general form is worth stating plainly. A parameter that has been fitted is a summary of the data it was fitted to, until something it was not fitted to agrees with it. Fitting more parameters makes the summary tighter and the claim weaker, and it can be made so weak that the parameters stop referring to anything — which is what methane’s hundred-thousand did.

What this does not say #

It does not say force constants are useless. Water’s fitted field, once an isotopologue has pinned it, reproduces six frequencies from four numbers, predicts HOD’s spectrum without further adjustment, and gives mode compositions that agree with the symmetry analysis exactly. That is a working model.

Nor does it say force constants are arbitrary. The family has ends, and the region it occupies is a small part of the space; the O–H stretching constant varies by 2 per cent across the whole family while the bending constant varies by 37. A stretching constant read off a spectrum is a much better number than a bending constant read off the same spectrum, and the difference is computable.

What it says is that the uncertainty in a fitted constant is not the uncertainty in the measurement, and that the two are usually reported as though they were the same. The family in the figure at the top of this page is the honest version of an error bar for a quantity of this kind, and computing it takes about as long as fitting the field once.

Which molecules are determined, and the rule that decides #

Three frequencies, four constants is water’s arithmetic, and the general form of it is worth having, because it says in advance which molecules a spectrum can determine and which it cannot — without computing anything.

Symmetry blocks the problem. Modes of different symmetry species do not mix, so the force constant matrix is block-diagonal and each block can be counted on its own. A block of dimension $d$ contains $d$ frequencies and a symmetric matrix of $d(d+1)/2$ independent constants.

Setting the two equal gives the condition:

$$d(d+1)/2 \le d \quad\Longleftrightarrow\quad d \le 1.$$

A symmetry block of dimension one is exactly determined by its own frequency. A block of dimension two or more never is. There is no intermediate case, and no molecule is determined unless every one of its blocks is one-dimensional.

Water’s blocks are a two-dimensional one holding the symmetric stretch and the bend, and a one-dimensional one holding the antisymmetric stretch. Three constants against two frequencies in the first, one against one in the second — so the whole of the deficiency is in the $A_1$ block, and the antisymmetric stretch’s force constant is determined outright by its own frequency.

That is more useful than the total. It says which constants are safe to quote from a single spectrum and which are members of a family, and it locates the family in a block rather than leaving it spread across the molecule.

The rule also explains why the repair works. Force constants do not depend on mass, so a second isotopologue supplies a fresh set of frequencies against the same unknowns — new equations, no new variables — and the deficiency falls by however many independent frequencies the substitution adds. It is the only kind of extra data that helps: a more accurate measurement of the same three frequencies narrows the family without shrinking it, because the deficiency is structural rather than experimental.

And it says which molecules were always going to be difficult. Deficiency accumulates as $d(d-1)/2$ summed over the blocks, so a molecule with one large block is far worse off than one with several small ones — and a molecule with no symmetry at all has one block containing everything, $3N-6$ frequencies against $(3N-6)(3N-5)/2$ constants, which for anything larger than water is hopeless without isotopes.

Two consequences follow that are worth carrying past this molecule.

A published force field for a molecule of any size is a fit to more than its own spectrum. It has used isotopes, or constraints transferred from related molecules, or a calculation — and which of those it used is part of the result. A field quoted without that provenance is a member of a family whose dimension the reader could have computed from the symmetry in a minute.

And the deficiency is not reduced by fitting better. Every member of the family reproduces every frequency exactly, so no residual distinguishes them and no amount of care with the optimiser will. That is the difference between a quantity that is uncertain and one that is undetermined, and only the first is improved by a better measurement.

Where to read on #

The next question is the one measurement this calculation makes a real prediction about. Force constants do not depend on mass, so an isotopologue’s whole spectrum follows from a field fitted to another isotopologue with no further freedom at all — and there is an exact identity, the Teller–Redlich product rule, that the prediction must satisfy whatever the force field turned out to be. That is the isotope shift is arithmetic.

Further on, the same underdetermination reappears in a form that has a name and a large literature: group frequencies, and where they stop asks when a frequency can be attributed to a bond at all, which is the same question as which coordinate a mode is made of, and the answer is computed rather than tabulated.