Cold tests: killing your own result first

Cold tests are borrowed from detectors: before you trust an instrument on unknown signals, you run it on known ones, synthetics with planted answers and deliberate nulls, and you confirm it finds what is planted and reports nothing when there is nothing. The theory version is newer and, in this programme, it caught more than the real analyses would have. This post is about running pipelines on answers you already know.

The two cold runs

A cold test has two halves, and both must pass. The null: feed the pipeline data with no signal, synthetic data generated under the competing model, and the pipeline must report no detection. The recovery: feed it synthetic data generated with a planted signal of known parameters, and the pipeline must recover the parameters within stated tolerances. A pipeline that passes the recovery but fails the null has a false-alarm problem: it will find your signal in noise. A pipeline that passes the null but fails the recovery has a sensitivity problem: it will miss the signal when it is real. Both failures are common, and neither is visible from inside the code, because in both cases the pipeline runs and produces a plot.

The programme’s cosmological scaffold is the worked example, and the cold tests are written as requirements, not aspirations: recovery of the standard cosmology to a tenth of a per cent when the framework’s amplitude is set to zero, that is the null. Grid convergence to show the recovery is not an artefact of sampling. Fixed seeds so that any run can be replayed and challenged. Versioned data so that a challenge can be run on identical inputs.

What the cold tests caught before any data arrived

Three things, all real, none of them cosmetic. The scaffold, as first written, referenced fields in the cosmology codes that do not exist, meaning the pipeline would have crashed or, worse, silently taken defaults. Its parametrisation could go negative, producing an unphysical spectrum where a feature hunt would find monsters. And its interface to the sampling engine had never been exercised, so the first real chain would have been the first test of code that had never run end to end. Every one of those would have been discovered mid-analysis, at the worst possible time, when the temptation to patch quietly is highest.

The general rule

The deep pattern: surprise is not evidence. A result that emerges from a pipeline that has never seen a known answer has no denominator, no way to say how often such a result appears when there is nothing there. The cold runs supply the denominator. In the programme’s lab protocol, the same logic appears as the blinded correlation analysis with pre-registered thresholds: the analysts see synthetic injections and blank patches mixed with real data, and the statistic’s behaviour on the blanks is what licenses any claim about the reals.

The cost of cold testing is real but front-loaded: days of work before any result exists, on code that would otherwise have run immediately. The payoff is that when the real analysis produces a number, the number is a measurement instead of an anecdote. The archive’s whole method compresses to this: earn your surprises by buying the denominator first, because a surprise you have not paid for is indistinguishable from a bug.

The series that follows is the record of what these tests and audits found, beginning with a number I chose rather than derived: I chose the exponent.