Can you prove chocolate is good for you?

Why a study of nothing finds something

The usual test calls a result significant when chance alone would produce it less than one time in twenty. That is a fair bar for one test. It is a poor bar for eighteen. Each measure is a ticket in a lottery that pays out false alarms, and with eighteen tickets the chance of at least one winner is about 60%.

In 2015 the science journalist John Bohannon ran this game for real. Sixteen volunteers came, and fifteen finished. Some went on a low-carb diet, some on the same diet plus a daily bar of dark chocolate, and some changed nothing. The team took eighteen measurements, among them weight, cholesterol, sodium, blood protein, sleep quality and well-being. The chocolate group lost weight 10% faster. A journal printed the paper, and the story made the front page of Germany's Bild and the news in more than 20 countries. Bohannon then explained the trick himself: "Think of the measurements as lottery tickets."

Food and cancer works the same way. Jonathan Schoenfeld and John Ioannidis took 50 ingredients from random recipes in a cookbook and looked up recent studies on each. Forty had studies on cancer risk. Of 264 single studies, 103 said the food raised the risk and 88 said it lowered it. When the studies were pooled, the effects shrank to about nothing.

The same game in a trading room

A backtest is a study with a trading rule in place of chocolate. Each setting you try is another ticket, and the best one always looks good on the years it was picked on. In 2014 David Bailey, Jonathan Borwein, Marcos López de Prado and Qiji Jim Zhu put a number on it. With five years of data, trying more than forty-five independent settings all but guarantees a backtest with a Sharpe ratio of 1 that will earn nothing after. They also wrote that a backtest which does not say how many trials were run cannot be judged at all.

The same year Campbell Harvey, Yan Liu and Heqing Zhu counted the hundreds of papers that claimed a new way to explain stock returns. Given all that searching, they argued, a new factor should clear a t-ratio of 3.0 rather than the usual 2.0, and "most claimed research findings in financial economics are likely false." Our lab gives the rule only two years to prove itself. Its 90 settings overlap a lot, and half are mirror images of the other half, so they act like fewer tickets than Bailey's sum counts, but the ending is the same: what it earned before tells you almost nothing about after.

Sources

1. John Bohannon, "I Fooled Millions Into Thinking Chocolate Helps Weight Loss. Here's How.", io9, 27 May 2015.
2. Jonathan D. Schoenfeld and John P. A. Ioannidis, "Is everything we eat associated with cancer? A systematic cookbook review", American Journal of Clinical Nutrition 97, no. 1 (2013), 127–134.
3. David H. Bailey, Jonathan M. Borwein, Marcos López de Prado and Qiji Jim Zhu, "Pseudo-Mathematics and Financial Charlatanism", Notices of the American Mathematical Society 61, no. 5 (2014), 458–471.
4. Campbell R. Harvey, Yan Liu and Heqing Zhu, "…and the Cross-Section of Expected Returns", NBER Working Paper 20592 (2014).
5. The study and the prices are made fresh in your browser. Each measure compares eight chocolate eaters with eight others, all drawn by chance with a known spread. The rule is long when a fast moving average is above a slow one and short when it is below, or the reverse, on daily prices from a fair random walk. The 60% is an exact sum; the averages over 100 labs are simulations.