Receive free Quant investing updates
We’ll send you a myFT Daily Digest email rounding up the latest Quant investing news every morning.
Is there a replication crisis in finance? Some prominent academics say yes, obvs, and it’s endemic. However, a high-powered, recently peer-reviewed paper argues this is bunkum.
Perhaps the whole “replication crisis” thing needs explaining first though, for those lucky enough not to spend their time reading academic papers and following the weirdly intense debates around them.
Back in 2005, Stanford medical professor John Ioannidis published a paper showing how the results of many widely-cited medical research papers couldn’t actually be replicated by other researchers. Which was obviously very awkward. Since then, swaths of academia have discovered the same thing in their fields, including finance.
Duke University finance professor Campbell Harvey has been one of the loudest and most prominent critics of his own profession. In 2021 he calculated that at least half the 400-plus market signals detailed in various top academic journals over the years can’t actually be replicated. Cue much mirth in some corners, and consternation in others.
However, the latest edition of the Journal of Finance contains a paper that argues that the financial replication crisis is actually a myth. Here is the abstract:
Several papers argue that financial economics faces a replication crisis because the majority of studies cannot be replicated or are the result of multiple testing of too many factors. We develop and estimate a Bayesian model of factor replication that leads to different conclusions. The majority of asset pricing factors (i) can be replicated; (ii) can be clustered into 13 themes, the majority of which are significant parts of the tangency portfolio; (iii) work out-of-sample in a new large data set covering 93 countries; and (iv) have evidence that is strengthened (not weakened) by the large number of observed factors.
The paper is written by Theis Ingerslev Jensen, Bryan Kelly and Lasse Heje Pedersen. Jensen is an assistant professor of finance at Yale, the latter two work for AQR Capital Management, the big quant investment shop run by prominent screen-smasher Clifford Asness, in addition to teaching at Yale and Copenhagen Business School. It was actually first published in 2021 by AQR, when some mainFT rube wrote about it here.
But the paper’s appearance in the Journal of Finance indicates that it has now gone through the intense peer-review process. Harvey edited the JoF — one of the top journals in the field — in 2006-12, and is a one-time president of the American Finance Association that publishes it. So it is somewhat ironic that a paper attempting to counter his criticism has now published there.
The paper is still worth resurfacing and revisiting, simply because it’s such an interesting and important subject.
While the consequences of data mining and spurious signals in finance are piddling compared to those in other fields — if a market signal is hogwash you just lose some money, but if medical research is wrong the results can be lethal — it’s obviously matters to people that read Alphaville.
There are two primary facets to the replication crisis. Firstly, that the results simply cannot be replicated, or secondly that they can be replicated but only by contorting or cherry-picking the data, something known as “p-hacking”.
Jensen, Kelly and Pedersen argue that “neither criticism is tenable”, and say that they’ve got the data to prove it:
The majority of factors do replicate, do survive joint modelling of all factors, do hold up out-of-sample, are strengthened (not weakened) by the large number of observed factors, are further strengthened by global evidence, and the number of factors can be understood as multiple versions of a smaller number of themes.
These conclusions rely on new theory and data. First, we show that factors must be understood in light of economic theory, and we develop a Bayesian model that offers a very different interpretation of the evidence on factor replication. Second, we construct a new global data set of 153 factors across 93 countries. To help advance replication in finance, we have made this data set easily accessible to researchers by making our code and data publicly available.
Prof Harvey remains unimpressed, however, even if he says that Jensen, Kelly and Pedersen’s replication results are in fact replicable. He just thinks it rests on an unreasonable assumption on how many anomalies are true. Here are some slides he prepared for a debate with the authors at last year’s annual meeting of the American Finance Association where you can see his counterargument.
We’ve gotta say that we feel a little unqualified to pass judgment either way on this. But it feels correct to say that the academic imperative to “publish or die” and top journals’ requirement for statistically significant findings have probably led to some data-mining (conscious or unconscious) and some pretty silly results have followed.
Or maybe we really need to ban cheese.
Read the full article here