No. It is extremely difficult – perhaps even pointless – to use statistical analysis to determine with any degree of confidence whether a fund manager has beaten the market due to luck or skill.
As professor Jeremy Siegel explains in Chapter 23 of the investment classic Stocks for the Long Run 5/E: The Definitive Guide to Financial Market Returns & Long-Term Investment Strategies.
The length of time needed to be reasonably certain that superior managers will outperform the market will most surely outlive their trial period for determining their real worth.
On the one hand, the “trial period” granted most active managers is very short. For example, the State Street Centre for Applied Research * asked 3,774 investors, investment providers, government officials and regulators across 19 countries the following question.
If you followed the investment process of your firm but underperformed your peers, do you believe that here would be negative consequences to your job safety? Yes, I assume that I would be made redundant / fired after _ months of underperformance; No, regardless of how many periods of underperformance.
Most investors surveyed believed that they would be sacked after 18 months of under-performance!
On the other hand, the period required to differentiate between luck and skill is much, much longer than 18 months. This can be seen in the table below taken from the 5th edition of Stocks for the Long Run.
This table assumes that fund managers have a beta = 1 and a correlation = 0.88 to the stock market, which is consistent with the historical performance of US mutual funds since 1971. The columns represent different holding periods from 1 through to 30 years, while the rows show different levels of expected out-performance.
Let’s assume that an investor can select a portfolio of stocks that has an expected return that’s 1% better than the stock market. Even if they can sustain the 1% out-performance for 10 years, there’s only a 62.7% chance that their 1% per year out-performance is due to skill rather than luck.
As the table shows, even when an investor has beaten the market by a wide margin, it takes at least 10 years to be reasonably certain – more than 70-80% sure – that it was due to skill and not luck.
Investors with short attention spans (e.g. 18 month) risk hiring and firing the wrong managers several times over. Siegel provides a vivid example of how this might happen.
Assume you gave a young, undiscovered Peter Lynch – someone who over the long run will outperform the market with a 5 percent per year edge – an ultimatum: that he will be fired if he does not at least match the market after two years. Table 23-3 shows that the probability he will beat the market over two years is only 76.1 percent. This means that there is almost a 1-in-4 chance that he will still underperform the market and you will fire Lynch, judging him incapable of picking winning stocks!
Even the best fund managers will experience periods of under-performance (see my earlier post on short term under-performance). Meanwhile, an investor waiting for statistical certainty risks losing a lot of money as Siegel explains.
Detecting a bad manager is an equally difficult task. In fact, a money manager would have to underperform the market by 4 percent a year for almost 15 years before you could be statistically certain (defined to mean less than 1 chance in 20 of being wrong) that the manager is actually poor and not just having bad luck. By that time, your assets would have fallen to half of what you would have had by indexing to the market.
The table above shows that, all other things being equal, the greater the level of out-performance, the shorter the time period that’s required to establish whether or not the result was due to luck or skill. So why don’t fund managers all aim to hold portfolios that will beat the market by five percent or more each year?
The simple fact is that creating a portfolio of stocks that beats the market by 5% over the long-term is very hard to do. We can demonstrate this using a simple formula for assessing the breakeven return for active management **
= [(Turnover × Cost) + Management Fee + Target Return / Market Return.
Let’s assume that:
- US Equities earn the historical 9.86% (US equity returns from 1967-2010).
- Target return = market return +5%
- Management fee = 0 bps (we’re looking at gross returns in this example).
- Turnover = 66% (that is an average holding period of 18 months. The turnover of active managers is often much higher).
- Trading costs = 25 bps (includes market impact).
This means that, to beat the market by 5% gross of investment management fees, our fund manager would need to earn a return that is equal to 150.87% of the market return.
How many stocks do you think there are in the market that offer a total return that’s more than one and a half times higher than the market? My guess is that it’s not many. A handful at most. Our fund manager is probably going to struggle to get anywhere near 5% outperformance unless they limit themselves to managing a reasonable sum of money in a concentrated portfolio that probably looks nothing like the market benchmark.
Research suggests that approximately 10% of fund managers invest this way (see my earlier post on what market-beating fund managers have in common). Sadly, the other 90% are more concerned about losing their job in 18 months or less!
So, if we can’t rely on statistics, how might we tell if an investor has skill or if an investment strategy will work? We can take Warren Buffett’s advice and investigate the methods of market-beating investors to see what they have in common.
In 1984 Buffett wrote an article for Hermes, the Columbia Business School magazine, entitled The Superinvestors of Graham-and-Doddsville. The article was based on a lecture that Buffett presented at Columbia to celebrate the 50th anniversary of the publication of Benjamin Graham and David Dodd’s investment textbook Security Analysis.
Buffett reasoned logically that if a group of individual investors, each with a different investment process and a unique portfolio but a shared investment philosophy could beat the market, then their results might be due to more than just luck.
He made this point by giving the example of a national coin flipping contest, where 225 million Americans each bet $1 on the outcome of a coin toss. Each person who calls correctly, wins a dollar from someone that calls incorrectly. Each day the losers drop out and the games is repeated, but the stakes are increased.
After 10 days, there will be 225,000 people who will have correctly called 10 coin flips in a row, each having won a little over $1000. After 20 days, there will be 215 people who will have correctly called 20 coin flips in a row, each of them having earned over $1 million dollars.
Overall, the game has created no value. The original $225 million dollars that was invested at the start of the game has simply been re-distributed from the winners to the losers. In other words, much like investment, the contest is a zero-sum game.
Anyone with an understanding of statistics would quickly realize that, with a population of 225 million, the laws of probability suggest that there would be 215 successful 20-in-a-row coin flippers. In other words, there’s no way to tell if the 215 winners were lucky or skilled, since we’d expect there to be 215 winners randomly distributed across a population of 225 million.
In fact, Buffett with his characteristic use of hyperbole and humour suggests that we’d get 215 winners even if we played the same game with 225 million orang-utans! But here’s where things get interesting. What if most of the winners shared one or more common characteristics? Would we be so quick to come to the conclusion that they were simply lucky? Buffett explains.
I would argue, however, that there are some important differences in the examples I am going to present. For one thing, if (a) you had taken 225 million orang-utans distributed roughly as the U.S. population is; if (b) 215 winners were left after 20 days; and if (c) you found that 40 came from a particular zoo in Omaha, you would be pretty sure you were onto something… That is, if you found any really extraordinary concentrations of success, you might want to see if you could identify concentrations of unusual characteristics that might be causal factors.
What are the attributes that some of the best “coin flippers” in the investment world have in common? Many of them share the same investment philosophy.
I submit to you that there are ways of defining an origin other than geography. In addition to geographical origins, there can be what I call intellectual origin. I think you will find that a disproportionate number of successful coin-flippers in the investment world came from a very small intellectual village that could be called Graham-and-Doddsville. A concentration of winners that simply cannot be explained by chance can be traced to this particular intellectual village…
… In this group of successful investors that I want to consider, there has been a common intellectual patriarch, Ben Graham. But the children who left the house of this intellectual patriarch have called their “flips” in very different ways. They have gone to different places and bought and sold different stocks and companies, yet they have had a combined record that simply cannot be explained by random chance…
…The common intellectual theme of the investors from Graham-and-Doddsville is this: they search for discrepancies between the value of a business and the price of small pieces of that business in the market… Our Graham & Dodd investors, needless to say, do not discuss beta, the capital asset pricing model or covariance in returns among securities. These are not subjects of any interest to them. In fact, most of them would have difficulty defining those terms. the investors simply focus on two variables: price and value.
Buffett goes on to consider in detail the long-term performance of 9 intellectual residents of Graham-and Doddsville; all having beaten the market by a wide margin. In his concluding remarks, Buffett again emphasises the common linkage between each of the superinvestors of Graham and Doddsville.
While they differ greatly in style, these investors are, mentally, always buying the business, not buying the stock. A few of them sometimes buy whole businesses, far more often they simply buy small pieces of businesses. Their attitude, whether buying all or a tiny piece of a business is the same. Some of the hold portfolios with dozens of stocks: others concentrate on a handful. But all exploit the difference between the market price of a business and its intrinsic value.
In summary, when it comes to distinguishing between luck or skill statistics are of little use unless we can afford to wait at least 10 years to be reasonably confident that our answer is correct.
While it may lack the apparent objectivity and mathematical rigour of statistical analysis, an arguably more effective way to decipher the luck vs. skill puzzle is to use common sense and look for shared attributes that have been proven to work over the long-term.
P.S. You may be wondering, where do the superinvestors of the Graham and Doddsville school hunt for investment opportunities? This will be the topic of a future post.
** Charles Ellis – The Loser’s Game