Calling Bullshit on Hockey Stick Forecasts
Ever see an otherwise well-regarded industry analyst firm post a market forecast with an exciting hockey stick curve? These charts scream “Unlimited, exponential growth is just ahead! This is it, this is the market we should all invest in now!”
Well, everyone from PM’s to VC’s ought to acknowledge such non-linear predictions as mathematical BS and call them out as such. Now I’m not talking about system dynamics, physics, simulations or even queuing models where you have a resolute mathematical model with underlying exponential/differential formulas. I’m specifically referring here to complex systems like financial markets where predictions are being made based on regression of some kind – looking at past data to predict the future.
Why do the biggest research firm pundits get away with it? Maybe because it’s exciting now and no one ever holds them to it later? If you are susceptible to such chart junk, get your hands on the last 20 years of any market forecast reports with exponential curve forecasts and just compare them (internet archives can be useful here). Visually overlay them if you are handy with image manipulation. I actually ran into this first with a series of well-known market reports when I was a PMM doing research into a specific emerging IT market, and found that every 3-4 years the analyst firm was publishing the exact same future exponential curve as a “new” market prediction, just that the curve was pushed out 3-4 years in each successive report. It wasn’t even cleverly disguised – it was the exact same chart with the x-axis just relabeled each time.
That’s not to say a good non-linear forecast is uselessly wrong. It can serve to highlight that we expect a big change coming. It’s just that unlike linear forecasts1, there is no way to accurately or adequately predict the future curvy “shape” of an expected non-linearity from past data. An infinite number of future curves can be “fit” to any past data.
In many domains, the importance of predicting when non-linear behavior will start to happen can be critical. But even stating “when” something has gone or will go non-linear is problematic. One can’t really know when the steepest part of a non-linear curve will kick in. And don’t be fooled into thinking that you can mathematically predict how steep the hockey stick will be from past data.
I was reminded of this when my wife pulled a Nassim Nicholas Taleb book out of a stack in the corner of my office this morning. I realized I had never quite finished reading “Skin In The Game,” the fifth book in his excellent Incerto series on dealing with uncertainty, and that in this current environment of mistrust and misinformation I should put back it on top of my reading pile2. If you want actionable advice on how to best approach uncertainty and randomness, Taleb’s essays are where I’d recommend you start!