Stats About Statistics
Be humble about things that aren’t stable
Presentations of how various investment asset classes have moved as compared to each other are common. Illustrations of standard deviations and correlations drive trillions of dollars in allocations to various asset classes and investment products.
Behind these investment risk and return charts are statistics. It’s good to review statistics but it is also important to understand their limitations, especially in the financial world where the inputs used in models can be unstable.
To dive deeper into statistical issues that investors should keep in mind, we talked again with our friend Joachim Klement. For those who don’t know him, Joachim is a trustee of the CFA Institute Research Foundation and was previously, the head of the UBS Wealth Management Strategic Research team.
Beyond being a well respected investment researcher, Klement also has a deep background in statistics, having studied mathematics and physics at the Swiss Federal Institute of Technology, graduating with a master’s degree in mathematics.
The following are excerpts from our talk.
McSwain
Thank you for spending time with us on this.
Like you, I’ve been in the investment business for the better part of 30 years. During this time, I’ve seen many presentations that use the return streams of investment strategies and asset classes to create standard deviation and correlation charts that illustrate how one type of investment may move as it relates to another.
When pondering how they are presented, I often think back to early statistics classes that I took in undergrad when studying engineering. I don’t have near the background in this that you do, but I’ve been noodling this around for quite some time wondering about the proper, or maybe sometimes improper, use of statistics in our industry.
Klement
We both feel the same unease, as what we’re doing in finance is using concepts that were not designed for this purpose. They are sound mathematically, but were developed for natural sciences.
As an example, regression analysis was developed to investigate the motion of the planets around the sun and to test Newton’s law of gravity. That’s the origin of linear regressions.
McSwain
Designed for laws of nature.
Klement
Yes. Based on the work of Kepler and Newton’s law of gravity.
Natural systems have stability and gravity is a constant. When using these stable inputs, the outcomes of statistical models can be reliable.
Some natural systems can change, of course, but they change very slowly over millions and billions of years.
The problem when we use statistics in the financial markets is that the interactions between variables are constantly changing.
McSwain
This is what has been on my mind for years.
When we apply statistics to the financial world, we can create models that are not stable.
Klement
Correlations in the financial and investment world are never stable.
We might get some stability by accident and happen to find that standard deviation and correlation match up to the long-term average, but we only know this years later – not in advance.
McSwain
If you play this out, it makes you step back, especially related to correlation charts that can drive large investment allocations.
As you and many have written about, financial market inputs used in investment statistics may have more to do with human emotion than anything else, and us humans are not always emotionally stable about things related to money.
Klement
This gets into George Soros’s notion of reflexivity. The moment people think they understand the market, they change their behavior. This then changes the market – again. It is a constant race between people changing to try to catch something that is changing.
McSwain
The only constant in the financial markets might be change.
Related to how these changing inputs are used, when speaking to groups I often ask this question:
What is the most important word in the term Modern Portfolio Theory?
Theory.
We hope that all of our theories and hypotheses can be valid but it is wise to remember that they are not laws of nature.
Klement
Harry Markowitz has talked about this before. So has Bill Sharpe, as did Jack Treynor.
All were well aware that their work was based on rough approximations. They knew that they were using volatility, for example, as a concept because it was easier to deal with based on the computing power in the 1950s and 60s.
Then along came the 1960s and 70s, and somehow Modern Portfolio Theory (MPT) got morphed. Everyone in the industry started to apply it as if it was a law.
All know it has problems, but it has become almost canonical.
In reality, however, output from MPT models is just a first approximation.
McSwain
This does not mean that it is not useful.
Klement
Correct.
Approximations can be useful, as long as you just treat them as such.
McSwain
In putting together these statistical approximations, you need to have a series of observations.
For a model to be reliable, you need to have a lot of observations, though. This is why natural systems work well, correct?
You have millions or billions of years of observations.
Klement
Yes.
For models to be reliable, like a law, you need a lot of observations.
In addition, the observations should be close together in series.
McSwain
Back in undergrad financial stats classes, I was taught that the further away you get from daily observations, the more unreliable a model becomes.
Let’s talk a little more about this, though, especially your second point about the connections of time periods or a series.
Klement
When you look at the correlation of an investment asset class to another using monthly, annual, or three- to five-year returns they can change dramatically.
Next, you need to keep in mind issues that can occur with projections that are made using overlapping time periods.
If you only have a few years of rolling monthly returns your correlations might look much more stable than they really are.
The problem is that you just don’t have enough data that represents truly different time periods, not to mention that, as we previously discussed, for stats to be more reliable, you need to be able to analyze decades and decades of data.
McSwain
We could quickly go down a technical rabbit hole here, but how can we explain this in lay language?
Klement
Not easy for sure.
But how about this…
Many professionals in our industry like to look at three-year rolling periods of performance. This can be useful to spot trends but it can be problematic when running correlations.
As an example, assume that your first three-year period runs from January 2000 to the end of December 2002.
Some might then roll this forward monthly, with the next time period being February 2000 to January 2003, and so on.
The problem is that when doing this you create overlapping time periods that contain much of the same information.
Using our same example, if you start in January 2000 and go to December 2002, and then you go one month forward from February 2000 to January 2002, the data is identical for 35 of the 36 time periods.
Recycling can be good as it relates to the environment but, when running investment statistics, it can be bad.
McSwain
This gets to what we talked about off-line. The differences between stability and reliability.
Klement
Exactly.
Using overlapping periods can make things look more stable.
But…
It can make correlation charts less reliable.
Back to our previous example, unless you have a lot of three year periods, and take care to analyze the data properly, you are going to get models that make the correlation between two asset classes look stable. You just have too many overlapping periods and not enough new information.
McSwain
From my perspective as an adviser or allocator, this is where it gets tricky.
If you aren’t careful to look closely at how the data in a presentation is being put together, you might think you are making a decision based on something that is reliable when it is not.
As we’ve all talked about before, you can use data to fit the narrative that you want to promote.
Klement
That’s the problem with many new product or new asset class presentations.
How often have we seen wonderful looking, new product back tests that, when applied to real markets, look very different?
McSwain
Plenty.
Klement
Usually, this issue is put down to data mining, which can be a problem, but that is not always fair.
Back to where we started, in our industry we are just not working with stable data. This should be a first principle that we talk about more often.
McSwain
Agreed – again.
We want to be careful about this because it is good to keep our eyes open to new ideas and to support innovation.
Often the time frames we are working with, however, are just too short to be making some of the proclamations that we tend to make in our industry.
As I’ve often joked, we are very good at making theories and hypotheses sound like laws of nature.
We don’t have millions or billions of years of data like we do with laws of nature, but to be helpful and work with what we have, how many years of data, which would ideally include monthly observations, should an investor look for when reviewing correlation charts or models?
Klement
If you would have asked me this a few years ago, I would have said at least two full economic cycles, which might cover 20 or more years.
The problem, though, is that cycles can be odd. Take the last two for example. A global financial crisis and now a global pandemic. If you just looked at these two, how representative are they?
McSwain
Yes.
This last equity market cycle was up, down, up, all in about 12 months or less.
Klement
Exactly – again.
This last cycle might not have any predictive power at all unless there’s another pandemic.
Similarly, the financial crisis might have no predictive power unless we have another one that is caused by the exact same things.
McSwain
What I heard was this.
It is not only the number of cycles or time periods, but also how unique the time periods are.
Klement
Yes.
McSwain
If you listen to us too much, you might start to throw your hands up.
How about trying to come up with some rules of our own related to all of this?
Klement
This is tricky, as we will be proven wrong for sure.
But, how about these?
First, if something is being presented with too much conviction in our industry, beware – caveat emptor.
Next, and importantly, be humble. No one knows how the markets really work.
McSwain
Indeed.
Klement
I think this leads me to my first true rule.
Never be too sure about anything. The markets will always throw something at you that you didn’t expect or see coming.
Some people think they are better at seeing things coming than others but in general, we’re all pretty rubbish at that.
I think this one might actually stand the test of time.
McSwain
This all provides investors with a lot to ponder related to the reliability, or maybe unreliability, of some investment correlation and statistical models.
Any final thoughts?
Klement
I would again say, stay humble.
Also, don’t be dogmatic in the sense that the way things have worked in the past will be the way they work in future.
McSwain
Well, said.
Thank you, Joachim.
Related Reading:
The Double Life of Correlations – Kevin Kneafsey, Phd – Schroders
Are We Baking Portfolios with Bad Ingredients? – Tommi Johnsen, Phd
Fallible Forecasts? – Maneesh Shanbhag, CFA
Don’t Put Yourself in a Corner – Joachim Klement, CFA
Joachim Klement, CFA
Staying Disciplined and Avoiding Unforced Errors 
