blog-professional-flex-affordable.jpg

Pitfalls inherent in ranking financial professionals

From business schools to baseball batters, we use rankings to determine who's the best. Investors, too, look to rankings to assess the performance of financial professionals. The market has responded by providing several indices to gauge the success of funds, but investors need to look closely before deciding which ranking to use when making decisions. The indices can be misleading if you don't know what they are measuring.

"Past performance may not be indicative of future results."

Even a passing acquaintance with investing burns this boilerplate phrase into one's consciousness; it is the warning label of choice on advertising for financial instruments of all kinds. "Consult a financial adviser before investing" is a common follow-up. Finding and refining the mathematical and statistical models to assess the performance of financial professionals, however – portfolio managers at the top of the list – is a complicated and vexing problem.

Recent work by finance professors Federico Nardari and Jeffrey L. Coles of the W. P.  Carey School of Business and Naveen D. Daniel of the Robinson School of Business at Georgia State University, points up the importance of applying the right model and the right benchmark to each evaluative situation.

Their recent paper, Do Model and Benchmark Specification Errors Affect Inference in Measuring Mutual Fund Performance?, analyzes the degree to which failing to use the appropriate yardstick impacts the usefulness of studies or rankings of funds or of fund manager performance.

The study focuses on the three best-known performance evaluation models — those developed by Treynor and Mazuy (1966), by Jensen (1968), and by Henriksson and Merton (1981) — and two commonly used benchmarks: the S&P 500 and the CRSP value-weighted indices. The authors use Monte Carlo simulations to test what happens when the wrong model and/or the wrong benchmark is applied.

"The Jensen model," according to Professor Nardari, "assumes that the portfolio manager wants to beat the market, but isn't trying to time the market." Henriksson and Merton (HM) and Treynor and Mazuy (TM) involve market timing but use different strategies.

"One predicts the direction of the market return — meaning positive versus negative — and then adjusts the portfolio accordingly," he continues. "The other forecasts the magnitude of the market move and then adjusts the portfolio accordingly."

"The HM model is only interested in the direction of the market return. If the market is expected to go up, you want to be in the market; if the market forecast is for a decline, you want to be out. The TM model is concerned as well with magnitude, with how big the positive return or negative return is going to be."

The use of benchmarks in measuring the performance of fund managers — both in-house and as a way of having outside analysts rank fund managers — is even more straightforward. "Say you try to time and beat the Dow Jones and I try to beat the S&P 500," Nardari says, "we have different benchmarks." Similarly, "if they measure me with the S&P 500 and I'm not really trying to beat the S&P 500 – well, clearly there's a problem."

Sports provide another way to look at this problem. Born in Italy, Nardari evinces a statistician's affection for the endless — and endlessly idiosyncratic — ways in which baseball aficionados chronicle, tally, and rank player performance. How do we decide who "the best hitter" is? he asks rhetorically. It is probably only a minor exaggeration to say that you could look up "the best left-handed hitter ever, missing his middle finger."

"If we measure performance in that way," he suggests — if that's the benchmark we choose, "he's the best, in one sense." That still doesn't make him Babe Ruth.

"Similarly," he adds, "rankings of NBA players from major news web sites are based on a composite scheme — the "model" — relying on points scored, rebounds, assists, minutes played, etc. Altering the model may dramatically alter the rankings." This research points to a number of conclusions and paths for future investigation, some of it already under way.

First, the authors want to raise a warning flag on the value of fund manager rankings.

"You have to be very much aware of how misleading those rankings can be," Nardari cautions. "You can't rely too much on them. First of all, they may not be measuring the right thing. Even when they do measure the right thing, they don't really measure it that precisely. Someone that is classified in the top 20 percent and someone classified in the bottom 20 percent are not really that different.

We don't really have enough data to make statements about differences in performance. The fact that one portfolio manager ended up being number seven and that another ended up being number 81, doesn't really tell me that they are very different in terms of their abilities."

The question of how much data is available to the investing public and how accurate it is raises policy questions about what kinds of disclosure we might want to require from investment companies. Some of this treads on sensitive ground; there is a question of balancing the consumers' need to know with the portfolio managers' right to keep their working methods and strategies private.

"Disclosing your benchmark should not be such a terrible thing," Nardari argues.

"Disclosing the timing strategies is a more sensitive matter; that gets into how you really buy and sell stocks." He points out, however, that greater disclosure — at least of benchmarks — would entail benefits for at least some portfolio managers, as well as for people seeking to more accurately evaluate their work.

"If you're not measured the right way," he notes, "you may be a good manager but you don't get recognition for it. This [greater disclosure] is a way of leveling the playing field. If you're measured the right way, you will attract more resources, more fees, more money from investors. There would be a benefit on the investment company side."

Of course, he concedes, some people have doubtless benefited from being mis-measured. and "people who have consistently derived benefits from the lack of disclosure may be motivated to fight for the status quo."

Looking to the future, the same team of authors is about to embark on a survey project to collect and analyze data that will allow them to better understand what is actually going on in investment companies. They will be asking portfolio managers — who will be granted anonymity — what benchmark they time and what timing strategy they follow.

Getting a solid, representative data set in that manner, and on so sensitive a topic, is difficult. In one recent study, Nardari reports, financial professionals were asked to rate the risk premium of being in the U.S. stock market. The rate of response was under 25 percent — even though, in that instance, the only potential downside would be the embarrassment of being too far off the mark.

"This is one way we see of adding value to our current research," Nardari says. "The current research raises warning flags. We're saying, 'Beware: You are not really seeing things the way they are.' In order to do more — to really show how things are – well, that's going to take more work."

Latest news