The future of analytics: Testing folklore and intuition

Five years ago, the best-selling book "Competing on Analytics" made a case for the use of data to inform decision making. Despite evidence that analytics yields tremendous benefits across a range of industries and use cases, most organizations still don’t compete on analytics, said information systems department Chairman Michael Goul at the Deloitte Analytics Symposium, held recently in Phoenix. A sound first step? Developing a culture of experimentation, Goul said.

When Tom Davenport and Jeanne Harris wrote best-selling "Competing on Analytics" in 2007, they articulated the immense benefits companies can realize when they use data to inform decision making. Almost five years have passed, but most organizations still don’t compete on analytics, said Michael Goul at the Deloitte Analytics Symposium, held recently in Phoenix. “Especially with the recession, where companies have retrenched their expenditures, they’re not spending on analytics - but those who have are the ones surviving and even thriving today,” said Goul, who is chairman of the W. P. Carey School’s Department of Information Systems.

“The thing about analytics is that many companies think they know the right way to do things, but you never really know until you ask hard questions and look carefully to find evidence.” That “asking” is about using data to test hypotheses -- the intuition about the “right” decision -- so that the organization can be sure it’s operating on fact, not folklore. Goul offered an example: “When Edward Jones got its start in 1922, it operated only in rural areas of the country. The company’s CEO, Edward “Ted” Jones had decided that the most profitable operating model was to acquire new clients by operating exclusively in rural areas where there was little competition.”

In consultation with Peter Drucker, Edward Jones tested that folklore with an analysis, which revealed that the most profitable branches were actually those where there was competition. “Now where are Edward Jones branches?” Goul asked. “They’re in urban strip malls.” But only because the company allowed analytics to inform a strategy shift.

In the future, is human judgment out?

Goul joined five other experts in the panel discussion. In 10 years, will analytics have replaced human judgment? The consensus of the panel was that no, human judgment will still have its place -- even in a world where data informs all of our decisions. “Analytics won’t make the decision for you,” said Scott Mitchell, chairman and CEO of Open Compliance & Ethics Group.

“The key is thinking about how we use analytics to get at the interesting stuff. For all that, analytics will do nothing more than assist our decision making.” Said Jane Griffin, a principal at Deloitte, “Analytics is a tool organizations can use to become better decision makers.” Mitchell and Griffin agreed that humans are vital in the analytics process as askers of the questions that data can answer.“If I don’t know which questions to ask and how I’m going to apply the answers within my decision making, big data is just a lot of noise,” Griffin said. Added Mitchell, “The first question in my ‘Are you human?’ quiz is ‘Are you good at asking questions?’ If yes, you’re probably a human (and not a machine).”

So the role of analytics is not to replace human decision making, but to make it better. And, analytics are valuable in confronting human biases. “You overcome biases [like Edward Jones’s] by seeing that the analytics works,” said Wayne Winston, professor of operations and decision technologies at the Kelley School of Business at Indiana University.“The proof is in the pudding.” Said Mitchell, “It’s amazing how well our brains are wired to predictably make bad decisions in certain contexts.”

He did an experiment, asking everyone in the audience to visualize the last two digits of their social security number. Then to tell him how much they would pay for a very fancy pen that makes a digital record as the person writes. “The price of the pen is always higher the higher those last digits of a person’s social security number,” Mitchell explained. “It’s crazy, but it’s utterly statistically predictable.” Analytics programs, then, can be designed to break through cognitive biases like the one Mitchell demonstrated.

Where can analytics be applied?

“The services industry is one area where organizations can really benefit from analytics,” said Goul. “Whenever I talk to students I say that if you want to go out there and create something new, you need to think up new metrics to help gauge services.” In the healthcare industry, said Vasant Dhar, professor and head of the information systems group at New York University, the wealth of available data and better systems for analyzing it should allow healthcare providers to both improve their services and to lower costs.

Mitchell, whose organization helps enterprises understand and manage risk, said that analytics can help inform performance measurement in business. Corporate boards have always had a hard time measuring performance completely, Mitchell said. “Only now through analytics are board members able to truly understand at what risk organizations are able to meet classic performance metrics.”

Goul explained how analytics helped researchers determine what next generation software apps will look like. “We pulled 18,000 comments on the top ten productivity apps. We looked closely at 5,000 of those comments and ran a sentiment analysis tool [designed to identify and extract subjective information] that showed us the requirements that will go into the next generation of each app.” “No one had thought to use sentiment analysis for requirements, but there a lot of these examples of how we can apply analytics tools in ways we wouldn’t have thought of before,” Goul said.

So in addition to driving decisions in new ways in new industries, analytics tools can also be applied in new ways. Goul also described a case study he recently wrote about eBay. The company had created virtualized areas within their data warehouse -- sandboxes -- where departments can upload data and do their own experimentation to come up with new ideas. The ideas that work are rolled into production on eBay.com; those that don’t are scrapped, without a lot of time-already-spent waste.

The sandboxes, Goul said, are reflective of an “experimentation” culture at eBay. “They’ve developed a culture of analytics around experimentation for service innovation,” he said. That culture allows ideas to “fail fast” -- before the organization has infused lots of time and energy into them. “eBay is looking for metrics to quickly assess whether an experiment is failing -- to get it out and get a new one in. Those are metrics we haven’t thought about that need to be created,” Goul said. He added that using data to drive the experiments that allow enterprises to fail fast is “hard work” but represents a very exciting future for analytics.

How do we get to a future where we compete on analytics?

So companies still aren’t competing on analytics, yet doing so could yield tremendous benefits across a range of industries and use cases. How do we get from here to there? In part by developing cultures of experimentation within organizations, Goul said. “It is amazing how much folklore we operate on -- metrics that drive our decision making that we’ve simply accepted. We really need to step back and question whether those are the right metrics. Look at the data for answers. Have that experimentation culture to see if the metrics you’re using are the right ones.” That’s the future of analytics.