. . . And the role of intuition

The big data revolution is requiring a seismic shift inside organizations, both in the way we build relationships and the way we make decisions. Each is now driven by data rather than intuition.In general, decision making is accomplished via a thought process of selecting a logical choice from the available options. In a technical sense, when one makes a decision, all of the positives and negatives of each option are weighed against each other and/or a set of criteria, as well as all other considered alternatives. This process can be rational or irrational, and is often influenced by the decision maker’s utility and attitude toward risk.

In many cases, decisions seem to be plucked out of thin air, and the decision maker expects affected parties to trust his or her “intuition.” In coming up with a decision, the decider is, in effect, relying on past experiences to feed his or her subconscious, in an attempt to run algorithms in the background on all of the pros and cons of the options. This individual’s gut feeling may or may not prove to be correct. If put on the spot, the person might try (after the fact) to justify all subsequent actions by plugging in data favorable to the decision that has been made.

Intuition or data analysis?

Thus, what determines whether a decision is made based on intuition or data analytics depends on the sense of urgency, the decision maker’s utility, and/or what is at stake. The value of data-based decision making, however, is becoming more and more clear. This is true even if the initial opinion might have been triggered by intuition.

Quality professionals have been using data to track and solve quality-related problems for years. Marketing professionals also know the importance of gathering data, analyzing it, and creating models to help them improve their effectiveness. Sports enthusiasts can rattle off player team statistics, and coaches make data-driven decision in their player selections and in-game strategies. The baseball industry has a special term for their data, “Sabermetrics,” a term which is taken from the pioneering organization in this field, the Society for American Baseball Research (SABR). Sabermetrics offer an empirical analysis of baseball, especially the statistics that measure in-game activity.

In keeping with the sports theme, the benefit of data-analytics in decision making was well demonstrated in the movie Moneyball, a 2011 U.S. sports drama film. In the movie, the general manager of the Oakland Athletics, Billy Beane, takes a sophisticated sabermetric approach towards scouting and analyzing players. The team eventually goes on to win 19 consecutive games, tying for the longest winning streak in American League history. This example shows how data analytics-based decision making can lead to outcomes often superior to intuition (or experience-based) decision making, especially when live dynamic data is used.

Big data

Many factors have brought big data into the everyday vocabulary:

Big data provides businesses with access to vast amounts of data that is live, dynamic, and varied. Decision makers are realizing that within this data lie valuable patterns and information. They no longer need to look at historic data to drive forward; they can base their decisions on live data, changing trends/norms, preferences, etc. as they apply it to all areas of the business.

These decision makers acknowledge that, although they may be  uncomfortable with building and interpreting statistical models, they do want to be able to use this dynamic information to enhance business performance. Analytics can reveal insights previously hidden by data that were too costly to process. Intuition perhaps worked better in the past when sampling and static reports (drawn on past events) were the only option on which to base decisions. However, today, consumer preferences, regulatory requirements, and operations within the business environment change quickly. Successfully exploiting data can lead to better decision making, whether it has to do with faster product development, improved analysis of clinical trials, better control of inventory, more efficient regulatory compliance, streamlined supply chain activities, or even the more frequent winning of elections.

Walmart studied the product sales in their Florida stores in 2004 when several hurricanes passed through the state. Intuition predicted that they should see a marked increase in the sales of products such as water, diapers, toilet paper, plywood, generators, flashlight batteries, etc. However, as they analyzed all of the data they had collected on  their customers, their histories, and their purchases, in all the stores, they discovered that the one single product that customers bought at a rate seven times (700%) higher than anything else during the hurricane season was Strawberry Pop-Tarts. Intuition did not work here, and many sales were likely lost as a result.

Data-driven decision making in business is really about capturing or creating new markets through:

Consumers can no longer be divided into the traditional groups of individuals sharing similar characteristics (e.g., age, gender, interests, spending habits, etc.). Today’s “evolved consumers” find value in engagement, and the kind of engagement they seek cannot be identified by intuition alone. Today’s generation (not defined by age alone) buys with one click; they want to learn and not be “sold-to,” rather, they want to be engaged and rewarded. Analytics-based decision making can help practitioners stay on the leading edge of this evolutionary path.

Although statistical models used for decision making may be limited when predicting individual behavior, Amazon and Facebook have shown that they are good predictors of consumer choices and trends. Our subconscious mind condenses a massive amount of diverse information into a summary judgment in making decisions, however, we are all a culmination of our personal experiences and biases, thus making intuition-based decision making unreliable. Even if our intuition turns out to be correct, the inability to provide the rationale behind such decisions makes it almost impossible to replicate.

Creating a balance

Thus, it all boils down to balance. Creating good statistical models is difficult, neglecting biased information is sub-optimal, and decision making cannot be left to algorithms alone. Therefore, analytics needs to play a supporting role. As Steven Hillion, co-founder of Alpine Data Labs, noted, “It’s foolish to assume that number-crunching alone can provide the answers a business needs to get ahead. In data science, intuition and analytics work together in tandem, each informing the other.”

For example, an engineer working for an aerospace engine manufacturer cannot expect diagnostic software alone to determine the causes of problems, write Donald A. Marchand and Joe Peppard in the Harvard Business Review article, “Why IT Fumbles Analytics.” According to the article, “Rather, the engineer must have considerable expertise and knowledge to identify relationships in and ask questions about the data, often through the testing of hypotheses—intuition and analytics working together. And in interpreting the results of any analysis, he or she must draw on experience to weed out misleading or false explanations.” The premise is that although information might be seen as a resource that resides in databases, it is the people and their insight that make it valuable.

The key question, then, is can companies and people successfully make the transition, culturally and technologically, to take decisions supported by data rather than intuition—and if they should, how and when?

 

.