'Big Data vs Intuition' paradox explained

Industry   |   
Published June 26, 2013   |   

Every business leader will have a dad’s story, I believe, — a memory or nostalgia how he/she was inspired by the attitudes/thoughts/decisions (sometimes seemingly tough, painful and even stupid) of his/her dad in paving a unique way to success and beyond. Here is one such story:

“We didn’t pick our dads. We either were lucky or unlucky with the dad we ended up with. In my case, I was lucky. I didn’t always think so. Permit me to explain. I grew up around the insurance business as my dad owned a small insurance agency in Michigan. I had fun doing the insurance company account payable statements for him, running to the post office for the mails, cleaning the office and other important details, or least they were important to me. Watching what happened and what my dad did made me want to become an insurance agent, just like him. So, off to the college I went. Finally, graduation day came and my dad and mom came to my graduation exercises and afterwards, he told me he had sold the agency. I just couldn’t believe it. I asked why? We are moving to Arizona and if you want an agency, go start one just like I did. My heart sank, and I wondered why would he do that to me? Yes, poor me. I wanted the going business and all I got was his advice to start my own. Can you imagine how I felt? Well, I did start my own in California and my company is now starting its 46th year. Do you think my dad made me successful? I do. By working hard and learning from the bottom up, I’m a stronger person, have had more fun, met more people and enjoyed having more employees and relationships than I would ever have had in a small town. My dad made me successful.” [Source]

Honestly, I’m proud of this guy, who wrote this testimony, because despite his dad’s bloody/annoying/stupid/insane/self-centred/iron-headed/backstabbing decision (most of us feel the same about our dads at some point, though we later change) to sell the agency, he decided to stay and abide by his dad’s advice. If I were him, I would’ve plucked his eyes out and run away. (I almost did the same, when I was told by my late dad immediately after my schooling that I should “stop wasting time in the school” (I don’t know how he found out I was wasting time in the school. Probably, he had met my teachers somewhere, especially the curly-haired Hindi teacher. She totally hated me) and start a small business – a fruit shop or a provisional store — in my village with my dad as an investor. I first thought he was kidding, but when I realised he wasn’t, I became upset, thinking how narrow-minded my dad was to refuse education for his son for some extra bucks from business. Looking back, I now feel that I should’ve obeyed him, because all my classmates, dropped out of the school and started their own businesses, are rich now, having earned their own names in their respective fields and making more money (maybe double, triple or more) than I do now in a month. Sometimes (not all the time), I feel my dad was right; because after all, what did I become through education?)

Anyways, the point is most of our dads were the men of limited means: they never had devices that we use today to access/store/transfer information in decision making, no MBAs, no queries to run etc….; still, they made smart decisions through observation, common sense, experience and most importantly intuition. Talking about intuition, the best example coming to my mind is Steve Jobs, well known as the “champion of intuition”, who once said: “I began to realize that an intuitive understanding and consciousness was more significant than abstract thinking and intellectual logical analysis. Intuition is a very powerful thing, more powerful than intellect, in my opinion. That’s had a big impact on my work.” In fact, the word “Intuition” has a deeper meaning (esp in Indian context/Philosophy) than the meaning it actually conveys in the West. I’m pretty sure Jobs knew it well: both its philosophical and spiritual implications. That’s clear in his following quote.

“The people in the Indian countryside don’t use their intellect like we do, they use their intuition instead, and their intuition is far more developed than in the rest of the world. Intuition is a very powerful thing, more powerful than intellect, in my opinion. That’s had a big impact on my work. Western rational thought is not an innate human characteristic; it is learned and is the great achievement of Western civilization. In the villages of India, they never learned it. They learned something else, which is in some ways just as valuable but in other ways is not. That’s the power of intuition and experiential wisdom. Coming back after seven weeks in Indian villages, I saw the craziness of the Western world as well as its capacity for rational thought. If you just sit and observe, you will see how restless your mind is. If you try to calm it, it only makes it worse, but over time it does calm, and when it does, there’s room to hear more subtle things–that’s when your intuition starts to blossom and you start to see things more clearly and be in the present more. Your mind just slows down, and you see a tremendous expanse in the moment. You see so much more than you could see before. It’s a discipline; you have to practice it.”

Of course, we are in the brink of information revolution, where the accelerating volume and velocity of data being created and consumed by devices through collaboration and social media are unprecedented, and big data companies have immense opportunity to process these data to extract insights from information. But does it mean that yesteryear businessmen were less successful than today’s because they did not have a computer? Certainly, with the data and automated data mining tools, analysts can easily find interesting patterns on their own, but that is just one ingredient in the recipe. Intuition and judgments are necessary to set priorities for what to look for, what matters to the customer, what the competitive landscape looks like, how behaviours are changing over time and ultimately what to do with the data. Tools can help the analyst’s brain, but shall not replace it.

But, we have a problem here. We know that intuition is important, but what if intuition goes wrong? Will flipping a coin always work? Experts say that under certain circumstances, intuition can lead us to incorrect decisions and judgments. “For example, every person who plays basketball and nearly every person who watches it believes that players go through hot streaks, when they are in the groove, and cold streaks, when they are just not feeling it. But researchers have found that a player who has made six consecutive foul shots has the same chance of making his seventh as if he had missed the previous six foul shots. When a player has hit six shots in a row, we imagine that he has tapped into some elevated performance groove. In fact, it’s just random statistical noise, like having a coin flip come up tails repeatedly. Each individual shot’s success rate will still devolve back to the player’s career shooting.” Besides, there is a data processing problem too. There is no way human brain can even dream to process massive data of large companies without machines and algorithms that tackle personalised problems. Too much information is way too complex for an objective analysis in very short time. So, it’s a practical problem.

So, what’s our take on Big Data vs Intuition? Are they mutually complimenting or…? Yes, they are. Big data and intuition should not be seen in silos. They are to be in a perfect harmony — like mind-body relationship. Analysis of data helps to bring down the risk involved in decision making, as an intuition can go horribly wrong and it happens most of the time. In Steve Jobs’ case, it is extremely tempting to call/project him as “a champion of intuition”, but I believe that his ideas must have been based on/influenced by some sort of personal analysis (not necessarily a big data analysis), based on technology trends, recommendations of his associates and understanding the market and its demands. So, can we say there is something called “Pure Intuition”? Nope. In a nutshell, there is no “Big Data vs Intuition” scenario, rather it’s Big Data plus Intuition. Big Data actually supports decisions based on intuition. If you want your organization to take better decisions, your decisions will need to be based on both explicit information (data) and implicit information (intuition).

Now, what’s next? Let’s look into some possible questions in “Big Data-Intuition” situation! We’ve seen they are complimentary, but how do we combine big data and intuition in a situation, especially when they give contradictory output? Let me put the question in this way: How do you respond to the instances where the data view is a 100% contradiction of your gut feeling and passion? You would love to chase your passion/ intuition because that’s what had been deeply-rooted into your DNA over the years, but you are not backed up by data. What will you do? Will you take risks and tend to write your own destiny? Second, how do you manage conflicting intuitions in a company/situation, when there is no data to prove which one is correct/wrong? Leaving these questions open for all to reflect on… Thanks for reading.