Does your business need a big data initiative?

“It is a very sad thing that nowadays there is so little useless information.” Oscar Wilde
As far as we know Wilde did not live in a Big Data age but his quote is strangely prescient. It would seem that today almost everything we do generates data that is captured, stored, analyzed and in many cases acted upon. Wearables, sensors, smartphones and other devices are always tracking our presence, actions, and preferences, and there are organizations willing to apply the power of big data and analytics to bear on the data thus generated. If you are at the stage where you are wondering whether to climb on the big data bandwagon, then this is the post for you.
First let’s reassure you that you are obviously not alone. Last year a Dell / Competitive Edge Research of medium sized businesses found that in addition to the 41% of such organizations already involved with big data initiatives a further 55% were planning such an action in the near future. Clearly there are a lot of organizations that sense value in big data initiatives but should you care?
What are the benefits that these organizations believe will accrue to them from a well-designed big data initiative? In a nutshell, what they seek is actionable intelligence on how to react effectively on time to changing market conditions, customer needs, and the wider environment. All organizations generate as well as consume data. Smaller organizations with more manageable data sets draw these insights from data stored in excel sheets and easy to manage Relational Databases. These organizations may also use simple Business Intelligence and reporting and dashboard tools to try to make sense of the data available to them. The time to consider a step up to the big data game is when the insights available from this approach start falling short of the desired impact because the data becomes unmanageable across three parameters – Volume, Velocity, and Variety.
Volume refers to extremely large quantities of data. Consider organizations like a retail giant with dozens, or maybe hundreds, of store locations catering to many hundreds of thousands of customers each interacting with the organization across so many different channels. The sheer amount of relevant data the organization has to deal with is enormous – for, e.g., Walmart is believed to handle 1 million customer transaction per hour generating 2.5 petabytes of data. If your organization deals with a large number of clients or a vast number of business transactions or a small number of individual business transactions, but each is generating several different data points, then the resulting data tsunami itself will force you into a big data initiative.
Velocity addresses the speed with which the data is created. Here the complexity is not only that the data sets are enormous but also that they are built in a very short time span. As an example consider the scientists in NASA were working with the images generated by the Square Kilometer Array, a combination of thousands of telescopes around the world. Their task is to archive high-resolution images at the staggering rate of 700 terabytes every day. Another example is YouTube where users upload 48 hours of video every minute – high volume by any count. The task is made more difficult when you factor in variable rates of production of data – ebbs and flows in the velocity of data production. Chances are if your business faces a situation where the data arrives in such bursts then a big data initiative may be for you.
Whoever said Variety is the spice of life was clearly not talking about data. Chief among the data-related issues organizations have to deal with is the different ways in which data is created. There are structured sources like databases and tables, but there are also unstructured sources like customer interactions over email, the contact center, social media platforms or the wider web. The complexity is that some of the interactions have a history that spans channels and to draw meaningful insights they have to be considered in their entirety. Take the case of an insurance customer who seeks information on the product offering online, then speaks to an agent to address some specific queries before making a purchase through a mobile app utilizing a discount code – taken in isolation each action would give misleading information but made together the story is very different. The point is that if your organization has to deal with data that arrives in many different forms, then you may be well advised to look at a big data initiative.
The big data age is upon organizations and us all around are falling in line but don’t follow blindly. Look at the Volume, Velocity, and Variety of data your organization has to deal with and make your decision. Chris Lynch, former President and CEO of Vertica Systems said, “Data is becoming the new raw material of business.” –that is true, but it’s still up to you to decide what to make of it!