The past few years have witnessed the rise of Big Data to almost epic proportions. Along with Big Data, we also saw the rise of Data Science and saw the Data Scientist’s job become one of the most coveted profiles in 2015. Businesses continue to invest in Big Data and Big Data products, modernize data platforms and help employees derive more value from data. That being said it is also true that most Big Data projects today cost too much and take too long to get completed and are not quite delivering ROI.
Dr. David Johnston, Lead Data Scientist, ThoughtWorks blames this inability to deliver to the lack of critical insights that stem from “insufficient attention given to data science skills and overemphasis on infrastructure”. He quite rightly says, that while the companies have the right set of tools, they lack the ideas and the skill or talent to use those tools optimally. He highlights that more than the tools and the technology it is the lack of talent and their inexperience that keeps data science teams from exploiting their full potential by integrating data science into complete software applications. Here’s a look at what should be done to resolve this problem:
Focus on agile
Agile is important. Building iteratively and only what is required at the time save costs and enables flexibility. Big Data products too need to enable agility and have to move away from being “inflexible systems suited only for scaling up systems whose idea development is nearly completed”.
Big data is important but so is small data
Since the focus is on a scalable environment, you need to learn to extract small amounts of valuable information from larger data sets by compression, sampling, employing variable selection etc. to understand the kind of scaling that may be required in the future. By working on small data initially, data scientists can assess what kind of technology they will require and only then make a technology investment.
Get the right talent – if you don’t have it hire it
Many companies still expect less experienced data analysts or software developers to do the job of a data scientist. Since data has become too large and too complex, companies need expert data scientists who can create and use advanced data algorithms to add value to a Big Data platform. Naturally I believe that in the absence of a qualified data science team, it makes sense to get a high-level data science consultant on board to develop high-level strategies around data for a positive business impact and to help your team understand how to become more efficient and effective.
The true and complete potential of Big Data can only be realized when companies put in as much effort into hiring data scientists ( as permanent employees or consultants) and treat them as an equally important investment as they do in the Big Data infrastructure. After all, it is the data scientist who will ensure that your data initiatives get off on the right foot.
Originally appeared on Linkedin.
