Key trends that may ease demand for big data skills in 2019

Ever since big data went mainstream over the last several years, companies all over the globe have been working feverishly to build out their data collection, preparation, and analytics operations. The technology involved has been evolving at breakneck speed, leaving interested businesses with no shortage of technical solutions to their big data needs. On the human capital side of the equation, though, results have been mixed.

In the beginning, experts feared that there would be a yawning divide between the demand for workers with the needed big data skills and the actual ability of the educational establishment to prepare them. As it turns out, the experts may have underestimated how big the demand would be, as the big data labor situation is already direr than almost anyone expected. The good news is that there are some trends in both the big data and educational realms that offer some hope that things are about to get better. Here’s what they are.

A shift from building to operating

At the beginning of the big data revolution, businesses were forced to come to grips with the fact that their organizations had, for years in some cases, been collecting and storing all manner of data in a random and siloed manner. For the most part, there were no standards and no unified systems to deal with all of it – much less make use of any of it. Fast forward a few years and the situation has changed.

Today, more companies than ever have implemented standards for data collection and storage, and the first waves of data scientists laid the groundwork of the systems that those companies could use to exploit said data. That has substantially lowered the day-to-day operational skills needed to run a useful data analytics operation. For example, much of the demand for employees in the big data space is now centered on data analysts, who require only a bachelors-level education. That’s a far cry from the masters-level data scientists that were previously needed to do the pioneering work that used to be the norm.

A move towards data democratization

Another side effect of the standardization that’s been happening in the big data sphere is that there’s also a pronounced shift underway towards data democratization. In short, businesses are beginning to see the value in providing access to data across operational environments, up and down their organizational ladders. The big industry players are pushing towards more end-user focused self-service analytics and visualization tools, and they’re starting to gain traction across a variety of industries. Their ease of use means that less specialized knowledge is needed by end-users, further lowering the demand for high-end big data skills in the broader workforce.

A rise in micro-credentials

On the educational side of the equation, it finally appears that the industry has found an effective way to help employers up-skill existing employees and prepare a new generation of data-savvy candidates. They’re doing it by emphasizing and expanding the availability of micro-credentials, which offer specific and targeted education designed to impart needed skills in a short span. With them, companies can react far more quickly to changing data skill needs in their organizations by either offering subsidized access to micro-credential coursework to existing workers, or by targeting new employees that don’t have traditional data science degrees but that have earned the required micro-credentials already. In both instances, businesses can now afford to be more agile and responsive in their big data initiatives, without fear of running short on required skilled employees as their specific needs change.

Pressure will continue to ease

Taken together, these trends point towards both an increase in the supply of credentialed big data workers, coupled with the gradual reduction in the level of skills those employees will need to do their jobs. If you project these trends out into the future, it becomes clear that data science teams will continue to shrink, and will contain a few high-level data science specialists, while the bulk of the skills demand shifts towards mid and low-level workers. That’s good news for an industry that once faced the possibility of losing steam amidst a labor shortage – and even better news for businesses that can now adopt big data with lower overhead and decreased staffing needs.