It is more than likely that you’ve already heard most of the business intelligence and analytics tropes by now: data-driven management, revealing hidden insights in your data, storytelling, etc. Everything you’ve heard isn’t necessarily wrong – certainly the world of business analytics offers many exciting possibilities, which the current technologies have only begun to scratch the surface of. But is important to consider this crucial question: is your business intelligence ready for complex data?
What Makes Data Complex, And Why It Matters
While there is no single test that can determine the level of complication that your data represents, here are two telltale signs that you might be facing with complex data:
- You have big data: Despite the endless jargon surrounding this term, the fact of the matter remains that terabyte (or petabyte scale) data is still one of the biggest challenges and opportunities in today’s world of business analytics. The opportunities lie in the possibility of finding new patterns and correlations within huge and seemingly unconnected datasets, originating either from internal or external sources. The challenge is that this data is usually very difficult to analyze, requiring extensive hardware resources and a dedicated data science team to structure it into submission using traditional analytics tools.
- You’re working with disparate data sources (‘wide data’): Even if you don’t have terabytes or petabytes of data, if your business intelligence relies on several distinct data sources, it’s likely that analyzing this data will present some difficulties, stemming from the fact that data is stored in many different formats, file types, and physical locations. And once you bring in unstructured data, such as textual content, video streams, or machine logs, which you want to analyze alongside traditional structured data, the task of making all these different datasets speak in the same language becomes even more laborious. This issue can also arise when working with a large amount of tables, even if they all originate from the same source. This is especially true if these tables do not follow a clear and consistent structure between them, which is often the case when data is “fed” by users rather than automatically.
Perhaps your own data doesn’t currently suit the above mentioned scenarios. Well, here’s the bad news (or the good news, depending on which ‘half of the glass’ you prefer to look at): even if your data isn’t complex right now, you should assume it will be very soon.
This is due to the fact that the ability to gather and store data is constantly evolving, and technological advancements are producing new data sources that previously would not have even been conceivable. And as more industries move to digital and online business models, they also become more ‘measurable’ and more data-driven. If you want to stay competitive, keeping up with the latest developments in data analysis seems like the obvious choice.
Three Things to be Concerned About
If you’re currently working with a BI system – whether internally developed or purchased from one of the many available business intelligence software vendors – and are currently not working with complex data, it might be a good time to ask yourself whether your system will be able to handle the increasingly difficult scenarios it might face in the future. Here are three things to look out for:
Performance
Business intelligence technology has taken some great leaps forward in the past few decades, but it has not developed at the same rate as our ability to collect data. Many existing tools still rely on outdated OLAP or in-memory technology, which tends to significantly slow down when working with larger datasets or requires massive hardware infrastructure to provide reasonable performance.
It’s important to note that this issue occurs, not only when your data is initially very large (as measured in gigabytes or billions of rows), but also when it changes frequently as new sources are added or existing ones are modified. If the system takes days to ‘assimilate’ new sources into the analytical repository, you’ll find it difficult to see results in an actionable timeframe.
To avoid these scenarios, you should make sure your business intelligence software has the technological ‘muscle’ required to take on these unwieldy datasets without burning a hole in your pocket. This will usually mean looking in new directions, rather than using workarounds to have existing tools handle loads they were never meant to bear. You can read more about database technology here.
Ability to Work with Multiple Sources
As previously mentioned, the complexity of your data does not derive solely from its size, but also from the number and type of sources you’re working with. Here are some questions you might want to ask to determine how suitable your BI system is for working with disparate data:
- Can it connect to the data sources you’re working with, either via API or built-in native connectors?
- Does it come with data preparation and modeling capabilities, or will you need additional tools for that?
- Can you explore the data in any direction you can think of and on an ad-hoc basis, or are you limited to certain pre-defined queries or data fields that must be prepared in advance?
If the answer to one or more of these questions is ‘no’, you might find yourself running into some brick walls when attempting to perform meaningful analysis based on multiple data sources.
Extensibility
The final thing you want to ask is whether your existing platform will still be feasible when your needs grow. Right now you might only be delivering reports to a few key stakeholders on a monthly basis, but with the rapid evolution of the data analysis field, you should consider the possibility that you’ll want to do much more in the near future: real-time analysis, predictive modeling, customer-facing analytics… the list is endless. Is your system flexible enough to allow for this? Does it offer well-documented API access that your own developers can build upon? Will it be able to support your requirements in terms of scalability and security?
Closing
The future of data analytics presents bright possibilities, but these come bundled with some challenges. Preparing for the latter will ensure you can reap the most benefits from the former. So, is your current BI platform up to the task? The age of complex data is upon us – it’s time to face it head-on.
