A daunting 2.5 billion GB of data (2.5 exabytes, to be precise) was generated per day in 2012. After a 5-year jump, today the figures will be unimaginable.
Not only has the number of smartphones rocketed, new technology like the Internet of Things has begun contributing to the daily data being logged in data warehouses across the world. The fact that data processing and storage costs have gone down by a factor of 1000 has enabled systems to record data without filtering.
It is interesting to note that although recorded data figures are nearing the figurative infinity, only a fraction of data is actually used. Case in point – 1/3rd of bank data is never used. (Source : Boston Consulting Group) This is because there are limitations of present day ‘systems of record’ that make it difficult to retrieve and use specific data.
Sizing data generated from IoT
While most of the present day data (texts, images, videos) is generated by social media users through their smartphones, sensorification of the physical world is helping connect other home and industry appliances to the Internet and thus, generate data logs that are fairly larger than their smartphone counterparts.
Internet of things (IoT) is being hailed as the 3rd wave in IT evolution. Harnessing and big data analytics company data analyzing streams from these devices is going to be the biggest challenge for developers and data scientists in the next 10 years.
Traditional BI systems prove inadequate in handling IoT data
Business Intelligence (BI) systems were devised in 1990s to perform basic data analytics operations of organization and visualization. While they are adequate in handling individual app data in scale-up environments, the amount of data generated by IoT devices can only be handled by big data solutions applied to cloud based systems.
The following factors make it absolutely necessary to switch from traditional data storage to cloud based systems that can handle big data:
- The data stored in legacy systems of record is difficult to retrieve, because of the limitations of BI systems. Big data solutions that will be required for IoT systems should not only store data but also be able to efficiently search, analyze and visualize the data streams.
- The analytics implemented in big data solutions should not only provide insights into the behavior of logged data but also ‘foresights’ through predictive analysis.
- Offline data analysis is a default requirement for any kind of data. However, since IoT devices are used in real world scenarios, occurrence of certain events should be detected at run-time and appropriate action should be taken during real-time data logging.
All these factors require a ‘system of intelligence’ rather than a ‘system of records’.
Systems of Intelligence (SI)
This term has been coined to describe a combination of ‘systems of records’ and ‘system of engagement’, which work in collaboration to provide real-time logging and analysis that can enable end users (consumers) to participate in the functioning of the IoT devices. This is essentially a self-service model, where there can be an automated response to an identified event, or action taken by the consumer as per the application settings.
To understand the concept of self-service model and SI, it is important to know that the IoT devices will be logging sensor data through an app interface (consumer app) provided by the consumer’s smartphone or other app-bearing device. The consumer apps should have one-click installation, ergonomic user interface, and syncing facility to all other smart and IoT devices owned by the consumer.
A good SI has the following features
1. Real-time data logging and API interface
The system should log sensor data in real time, and the data streams should be easily accessible through an API interface so that intermediate big data analytics solutions and the connecting devices can utilize the records.
2. Easy capturing of data streams
The data being logged should not just be stored to the cloud. While the data logging occurs, the SI should analyze the data in real-time to log any fluctuations in normal levels of sensor values.
3. Monitoring and analysis
Apart from dashboard interfaces to monitor and analyze data streams (online and offline), the SI should also be equipped to apply business rules dynamically to large chunks of data.
4. Intelligent output
Like in social media feeds, systems of engagement should log time, context, and place of data origin to enable the SI to provide intelligent output related to current trends, frequent device failure reasons, and other scenarios that might not be anticipated by data scientists while creating the log queries.
5. Information leading to business transformation
These analytics should help enterprises to implement business strategies that benefit the consumer and aid the sales of their consumer facing IoT enabled devices.
To conclude
While Systems of Intelligence in their presently nascent stage (and even in the developed stage in future) may not entirely replace data scientists and business analysts in the workforce, they will complement their efforts very effectively through big data analytics that provide intelligent real-time insights.
