Kx Insights: Sensor Data Storage for Industrial IoT

7 Jul 2017 | , , , , , , ,
Share on:

By Michael Donaghy

 

Businesses adopting Industrial Internet of Things (IIoT) technologies face a herculean task. In an industry where the number of connected devices and equipment is increasing exponentially, the amount of operational data coming online which must be analyzed and stored rises in tandem.

Though many industrial businesses utilizing traditional database management systems find themselves drowning in the sheer volume of sensor data they are producing, those who are forward-looking are turning towards new platforms to help manage their data and, in turn, reduce costs and improve overall performance.

In this paper I take a look at the lifecycle of Industrial IoT sensor data from its initial point of entry into a system, through various transformations and tiers of storage, to finally its archival or disposition.

As a device or sensor receives data, the data needs to be ingested and stored so that it can be analyzed. This process typically happens close to the physical location of the sensor as well as at central facility.

Data first is ingested into an in-memory database where it is processed to perform validations or checks, identify patterns or trends, detect anomalies, and make decisions on this fresh data.  As the volumes of data and the richness of processing increase, the amount of available memory can quickly be exhausted.

To optimize the amount of space available in-memory, data can be filtered, that is, it can have any redundancies or noise removed, either at the point of ingestion or just prior to the data being stored to a permanent storage medium.

As data is moved from memory to disk, it can be placed on storage media based on performance and cost criteria, where it can then be compressed to reduce the amount of data which is stored. When the high-precision raw sensor data you have stored has reached the end of its usefulness, you can aggregate or “downsample” that data so that it is reduced in size whilst still maintaining various statistical measures.

To read the complete paper, click here.

 

SUGGESTED ARTICLES

Signal processing in kdb+

Signal processing with kdb+

6 Sep 2018 | , ,

In the latest in our ongoing series of kdb+ technical white papers published on the Kx Developer’s site, Kx engineer Callum Biggs examines how kdb+/q can be used instead of popular software-based signal processing solutions. Signal processing is used for analyzing observable events, such as IoT sensor data, sounds, images and other types of pulses […]

Kx Insights: Machine learning and the value of historical data

2 Aug 2018 | , , ,

Data is being generated at a faster rate now than ever before. IDC has predicted that in 2025, there will be 163 zettabytes of data generated each year—a massive increase from the 16.1 zettabytes created in 2016. These high rates of data generation are partially an outcome of the multitude of sensors found on Internet of Things (IoT) devices, the majority of which are capable of recording data many times per second. IHS estimates that the number of IoT devices in use will increase from 15.4 billion devices in 2015 to 75.4 billion in 2025, indicating that these immense rates of data generation will continue to grow even higher in the years to come.

Top three ICT data themes at NXTAsia 2018

25 Jul 2018 | , , , ,

By Thomas Kearney  Collecting and analyzing big data in order to differentiate services in a competitive marketplace has been a common theme for many years­­­­ among Western developed nations. In recent years, emerging markets have seen double-digit growth rates from similar provision of cheaper, faster telco services and from digitizing and optimizing their delivery chain. Now, […]