Kx Insights: Sensor Data Storage for Industrial IoT

7 Jul 2017 | , , , , , , ,
Share on:

By Michael Donaghy

 

Businesses adopting Industrial Internet of Things (IIoT) technologies face a herculean task. In an industry where the number of connected devices and equipment is increasing exponentially, the amount of operational data coming online which must be analyzed and stored rises in tandem.

Though many industrial businesses utilizing traditional database management systems find themselves drowning in the sheer volume of sensor data they are producing, those who are forward-looking are turning towards new platforms to help manage their data and, in turn, reduce costs and improve overall performance.

In this paper I take a look at the lifecycle of Industrial IoT sensor data from its initial point of entry into a system, through various transformations and tiers of storage, to finally its archival or disposition.

As a device or sensor receives data, the data needs to be ingested and stored so that it can be analyzed. This process typically happens close to the physical location of the sensor as well as at central facility.

Data first is ingested into an in-memory database where it is processed to perform validations or checks, identify patterns or trends, detect anomalies, and make decisions on this fresh data.  As the volumes of data and the richness of processing increase, the amount of available memory can quickly be exhausted.

To optimize the amount of space available in-memory, data can be filtered, that is, it can have any redundancies or noise removed, either at the point of ingestion or just prior to the data being stored to a permanent storage medium.

As data is moved from memory to disk, it can be placed on storage media based on performance and cost criteria, where it can then be compressed to reduce the amount of data which is stored. When the high-precision raw sensor data you have stored has reached the end of its usefulness, you can aggregate or “downsample” that data so that it is reduced in size whilst still maintaining various statistical measures.

To read the complete paper, click here.

 

SUGGESTED ARTICLES

How to Avoid a Goat in Monte Carlo – Elegantly

19 Feb 2019 | ,

by PJ O’Kane and Rian O’Cuinneagain We are all encouraged to be problem solvers. Solutions, however, depend not only on the complexity of the problem but also on the skillset of the solver and the tools at their disposal. Moreover, some can be faced head-on whereas others are best addressed indirectly. If, for example, you […]

Kx Product Insights: Kx for Love!!

14 Feb 2019 | , ,

By Declan Fallontine This month’s Insights post falls on Valentine’s day so today’s article will feature a dashboard visualizing data centered on romance.  As a source we will use a blog post by dating site, Zoosk, which investigating generational differences in dating[1].   Zoosk has over 750K dating profiles and its members have generated nearly 250 million messages […]