Kx Insights: Sensor Data Storage for Industrial IoT

7 Jul 2017 | , , , , , , ,
Share on:

By Michael Donaghy

 

Businesses adopting Industrial Internet of Things (IIoT) technologies face a herculean task. In an industry where the number of connected devices and equipment is increasing exponentially, the amount of operational data coming online which must be analyzed and stored rises in tandem.

Though many industrial businesses utilizing traditional database management systems find themselves drowning in the sheer volume of sensor data they are producing, those who are forward-looking are turning towards new platforms to help manage their data and, in turn, reduce costs and improve overall performance.

In this paper I take a look at the lifecycle of Industrial IoT sensor data from its initial point of entry into a system, through various transformations and tiers of storage, to finally its archival or disposition.

As a device or sensor receives data, the data needs to be ingested and stored so that it can be analyzed. This process typically happens close to the physical location of the sensor as well as at central facility.

Data first is ingested into an in-memory database where it is processed to perform validations or checks, identify patterns or trends, detect anomalies, and make decisions on this fresh data.  As the volumes of data and the richness of processing increase, the amount of available memory can quickly be exhausted.

To optimize the amount of space available in-memory, data can be filtered, that is, it can have any redundancies or noise removed, either at the point of ingestion or just prior to the data being stored to a permanent storage medium.

As data is moved from memory to disk, it can be placed on storage media based on performance and cost criteria, where it can then be compressed to reduce the amount of data which is stored. When the high-precision raw sensor data you have stored has reached the end of its usefulness, you can aggregate or “downsample” that data so that it is reduced in size whilst still maintaining various statistical measures.

To read the complete paper, click here.

 

SUGGESTED ARTICLES

Kx Product Insights: Inter-Trading Alert

5 Dec 2018 | , , ,

by Aidan O’Neill Kx has a broad list of products and solutions built on the time-series database platform kdb+ that capitalize on its high-performance capabilities when analyzing very large datasets. Kx for Surveillance is a robust platform widely used by financial institutions for monitoring trades for regulatory compliance. The Surveillance platform instantly detects known trading […]

Kx extends relationship with NASA Frontier Development Lab and the SETI Institute

The Exploration of Space Weather at NASA FDL with kdb+

4 Dec 2018 | , , , ,

Our society is dependent on GNSS services for navigation in everyday life, so it is critically important to know when signal disruptions might occur. Physical models have struggled to predict astronomic scintillation events. One method for making predictions is to use machine learning (ML) techniques. This article describes how kdb+ and embedPy were used in the ML application.