Edge Analytics and IoT – How Kx is Revolutionizing Manufacturing from the Edge

27 Mar 2017 | , , , , , , , ,
Share on:

By Michael Donaghy


Edge analytics is a method of data analysis and collection that allows an automated analytical computation to be performed on a tag or sensor rather than having all of the ingested data sent back to a centralized data warehouse before being acted upon. Manufacturing is an industry which requires real-time action on analytics at their source. This is a challenge which Kx excels at.

An average offshore oil rig has around 30,000 sensors. These sensors measure gas emissions, track pressure, monitor drills and yet only one percent of what they ingest and send to a data warehouse is of value to the decision-making process. Setting up sensors to route all of the data to a centralized system in a raw state is time-consuming and costly. It requires the data be cleansed and analyzed, eating heavily into available bandwidth. Therefore a solution is needed to carry these tasks out at the point of collection to take advantage of devices or gateways on the “edge.”

The “edge” aggregates the tags and sensors scattered across a network or embedded within devices, such as a sensor in a connected car. By embedding computational capability in the form of Complex Event Processing (CEP) at the furthest points of the network, Kx can take advantage of the ‘chatty’ or ‘noisy’ data which is generated by making sense of it at the point of ingestion and sending on securely only what it deems to be useful to the data historian for situational awareness.

However, this is not the only advantage of performing edge analytics with Kx. By running your data through an analytics algorithm at the point of ingestion, not only will you create a framework that only allows the data that passes a certain criteria to flow through to the cloud or data warehouse, but it will allow you to perform decisions on malfunctioning or faulty equipment in real-time, thus preventing unnecessary downtime.

As manufacturing is an incredibly time-sensitive industry where every moment of equipment downtime counts, the sub-millisecond CEP latencies of Kx have proven popular with manufacturing clients who benefit greatly from rapid automated decisions.

One needs only to look at the autonomous vehicle industry to see why rapid, real-time automated decisions are necessary. For example, an autonomous “smart” car produces around 25 GB/hour in data.  However, only a small portion of this data is relevant when carrying out hazard detection. The CEP performed by Kx at the edge allows abnormalities to be detected in real-time with an alert generated immediately. The same applies in the aviation industry where the average aircraft produces data volumes of 51 TB/hour, or in locomotives, where a “smart” train contains 250 tags that ingest data from 150,000 points per minute.

These planes, trains and automobiles cannot afford to rely on decisions coming from the centralized data warehouse when it comes to fault and hazard detection and therefore require technology capable of performing analytics and generating alerts in real time. Manufacturing is no different.

By building edge-based analytical systems into machinery, manufacturers can utilize Kx’s ability to work on the edge of the network by having it send alerts and load new scripts into appliances when conditions change.

For example, think of an automated drilling station. The drill may have sensors that measure vibration, rotation speed, temperature, velocity, etc.  Through CEP analysis, Kx may notice from the data it has ingested from the tag points that the drill is overheating. An alert is generated when the anomaly is discovered, a new command script loaded to halt the drill and the resulting data fed into a machine learning model.

The small footprint of Kx technology (500KB) and its ability to perform edge analytics with sub-millisecond latencies while compressing data to within a fragment of its original size, all make Kx a prime candidate for manufacturers to store and analyze huge volumes of IoT data in a scalable, efficient and secure way.

In its most simplistic form, edge analytics allows manufacturers to have data ingested from a sensor, filter out the noise surrounding it, identify a potential fault and act on that fault. It is imperative that manufacturers take advantage of edge analytics to reduce downtime and improve throughput as many are currently doing. The platform which will allow these analytics and computations to be carried out in the fastest time and the most secure way is Kx.

For more information on Kx for Manufacturing click here.

Michael Donaghy is a Global Business Development Executive at First Derivatives based in Newry, Northern Ireland.

© 2017 Kx Systems
Kx® and kdb+ are registered trademarks of Kx Systems, Inc., a subsidiary of First Derivatives plc.


Kx collaborating with Fintech startup chartiq

Collaboration: The Dominant Trend in Finance

13 Dec 2017 | , , , ,

In December we are re-blogging some of our favorite content from Kx partners and affiliated companies, starting with this article on the ChartIQ blog. ChartIQ is an agile FinTech company that sells an advanced HTML5 charting library used in technical data analysis, trading configurations and for charting in the capital markets industry. Kx offers a ChartIQ integration as an addition to our Dashboards. In Collaboration: The Dominant Trend in Finance, ChartIQ’s Hanni Chehak writes about the rise of FinTech companies, and the role collaboration plays as FinTech companies are increasingly disrupting the traditional banking sector.

Water system workers with kdb+ historical database

Kdb+ Use Case: Machine Learning Water System Maintenance Application

6 Dec 2017 | , , , ,

Kdb+ is being used much more widely in machine learning applications today. Its ability to quickly ingest and process data, particularly large, fragmented datasets, is one way that developers are adding kdb+ to their technology stack of artificial intelligence and machine learning tools.
For Australian kdb+ developer Sherief Khorshid, who also develops machine learning systems, incorporating kdb+ into a predictive maintenance application gave him the edge in a hackathon win that landed him a cash prize and a contract with the Water Corporation of Western Australia.

kdb+ FFI

Kdb+ FFI: Access external libraries more easily from q

22 Nov 2017 | , , ,

Following on from the hugely popular Python library and interface embedPy and PyQ, Kx has released an FFI as part of the Fusion for kdb+ interfaces. As with embedPy and PyQ, this FFI is open-sourced under the Apache 2 license.
The kdb+ FFI is a foreign function interface library for loading and calling dynamic libraries from q code. It has been adapted and expanded upon from a library originally written by Alex Belopolsky of Enlightenment Research. With the kdb+ FFI you can now call your favorite C/C++ libraries directly from q without the overhead of having to compile shared objects and load into q using the 2: command.