IoT Analytics: How Kx can Supercharge your Legacy Systems

23 May 2017 | , , , , , ,
Share on:

 

By Hugh Hyndman

 

If your company is not already drowning in data now, it will be soon. Gartner Inc. forecasts that 8.4 billion connected things will be in use worldwide in 2017, up 31 percent from 2016, and will reach 20.4 billion by 2020. In 2017, 6 million new things will be connected every day. Each one of them will transmit a stream of data, adding up to sensor data volumes that will dwarf today’s volumes. How your business uses this streaming data to compete in the market may determine its long-term success within the world of Industrial IoT.

These volumes impose significant challenges and pressures on solutions based on traditional relational database management systems (RDBMS) to ingest, process, and store this data. These systems were not built to handle such enormous volumes of data and this is resulting in significant and unsustainable increases in license fees, infrastructure, and operating costs for industrial businesses.

To tackle these challenges, organizations are turning to a high-performance database technology from Kx Systems which has been used by leading organizations to achieve low-latency and high-volume event processing. Kx’s combination of in-memory and on-disk database, its efficient columnar design, and highly-integrated query language adds up to the fastest analytics engine on the market.

In this paper, we discuss approaches and techniques for preserving your investment in your software and RDBMS technology while addressing the performance, scale, and cost challenges when working with high volumes of data. We describe three architecture patterns (upstream, midstream, and downstream) for implementing Kx alongside existing systems and processes, discuss their advantages and disadvantages, and provide case studies of these patterns in action.

Kx offers a suite of enterprise-level products centered around kdb+; the world’s fastest time series database. Kdb+ is optimized for ingesting, analyzing, and storing massive amounts of structured data. The combination of the columnar design of kdb+ and its in-memory capabilities means it offers greater speed and efficiency than typical relational databases. Its native support for time-series operations vastly improves both the speed and performance of queries, aggregation, and analysis of structured data.

One feature that really sets Kx apart is its ability to combine streaming, in-memory and historical data in one simple and unified platform; there is no requirement to acquire disparate components to build a hybrid solution.

To read the rest of the paper click here.

Hugh Hyndman is the Director of IoT Solutions at Kx Systems, based out of Toronto. Hugh has been involved with high-performance big data computing for most of his career. His current focus is to help companies supercharge their software systems and products by injecting Kx technologies into their stack. If you are interested in OEM or partnership opportunities, please contact Hugh through sales@kx.com.

 

© 2017 Kx Systems
Kx® and kdb+ are registered trademarks of Kx Systems, Inc., a subsidiary of First Derivatives plc.

SUGGESTED ARTICLES

Kx collaborating with Fintech startup chartiq

Collaboration: The Dominant Trend in Finance

13 Dec 2017 | , , , ,

In December we are re-blogging some of our favorite content from Kx partners and affiliated companies, starting with this article on the ChartIQ blog. ChartIQ is an agile FinTech company that sells an advanced HTML5 charting library used in technical data analysis, trading configurations and for charting in the capital markets industry. Kx offers a ChartIQ integration as an addition to our Dashboards. In “Collaboration: The Dominant Trend in Finance,” ChartIQ’s Hanni Chehak writes about the rise of FinTech companies, and the role collaboration plays as FinTech companies are increasingly disrupting the traditional banking sector.

Water system workers with kdb+ historical database

Kdb+ Use Case: Machine Learning Water System Maintenance Application

6 Dec 2017 | , , , ,

Kdb+ is being used much more widely in machine learning applications today. Its ability to quickly ingest and process data, particularly large, fragmented datasets, is one way that developers are adding kdb+ to their technology stack of artificial intelligence and machine learning tools.
For Australian kdb+ developer Sherief Khorshid, who also develops machine learning systems, incorporating kdb+ into a predictive maintenance application gave him the edge in a hackathon win that landed him a cash prize and a contract with the Water Corporation of Western Australia.

kdb+ FFI

Kdb+ FFI: Access external libraries more easily from q

22 Nov 2017 | , , ,

Following on from the hugely popular Python library and interface embedPy and PyQ, Kx has released an FFI as part of the Fusion for kdb+ interfaces. As with embedPy and PyQ, this FFI is open-sourced under the Apache 2 license.
The kdb+ FFI is a foreign function interface library for loading and calling dynamic libraries from q code. It has been adapted and expanded upon from a library originally written by Alex Belopolsky of Enlightenment Research. With the kdb+ FFI you can now call your favorite C/C++ libraries directly from q without the overhead of having to compile shared objects and load into q using the 2: command.