Comparing and Contrasting Kx and the Hadoop Ecosystem

1 Nov 2016 | , , , , ,
Share on:

“Through 2018, 70% of Hadoop deployments will fail to meet cost savings and revenue generation objectives due to skills and integration challenges.”

Gartner, 100 Data and Analytics Predictions Through 2020

 

 

By Glenn Wright

The exponential increase in available digitized data, or Big Data, is transforming business and research. The appreciation of the potential of Big Data to change how companies operate has tracked the rise of the Apache Hadoop ecosystem, which includes open-source computing frameworks for working with large datasets.

Over the past two decades companies in the financial services industry working with extremely large datasets have turned to the Kx platform, a high-performance time-series database called kdb+ with a built-in programming language called q for high performance analytics. Kx predates the Apache Hadoop ecosystem by decades, and Kx is proven to be more performant, especially as data volumes increase.

In my latest whitepaper I discuss of some of the differences between the two approaches for tackling large-scale, complex business analytics. I highlight the advantages of both systems and inspect some of the key architectural differences between the two. While both approaches have merits, it is only when you are required to deploy the tools for complex analytics that the true merits of the Kx approach can be fully realized.

SUGGESTED ARTICLES

A comparison of Python and q for data problem solving

8 May 2019 | ,

This article takes a simple, real-life problem and analyzes different solutions in Python and q. The problem leads us to discover nice areas of both programming languages, including vector operations, Einstein summation, adverbs and functional form of select statements. Each solution has lessons that deepen our IT knowledge, especially when we consider performance.