Machine Learning in kdb+: k-NN classification and pattern recognition

31 Aug 2017 | , , , , , ,
Share on:

By Emanuele Melis

As a powerful array-processing technology, kdb+ can be used with great effect in machine learning algorithms. This latest Kx whitepaper on  k-NearestNeighbor classification and pattern recognition in kdb+ uses a non-parametric statistical method commonly used for Pattern Recognition.

k-NN  assumes data points are in a metric space and are represented using n-dimensional vectors, out of which distance metrics can be extracted. This makes it one of the easiest Machine Learning algorithms to implement, but impractical to use in some industry settings due to the computational complexity and cost of: (1) distance metrics; (2) feature extraction; (3) classification.

The paper further examines the implementation strategies in kdb+, and the performance of a k-NN classifier used to predict digits in a dataset of handwritten samples normalized in arrays of 8 (x,y) coordinate pairs. The training set, loaded in kdb+ as “label-to-arrays of features” mappings, was represented as a table keyed on the label and the distance metric calculated applying distance functions on it. A validation set has been used to measure the prediction accuracy of the classifier, leveraging q-sql syntax.

Adopting kdb+ to implement a k-NN classifier introduced the benefits of using a high performance array processing language with an easy to read q-sql syntax, which allows a performant and elegant algorithm implementation without using external libraries.

The code used in this white paper is available on the Kx Github.


Emanuele Melis is an expert kdb+/q software engineer currently based in Glasgow, Scotland.


Kx Insights: Machine learning and the value of historical data

2 Aug 2018 | , , ,

Data is being generated at a faster rate now than ever before. IDC has predicted that in 2025, there will be 163 zettabytes of data generated each year—a massive increase from the 16.1 zettabytes created in 2016. These high rates of data generation are partially an outcome of the multitude of sensors found on Internet of Things (IoT) devices, the majority of which are capable of recording data many times per second. IHS estimates that the number of IoT devices in use will increase from 15.4 billion devices in 2015 to 75.4 billion in 2025, indicating that these immense rates of data generation will continue to grow even higher in the years to come.

SEMICON 2018 Snapshot: Data and the Era of AI

24 Jul 2018 | ,

By Bill Pierson The future of the semiconductor industry is looking bright judging by the breadth of new developments, initiatives and innovations on display at SEMICON West in San Francisco this July. Industry leading companies presented the latest technical and business insights into today’s opportunities and challenges, particularly in the areas of smart manufacturing and […]