Machine Learning in kdb+: k-NN classification and pattern recognition

31 Aug 2017 | , , , , , ,
Share on:

By Emanuele Melis

As a powerful array-processing technology, kdb+ can be used with great effect in machine learning algorithms. This latest Kx whitepaper on  k-NearestNeighbor classification and pattern recognition in kdb+ uses a non-parametric statistical method commonly used for Pattern Recognition.

k-NN  assumes data points are in a metric space and are represented using n-dimensional vectors, out of which distance metrics can be extracted. This makes it one of the easiest Machine Learning algorithms to implement, but impractical to use in some industry settings due to the computational complexity and cost of: (1) distance metrics; (2) feature extraction; (3) classification.

The paper further examines the implementation strategies in kdb+, and the performance of a k-NN classifier used to predict digits in a dataset of handwritten samples normalized in arrays of 8 (x,y) coordinate pairs. The training set, loaded in kdb+ as “label-to-arrays of features” mappings, was represented as a table keyed on the label and the distance metric calculated applying distance functions on it. A validation set has been used to measure the prediction accuracy of the classifier, leveraging q-sql syntax.

Adopting kdb+ to implement a k-NN classifier introduced the benefits of using a high performance array processing language with an easy to read q-sql syntax, which allows a performant and elegant algorithm implementation without using external libraries.

The code used in this white paper is available on the Kx Github.

 

Emanuele Melis is an expert kdb+/q software engineer currently based in Glasgow, Scotland.

SUGGESTED ARTICLES

How to Avoid a Goat in Monte Carlo – Elegantly

19 Feb 2019 | ,

by PJ O’Kane and Rian O’Cuinneagain We are all encouraged to be problem solvers. Solutions, however, depend not only on the complexity of the problem but also on the skillset of the solver and the tools at their disposal. Moreover, some can be faced head-on whereas others are best addressed indirectly. If, for example, you […]

Kx Product Insights: Kx for Love!!

14 Feb 2019 | , ,

By Declan Fallontine This month’s Insights post falls on Valentine’s day so today’s article will feature a dashboard visualizing data centered on romance.  As a source we will use a blog post by dating site, Zoosk, which investigating generational differences in dating[1].   Zoosk has over 750K dating profiles and its members have generated nearly 250 million messages […]