Learning kdb+: From the Outside In

28 Dec 2016 | , ,
Share on:

By Conner McNicholas

 

My journey with q/kdb+ began after four years spent as a mechanical engineer in the heavy manufacturing industry. Naturally, I felt a bit out of place in my peer group of trainees at Kx, which was largely comprised of fresh-faced computer science graduates.  However, kdb+ experts would invariably assure me that my mind’s blank canvas would actually serve to my advantage, as the others would have to unlearn many paradigms prevalent in the common programming languages (specifically, loops) in order to harness the power of kdb+.  We would all be training ourselves to “think vectorially.”

My initial impression of the q language was a common one: I was intimidated by the terseness of the language and the overloading of operators.  It was frustrating to overcome these mental obstacles in order to solve simple problems.  I found myself thinking, “Wow, this would be so much easier in Excel.” However, when the problems began including datasets in the billions, my frustration gave way to disbelief, even amazement.  In no time at all, I found myself thinking, “Wow, that was FAST.”  In this way, I began feeling truly empowered by the capabilities of this technology.  This epiphany occurred often, and provided ample motivation during those most trying initial months of learning.

Like any other worthwhile skill, competence was gained through practice and mentorship.  The latter of which I found no shortage of, as the q/kdb+ community includes some of the brightest and most supportive individuals I have met in my career.

Fast forward two years, I am now responsible for developing kdb+ applications in the market surveillance space. These days, I welcome the challenge of writing analytics to handle millions of records per second.  Moreover, now that I no longer equate q with terseness, but rather elegance, I find myself reaching in my q/kdb+ toolkit to solve even simple problems.

For example, I stumbled upon an article about Myers-Briggs personality tests which provided the prevalence of each indicator out of the four dichotomous personality dimensions:

Energy Style: 49% Extrovert (E), 51% Introvert (I)

Thinking Style: 73% Sensors (S), 27% Intuitives (N)

Values Style: 40% Thinkers (T), 60% Feelers (F)

Life Style: 54% Judgers (J), 46% Perceivers (P)

While it is easy enough to intuit the order, I was curious to find the exact distribution of all 16 possible personality types given these figures.   At the beginning of my kdb+ journey, I likely would have first reached for Excel.  Instead, I yawn at the prospect of all of that copying, pasting, dragging, clicking, as I enter my q prompt.  20 seconds later, and there we have it:

 

personality test in kdb+/q

 

While it may be a shame that I willingly squandered 20 precious seconds of my life on such a trivial exercise, the results are encouraging: ENTP’s such as myself are the least prevalent of the personalities types, at EXACTLY 2.43432%.

Hands down, the most rewarding part of my job today is training clients in q/kdb+, as I witness their attitudes towards q/kdb+ evolve through the same stages that I personally experienced: intimidation, frustration, amazement, and finally epiphany.  After a sub-second demo analyzing hundreds of millions of records, I take great pleasure in assuring the room full of faces in disbelief:

“No, those query results were not previously cached. Yes, it really is that fast.  Now, let’s optimize it.”

SUGGESTED ARTICLES

kdb+ and Python

Kdb+ and Python: embedPy and PyQ

15 Nov 2017 | , , ,

In September, Kx announced a range of initiatives to put machine learning (ML) capabilities at the heart of future technology development. The first library to be released as part of this initiative is embedPy, which exposes powerful Python functionality to q developers. EmbedPy is the mirror image of PyQ, a set of software components that simplify the running of a Python interpreter alongside a kdb+ server, the rights to which Kx has acquired. Developed by Alexander Belopolsky of Enlightenment Research, PyQ covers all Python libraries, with a primary focus on numerical libraries such as NumPy and SciPy.

Kdb+ Mastermind Challenge

Kdb+ Mastermind Challenge

4 Oct 2017 | , , , , ,

The Kx Community NYC Meetup recently competed in a Mastermind coding competition moderated by expert kdb+ programmer Nick Psaris, author of the book “Q Tips: Fast, Scalable and Maintainable Kdb+.” The top four solutions were notable for the completely different approaches they took. Mastermind is a popular code-breaking game introduced in 1975 that pits a “codemaker” against a “codebreaker.” The “code,” represented by four pegs chosen from a universe of 6 colors, is picked by the “codemaker.” The “codebreaker” continually makes 4-peg guesses and is told by the “codemaker” how many are the correct color in the correct position, and how many are the correct color in the wrong position.

Kdb+ for Factom Blockchain application

Kdb+ and Factom: Securing data on the blockchain with qfactom

13 Sep 2017 | , , , ,

Blockchain applications are proliferating in the financial services industry and Kx technology is increasingly being used in these systems to provide high-speed data analytics, like by the Bitcoin Mercantile Exchange (BitMEX), a cryptocurrency derivatives exchange and by financial technology firm Cobalt DL. qfactom, is a kdb+ wrapper library designed to interact with the Factom blockchain via its client and wallet applications. The library provides convenient functions to create, publish data to, and read data from, Factom chains for the purpose of creating and validating secure audit trails.