Learning kdb+: From the Outside In

28 Dec 2016 | , ,
Share on:

By Conner McNicholas

 

My journey with q/kdb+ began after four years spent as a mechanical engineer in the heavy manufacturing industry. Naturally, I felt a bit out of place in my peer group of trainees at Kx, which was largely comprised of fresh-faced computer science graduates.  However, kdb+ experts would invariably assure me that my mind’s blank canvas would actually serve to my advantage, as the others would have to unlearn many paradigms prevalent in the common programming languages (specifically, loops) in order to harness the power of kdb+.  We would all be training ourselves to “think vectorially.”

My initial impression of the q language was a common one: I was intimidated by the terseness of the language and the overloading of operators.  It was frustrating to overcome these mental obstacles in order to solve simple problems.  I found myself thinking, “Wow, this would be so much easier in Excel.” However, when the problems began including datasets in the billions, my frustration gave way to disbelief, even amazement.  In no time at all, I found myself thinking, “Wow, that was FAST.”  In this way, I began feeling truly empowered by the capabilities of this technology.  This epiphany occurred often, and provided ample motivation during those most trying initial months of learning.

Like any other worthwhile skill, competence was gained through practice and mentorship.  The latter of which I found no shortage of, as the q/kdb+ community includes some of the brightest and most supportive individuals I have met in my career.

Fast forward two years, I am now responsible for developing kdb+ applications in the market surveillance space. These days, I welcome the challenge of writing analytics to handle millions of records per second.  Moreover, now that I no longer equate q with terseness, but rather elegance, I find myself reaching in my q/kdb+ toolkit to solve even simple problems.

For example, I stumbled upon an article about Myers-Briggs personality tests which provided the prevalence of each indicator out of the four dichotomous personality dimensions:

Energy Style: 49% Extrovert (E), 51% Introvert (I)

Thinking Style: 73% Sensors (S), 27% Intuitives (N)

Values Style: 40% Thinkers (T), 60% Feelers (F)

Life Style: 54% Judgers (J), 46% Perceivers (P)

While it is easy enough to intuit the order, I was curious to find the exact distribution of all 16 possible personality types given these figures.   At the beginning of my kdb+ journey, I likely would have first reached for Excel.  Instead, I yawn at the prospect of all of that copying, pasting, dragging, clicking, as I enter my q prompt.  20 seconds later, and there we have it:

 

personality test in kdb+/q

 

While it may be a shame that I willingly squandered 20 precious seconds of my life on such a trivial exercise, the results are encouraging: ENTP’s such as myself are the least prevalent of the personalities types, at EXACTLY 2.43432%.

Hands down, the most rewarding part of my job today is training clients in q/kdb+, as I witness their attitudes towards q/kdb+ evolve through the same stages that I personally experienced: intimidation, frustration, amazement, and finally epiphany.  After a sub-second demo analyzing hundreds of millions of records, I take great pleasure in assuring the room full of faces in disbelief:

“No, those query results were not previously cached. Yes, it really is that fast.  Now, let’s optimize it.”

©2016 Kx Systems
Kx® and kdb+ are registered trademarks of Kx Systems, Inc., a subsidiary of First Derivatives plc.