Learning kdb+: From the Outside In

28 Dec 2016 | , , ,
Share on:

By Conner McNicholas


My journey with q/kdb+ began after four years spent as a mechanical engineer in the heavy manufacturing industry. Naturally, I felt a bit out of place in my peer group of trainees at Kx, which was largely comprised of fresh-faced computer science graduates.  However, kdb+ experts would invariably assure me that my mind’s blank canvas would actually serve to my advantage, as the others would have to unlearn many paradigms prevalent in the common programming languages (specifically, loops) in order to harness the power of kdb+.  We would all be training ourselves to “think vectorially.”

My initial impression of the q language was a common one: I was intimidated by the terseness of the language and the overloading of operators.  It was frustrating to overcome these mental obstacles in order to solve simple problems.  I found myself thinking, “Wow, this would be so much easier in Excel.” However, when the problems began including datasets in the billions, my frustration gave way to disbelief, even amazement.  In no time at all, I found myself thinking, “Wow, that was FAST.”  In this way, I began feeling truly empowered by the capabilities of this technology.  This epiphany occurred often, and provided ample motivation during those most trying initial months of learning.

Like any other worthwhile skill, competence was gained through practice and mentorship.  The latter of which I found no shortage of, as the q/kdb+ community includes some of the brightest and most supportive individuals I have met in my career.

Fast forward two years, I am now responsible for developing kdb+ applications in the market surveillance space. These days, I welcome the challenge of writing analytics to handle millions of records per second.  Moreover, now that I no longer equate q with terseness, but rather elegance, I find myself reaching in my q/kdb+ toolkit to solve even simple problems.

For example, I stumbled upon an article about Myers-Briggs personality tests which provided the prevalence of each indicator out of the four dichotomous personality dimensions:

Energy Style: 49% Extrovert (E), 51% Introvert (I)

Thinking Style: 73% Sensors (S), 27% Intuitives (N)

Values Style: 40% Thinkers (T), 60% Feelers (F)

Life Style: 54% Judgers (J), 46% Perceivers (P)

While it is easy enough to intuit the order, I was curious to find the exact distribution of all 16 possible personality types given these figures.   At the beginning of my kdb+ journey, I likely would have first reached for Excel.  Instead, I yawn at the prospect of all of that copying, pasting, dragging, clicking, as I enter my q prompt.  20 seconds later, and there we have it:


personality test in kdb+/q


While it may be a shame that I willingly squandered 20 precious seconds of my life on such a trivial exercise, the results are encouraging: ENTP’s such as myself are the least prevalent of the personalities types, at EXACTLY 2.43432%.

Hands down, the most rewarding part of my job today is training clients in q/kdb+, as I witness their attitudes towards q/kdb+ evolve through the same stages that I personally experienced: intimidation, frustration, amazement, and finally epiphany.  After a sub-second demo analyzing hundreds of millions of records, I take great pleasure in assuring the room full of faces in disbelief:

“No, those query results were not previously cached. Yes, it really is that fast.  Now, let’s optimize it.”

© 2018 Kx Systems
Kx® and kdb+ are registered trademarks of Kx Systems, Inc., a subsidiary of First Derivatives plc.


Head of Products, Solutions and Innovation at Kx on Product Design and the Vision for the Future

16 Mar 2018 | , , ,

As the SVP of Products, Solutions and Innovation at Kx Systems, James Corcoran is part of a new chapter in software development at Kx. Since joining Kx parent First Derivatives as a financial engineer in 2009, James has worked around the world building enterprise systems at top global investment banks before moving to the Kx product team in London. James sat down with us recently to discuss his perspective on product design and our technology strategy for the future.

Kdb+ Utilities: Essential utility for identifying performance problems

28 Feb 2018 | ,

If you are a kdb+/q developer, you will find the utilities created by Kx Managing Director and Senior Solution Architect Leslie Goldsmith to be a valuable resource. The “Kdb+ Utilities” series of blog posts gives a quick introduction to the utilities, available at Leslie Goldsmith’s GitHub. In this third part of the series we look at Leslie’s qprof, which allows a programmer to drill down into q functions or applications to inspect performance and CPU usage in a fine-grained fashion.

kdb+ utility to search codebase

Kdb+ Utilities: Q code Workspace Utilities

6 Feb 2018 | , ,

If you are a kdb+/q developer, you will find the workspace utilities created by Kx Managing Director and Senior Solution Architect Leslie Goldsmith to be a valuable resource. This is the first in a series of blog posts that give a quick introduction to several utilities available at Leslie Goldsmith’s GitHub. In this part of the series we look at an essential tool which contains routines for summarizing and searching the contents of a workspace, ws.