Learning kdb+: From the Outside In

28 Dec 2016 | , , ,
Share on:

By Conner McNicholas

 

My journey with q/kdb+ began after four years spent as a mechanical engineer in the heavy manufacturing industry. Naturally, I felt a bit out of place in my peer group of trainees at Kx, which was largely comprised of fresh-faced computer science graduates.  However, kdb+ experts would invariably assure me that my mind’s blank canvas would actually serve to my advantage, as the others would have to unlearn many paradigms prevalent in the common programming languages (specifically, loops) in order to harness the power of kdb+.  We would all be training ourselves to “think vectorially.”

My initial impression of the q language was a common one: I was intimidated by the terseness of the language and the overloading of operators.  It was frustrating to overcome these mental obstacles in order to solve simple problems.  I found myself thinking, “Wow, this would be so much easier in Excel.” However, when the problems began including datasets in the billions, my frustration gave way to disbelief, even amazement.  In no time at all, I found myself thinking, “Wow, that was FAST.”  In this way, I began feeling truly empowered by the capabilities of this technology.  This epiphany occurred often, and provided ample motivation during those most trying initial months of learning.

Like any other worthwhile skill, competence was gained through practice and mentorship.  The latter of which I found no shortage of, as the q/kdb+ community includes some of the brightest and most supportive individuals I have met in my career.

Fast forward two years, I am now responsible for developing kdb+ applications in the market surveillance space. These days, I welcome the challenge of writing analytics to handle millions of records per second.  Moreover, now that I no longer equate q with terseness, but rather elegance, I find myself reaching in my q/kdb+ toolkit to solve even simple problems.

For example, I stumbled upon an article about Myers-Briggs personality tests which provided the prevalence of each indicator out of the four dichotomous personality dimensions:

Energy Style: 49% Extrovert (E), 51% Introvert (I)

Thinking Style: 73% Sensors (S), 27% Intuitives (N)

Values Style: 40% Thinkers (T), 60% Feelers (F)

Life Style: 54% Judgers (J), 46% Perceivers (P)

While it is easy enough to intuit the order, I was curious to find the exact distribution of all 16 possible personality types given these figures.   At the beginning of my kdb+ journey, I likely would have first reached for Excel.  Instead, I yawn at the prospect of all of that copying, pasting, dragging, clicking, as I enter my q prompt.  20 seconds later, and there we have it:

 

personality test in kdb+/q

 

While it may be a shame that I willingly squandered 20 precious seconds of my life on such a trivial exercise, the results are encouraging: ENTP’s such as myself are the least prevalent of the personalities types, at EXACTLY 2.43432%.

Hands down, the most rewarding part of my job today is training clients in q/kdb+, as I witness their attitudes towards q/kdb+ evolve through the same stages that I personally experienced: intimidation, frustration, amazement, and finally epiphany.  After a sub-second demo analyzing hundreds of millions of records, I take great pleasure in assuring the room full of faces in disbelief:

“No, those query results were not previously cached. Yes, it really is that fast.  Now, let’s optimize it.”

© 2017 Kx Systems
Kx® and kdb+ are registered trademarks of Kx Systems, Inc., a subsidiary of First Derivatives plc.

SUGGESTED ARTICLES

Kx collaborating with Fintech startup chartiq

Collaboration: The Dominant Trend in Finance

13 Dec 2017 | , , , ,

In December we are re-blogging some of our favorite content from Kx partners and affiliated companies, starting with this article on the ChartIQ blog. ChartIQ is an agile FinTech company that sells an advanced HTML5 charting library used in technical data analysis, trading configurations and for charting in the capital markets industry. Kx offers a ChartIQ integration as an addition to our Dashboards. In Collaboration: The Dominant Trend in Finance, ChartIQ’s Hanni Chehak writes about the rise of FinTech companies, and the role collaboration plays as FinTech companies are increasingly disrupting the traditional banking sector.

Water system workers with kdb+ historical database

Kdb+ Use Case: Machine Learning Water System Maintenance Application

6 Dec 2017 | , , , ,

Kdb+ is being used much more widely in machine learning applications today. Its ability to quickly ingest and process data, particularly large, fragmented datasets, is one way that developers are adding kdb+ to their technology stack of artificial intelligence and machine learning tools.
For Australian kdb+ developer Sherief Khorshid, who also develops machine learning systems, incorporating kdb+ into a predictive maintenance application gave him the edge in a hackathon win that landed him a cash prize and a contract with the Water Corporation of Western Australia.

kdb+ FFI

Kdb+ FFI: Access external libraries more easily from q

22 Nov 2017 | , , ,

Following on from the hugely popular Python library and interface embedPy and PyQ, Kx has released an FFI as part of the Fusion for kdb+ interfaces. As with embedPy and PyQ, this FFI is open-sourced under the Apache 2 license.
The kdb+ FFI is a foreign function interface library for loading and calling dynamic libraries from q code. It has been adapted and expanded upon from a library originally written by Alex Belopolsky of Enlightenment Research. With the kdb+ FFI you can now call your favorite C/C++ libraries directly from q without the overhead of having to compile shared objects and load into q using the 2: command.