Learning kdb+: From the Outside In

28 Dec 2016 | , , ,
Share on:

By Conner McNicholas

 

My journey with q/kdb+ began after four years spent as a mechanical engineer in the heavy manufacturing industry. Naturally, I felt a bit out of place in my peer group of trainees at Kx, which was largely comprised of fresh-faced computer science graduates.  However, kdb+ experts would invariably assure me that my mind’s blank canvas would actually serve to my advantage, as the others would have to unlearn many paradigms prevalent in the common programming languages (specifically, loops) in order to harness the power of kdb+.  We would all be training ourselves to “think vectorially.”

My initial impression of the q language was a common one: I was intimidated by the terseness of the language and the overloading of operators.  It was frustrating to overcome these mental obstacles in order to solve simple problems.  I found myself thinking, “Wow, this would be so much easier in Excel.” However, when the problems began including datasets in the billions, my frustration gave way to disbelief, even amazement.  In no time at all, I found myself thinking, “Wow, that was FAST.”  In this way, I began feeling truly empowered by the capabilities of this technology.  This epiphany occurred often, and provided ample motivation during those most trying initial months of learning.

Like any other worthwhile skill, competence was gained through practice and mentorship.  The latter of which I found no shortage of, as the q/kdb+ community includes some of the brightest and most supportive individuals I have met in my career.

Fast forward two years, I am now responsible for developing kdb+ applications in the market surveillance space. These days, I welcome the challenge of writing analytics to handle millions of records per second.  Moreover, now that I no longer equate q with terseness, but rather elegance, I find myself reaching in my q/kdb+ toolkit to solve even simple problems.

For example, I stumbled upon an article about Myers-Briggs personality tests which provided the prevalence of each indicator out of the four dichotomous personality dimensions:

Energy Style: 49% Extrovert (E), 51% Introvert (I)

Thinking Style: 73% Sensors (S), 27% Intuitives (N)

Values Style: 40% Thinkers (T), 60% Feelers (F)

Life Style: 54% Judgers (J), 46% Perceivers (P)

While it is easy enough to intuit the order, I was curious to find the exact distribution of all 16 possible personality types given these figures.   At the beginning of my kdb+ journey, I likely would have first reached for Excel.  Instead, I yawn at the prospect of all of that copying, pasting, dragging, clicking, as I enter my q prompt.  20 seconds later, and there we have it:

 

personality test in kdb+/q

 

While it may be a shame that I willingly squandered 20 precious seconds of my life on such a trivial exercise, the results are encouraging: ENTP’s such as myself are the least prevalent of the personalities types, at EXACTLY 2.43432%.

Hands down, the most rewarding part of my job today is training clients in q/kdb+, as I witness their attitudes towards q/kdb+ evolve through the same stages that I personally experienced: intimidation, frustration, amazement, and finally epiphany.  After a sub-second demo analyzing hundreds of millions of records, I take great pleasure in assuring the room full of faces in disbelief:

“No, those query results were not previously cached. Yes, it really is that fast.  Now, let’s optimize it.”

© 2017 Kx Systems
Kx® and kdb+ are registered trademarks of Kx Systems, Inc., a subsidiary of First Derivatives plc.

SUGGESTED ARTICLES

kdb+ for sensor analytics

Kx Insights: Benefits of Utility Predictive Maintenance Analytics

17 Oct 2017 | , , ,

Utilities are going through significant modernization, with the adoption of smart grid technologies such as advanced metering, advanced distribution management, outage management, customer engagement and analytics. This modernization is creating a wealth of diverse data about assets, operations, and customers. At the same time the job of utilities is becoming more challenging with pressure to reduce costs, competition from new forms of technologies and energy providers as well as the need to integrate renewable energy resources. These new challenges, competition and pressures are leading to innovation and transformation in the utilities industry.

Satellite Earth Observation Data with kdb+

Kx for Earth Observation and Astronomy Big Data challenges

11 Oct 2017 | , , , ,

The aerospace industry is increasingly becoming aligned with companies that can take advantage of high-value geospatial and global Earth observation data. In the current era of Big Data and IoT analytics, businesses in industries ranging from energy to civil engineering to facilities management that can commercially exploit non-traditional sources of data, such as data from satellites, can gain a critical edge. Kx technology has already been adopted in a number of such initiatives.

Kdb+ Mastermind Challenge

Kdb+ Mastermind Challenge

4 Oct 2017 | , , , , ,

The Kx Community NYC Meetup recently competed in a Mastermind coding competition moderated by expert kdb+ programmer Nick Psaris, author of the book “Q Tips: Fast, Scalable and Maintainable Kdb+.” The top four solutions were notable for the completely different approaches they took. Mastermind is a popular code-breaking game introduced in 1975 that pits a “codemaker” against a “codebreaker.” The “code,” represented by four pegs chosen from a universe of 6 colors, is picked by the “codemaker.” The “codebreaker” continually makes 4-peg guesses and is told by the “codemaker” how many are the correct color in the correct position, and how many are the correct color in the wrong position.