Kdb+ Utilities: Essential utility for identifying performance problems

28 Feb 2018 |
Share on:

 

By Simon Garland

If you are a kdb+/q developer, you will find the utilities created by Kx Managing Director and Senior Solution Architect Leslie Goldsmith to be a valuable resource. The “Kdb+ Utilities” series of blog posts gives a quick introduction to the utilities, available at Leslie Goldsmith’s GitHub.

The first part of the series looked at ws, an essential tool which contains routines for summarizing and searching the contents of a workspace.  The second part of the series looked at another tool for q programmers at every level, dpy, which displays q objects visually. In this third part of the series we look at Leslie’s qprof, which allows a programmer to drill down into q functions or applications to inspect performance and CPU usage in a fine-grained fashion.

There are some simple and effective timing utilities built into q (\t and \ts) that every programmer knows and uses daily. These are used for timing execution of a line to see how fast it runs, and they also allow you to see how much space it uses. But if you find, for example that a particular 15-line function is the performance bottleneck in your application, you don’t have a simple way of knowing if the problem is lines 3 to 9, or line 12, or one tiny part of line 3. You have no easy way of drilling down into the offending code. This tool is the essential utility for anyone needing to drill down into tricky performance problems. It lets you look at the “where,” “what” and even “why” code runs the way it does, and gives you the opportunity to amend it.

Over the years the problem has been people second guessing where performance problems are. As Knuth says, “Premature optimization is the root of all evil.” People should write their  applications and functions first, and then look at what is causing performance problems when they actually occur. It can be quite difficult to find where the problem is hiding. Even in a q application there can be quite a lot of code to be inspected — by definition, unexpected performance problems aren’t where you think they are. The beauty of Leslie’s q profiler is that it easily lets you drill down again and again, interactively until you find the problem.

Yet again, this is something from APL, it is the modern incarnation of a system-level tool that was available in Sharp APL, ⎕FM, influenced by Leslie decades ago. To me it came almost out of the blue and I can still remember that everyone went back to check over their applications looking for bottlenecks, even those running in production. This was a new and amazing capability and it was robust enough to run these monitoring functions safely in production.

Qprof allows the programmer to dig into individual lines and to look at the CPU usage at the current level and broken down in levels below — with timings to nanosecond precision, that we couldn’t have dreamt of back in the 80s. Have a look at qprof and take the time to play with it and get comfortable using it, because when a production system starts slowing down, you are going to need it in a hurry.

 

Leslie Goldsmith is currently a Managing Director and Senior Solution Architect at Kx, a division of First Derivatives (FD), based in Mississauga, Canada. Prior to this position Leslie was co-founder and Managing Director at Affinity Systems, a software consulting and products firm acquired by FD in 2015. In Leslie’s long and storied career as a technologist, he has built many large, complex enterprise software systems and has set a standard of excellence in engineering for generations of developers. He is also the author of one of the world’s first commercial email systems, Mailbox, which debuted in 1973.

Simon Garland is Chief Customer Officer for Kx. He is responsible for upholding Kx’s high standards for technical excellence and customer responsiveness. Prior to joining Kx in 2002, Simon worked at a database search engine company, before that he worked at Credit Suisse in risk management. Simon began his programming career working with APL, and subsequently worked with the original k and kdb, as well as q and kdb+.

SUGGESTED ARTICLES

kdb+ utility to search codebase

Kdb+ Utilities: Q code Workspace Utilities

6 Feb 2018 |

If you are a kdb+/q developer, you will find the workspace utilities created by Kx Managing Director and Senior Solution Architect Leslie Goldsmith to be a valuable resource. This is the first in a series of blog posts that give a quick introduction to several utilities available at Leslie Goldsmith’s GitHub. In this part of the series we look at an essential tool which contains routines for summarizing and searching the contents of a workspace, ws.

Q for Mortals guide to kdb+/q programming

Q for Mortals Version 3 Now Free Online

3 Jan 2017 | , ,

Kx is pleased to announce that Jeffry Borror’s book Q for Mortals Version 3, An Introduction to Q Programming, is now available for free online at code.kx.com here.
Q for Mortals serves as both an introduction and reference guide to programming in q/kdb+, and replaces an earlier version. In the latest edition, Jeffry includes expanded real-life applications based on his experience teaching and using q at a large financial institution where kdb+/q has been widely deployed for many years.