By Simon Garland
If you are a kdb+/q developer, you will find the utilities created by KX Managing Director and Senior Solution Architect Leslie Goldsmith to be a valuable resource. The “Kdb+ Utilities” series of blog posts gives a quick introduction to the utilities, available at Leslie Goldsmith’s GitHub.
The first part of the series looked at ws, an essential tool which contains routines for summarizing and searching the contents of a workspace. The second part of the series looked at another tool for q programmers at every level, dpy, which displays q objects visually. In this third part of the series we look at Leslie’s qprof, which allows a programmer to drill down into q functions or applications to inspect performance and CPU usage in a fine-grained fashion.
There are some simple and effective timing utilities built into q (t and ts) that every programmer knows and uses daily. These are used for timing execution of a line to see how fast it runs, and they also allow you to see how much space it uses. But if you find, for example that a particular 15-line function is the performance bottleneck in your application, you don’t have a simple way of knowing if the problem is lines 3 to 9, or line 12, or one tiny part of line 3. You have no easy way of drilling down into the offending code. This tool is the essential utility for anyone needing to drill down into tricky performance problems. It lets you look at the “where,” “what” and even “why” code runs the way it does, and gives you the opportunity to amend it.
Over the years the problem has been people second guessing where performance problems are. As Knuth says, “Premature optimization is the root of all evil.” People should write their applications and functions first, and then look at what is causing performance problems when they actually occur. It can be quite difficult to find where the problem is hiding. Even in a q application there can be quite a lot of code to be inspected — by definition, unexpected performance problems aren’t where you think they are. The beauty of Leslie’s q profiler is that it easily lets you drill down again and again, interactively until you find the problem.
Yet again, this is something from APL, it is the modern incarnation of a system-level tool that was available in Sharp APL, ⎕FM, influenced by Leslie decades ago. To me it came almost out of the blue and I can still remember that everyone went back to check over their applications looking for bottlenecks, even those running in production. This was a new and amazing capability and it was robust enough to run these monitoring functions safely in production.
Qprof allows the programmer to dig into individual lines and to look at the CPU usage at the current level and broken down in levels below — with timings to nanosecond precision, that we couldn’t have dreamt of back in the 80s. Have a look at qprof and take the time to play with it and get comfortable using it, because when a production system starts slowing down, you are going to need it in a hurry.
Leslie Goldsmith is currently a Managing Director and Senior Solution Architect at KX, a division of First Derivatives (FD), based in Mississauga, Canada. Prior to this position Leslie was co-founder and Managing Director at Affinity Systems, a software consulting and products firm acquired by FD in 2015. In Leslie’s long and storied career as a technologist, he has built many large, complex enterprise software systems and has set a standard of excellence in engineering for generations of developers. He is also the author of one of the world’s first commercial email systems, Mailbox, which debuted in 1973.
Simon Garland is Chief Customer Officer for KX. He is responsible for upholding KX’s high standards for technical excellence and customer responsiveness. Prior to joining KX in 2002, Simon worked at a database search engine company, before that he worked at Credit Suisse in risk management. Simon began his programming career working with APL, and subsequently worked with the original k and kdb, as well as q and kdb+.
