AxLibraries and Q Build Utilities

by Jonathan Turcotte

Introduction

KX Analyst and KX Developer provide a suite of useful features for q development, including code documentation, linting, testing, and more. With the release of KX Analyst and KX Developer 1.1.0, many of these features are now available in a separate library package, called AxLibraries for use outside the UIs and from other editors. This article introduces the AxLibraries package and details how it can be used by q developers to lint, test, and document their q code outside of the KX Analyst and KX Developer IDEs. It also details an example build process that is then automated and integrated with Jenkins.

AxLibraries

The AxLibraries package contains a number of useful libraries which can be loaded into existing q processes to provide useful development functionality outside of the KX Analyst or KX Developer UIs, such as from command line or from within other editors. Some of these libraries, such as the test framework qCumber, can be run as a command line script for more convenience. These scripts make it easy to integrate important code validation steps into automated build environments such as Jenkins. Note that all of the functionality included in AxLibraries is also available in the KX Analyst and KX Developer UIs.

How to Get AxLibraries

AxLibraries is packaged as a zip file, ax-libraries.zip, inside of the KX Analyst and KX Developer install packages. To get AxLibraries, download the latest KX Analyst or KX Developer zip file and decompress it, and then unzip the ax-libraries.zip file that it contains.

Included Libraries

At the time of writing, the AxLibraries package includes:

AxLibraries also includes all public KX Developer APIs, including q wrappers for PCRE (.pcre), PCRE2 (.pcre2), Date Parsing (.qdate), and more. Each of the included libraries and APIs can be loaded into any q process using \l.  For more details about the included libraries and their APIs, refer to the Libraries section of the KX Developer user guide.

Script Utilities

AxRepo, qCumber, qDoc and qLint can also be run directly from the command line as a script using q:

$ q $AXLIBRARIES_HOME/ws/qcumber.q_ -help

Each script has a -help flag that outputs more information on how to use that specific script. Optional flags are enclosed in [].

Setup

After unzipping the ax-libraries.zip archive, read the README.md file that it contains. This file details important information for installing and using the provided libraries. A simple installation outline is provided below, but it is recommended that you follow the specific instructions in the README.md.

First we have to copy the correct library files for the current operating system to ws/lib. In this example, run on macOS, we copy the files in ws/lib/osx to ws/lib:

// From inside the unzipped ax-libraries directory
$ cp -a ws/lib/osx/* ws/lib/

Next setup the AXLIBRARIES_HOME environment variable:

$ export AXLIBRARIES_HOME=$PWD

Once this environment variable is set, any included library can be loaded from any directory on the system. The libraries do not need to be copied to $QHOME.

A Simple Q Build Pipeline using AxLibraries

With AxLibraries, it’s easier than ever to export, lint, test, and document your q code. The following example will use a repository of q modules and q functions created in KX Developer, and will demonstrate an example build using AxLibraries.

The example here would work just as well using a project of q files created in any editor.

Clone the Repository

$ git clone <origin>/project-src

Convert to q Scripts

While using modules in KX Developer or KX Analyst provides many advantages, we must convert them to plain q scripts before proceeding. If your source code is already a collection of regular q scripts, this step is not necessary.

To convert from the custom kxscm repository format, we use the axrepo.q_ script:

$ q $AXLIBRARIES_HOME/ws/axrepo.q_ \
                -src ./project-src \
                -out ./workspace/out/project-src

This command reads the project-src repository and outputs modules as regular q scripts, placing them in the ./workspace/out/project-src directory.

Lint Source

Now we will lint the q source using qLint to statically identify potential bugs, errors, and stylistic issues:

$ q $AXLIBRARIES_HOME/ws/qlint.q_ \
                -src ./workspace/out/project-src \
                -out ./workspace/reports/lint.dat

Many common sources of q bugs can be identified this way, and qLint will organize issues into three categories: error, warning, and info. We can view qLint’s output directly if we do not provide the -out flag:

$ q $AXLIBRARIES_HOME/ws/qlint.q_ -src ./workspace/out/project-src

Run Tests

qCumber test files with the .quke file extension can be run by using the qcumber.q_ script:

$ q $AXLIBRARIES_HOME/ws/qcumber.q_ \
    -src ./workspace/out/project-src/load.q \
    -test ./workspace/out/project-src/ \
    -out ./workspace/reports/test.dat

The qcumber.q_ script takes several important flags:

• The -src flag points to a script to load before running any tests. In this case, the q script above simply loads all the scripts exported by axrepo.q_.
• The -test flag can point to a directory of tests or a singular .quke test file to run.
• The -out flag specifies the output file and format.

Omitting the -out flag and adding the -color flag prints the result to the console with colors to indicate success and failure:

$ q $AXLIBRARIES_HOME/ws/qcumber.q_ \
    -src ./workspace/out/project-src/load.q \
    -test ./workspace/out/project-src/ \
    -color

The qcumber framework, as with the other utilities, can be run as a library within a q process as well, making it possible to run tests in live development processes if needed.

Generate Project Documentation

Finally, documentation can be generated using qDoc and output in either Markdown or an mkdocs compatible format:

$ q $AXLIBRARIES_HOME/ws/qdoc.q_ \
                -src ./workspace/out/project-src/ \
                -out ./workspace/qdoc -render

Because we passed the -render flag, the output in ./workspace/qdoc/doc will be an mkdocs ready directory. With mkdocs installed, we can build HTML with mkdocs build, or serve by doing mkdocs serve and get the following output:

Result

We have now exported, linted, and tested our code, and generated project documention. The workspace directory structure at this point is as follows:

• the out directory contains the converted kxscm repositories as q scripts
• the qdoc directory contains both Markdown and mkdocs ready documentation
• the reports directory contains the result of the qcumber tests and qlint linting

Automatic Builds with Jenkins

We can fully automate this process using a build system like Jenkins. The exact steps for creating a Jenkins job are outside the scope of this blog, however creating a job that follows the same steps as those outlined above is trivial.

If desired, the JUnit plugin available for Jenkins can be leveraged to visually display historical test results within Jenkins. To interface with the plugin, we can modify the qcumber.q_ command to output a JUnit XML file:

$ q $AXLIBRARIES_HOME/ws/qcumber.q_ \
    -src ./workspace/out/project-src/load.q \
    -test ./workspace/out/project-src/ \
    -out ./workspace/reports/test.xml             # was .../test.dat

This will output a JUnit formatted .xml file instead of a q serialized .dat file. We can then have the Jenkins job publish our JUnit test results by pointing it at the test.xml file, making informative metrics about test result trends available in the Jenkins UI: