Key Takeaways
- qmamba is the package manager for KDB-X, enabling developers to find and install KDB-X modules directly from a q session.
- It automatically resolves dependencies and version conflicts, including third-party libraries sourced from channels such as conda-forge.
- Isolated environments prevent applications from interfering with one another, even when they require different versions of the same libraries.
- Exportable environment specifications support repeatable deployments, allowing teams to recreate the tested package environment across development, QA, and production.
The KX community has provided a huge amount of useful libraries to help other developers.
With KDB-X we’ve provided a module framework to help bring consistency to how these libraries can be written.
The package manager provides a simple way to install modules, while taking care of third-party dependencies and any resulting conflicts.
Why Do I Need a Package Manager
Installing a library to a kdb+/q project brings its own dependency tree. These dependencies often have sub dependencies that have specific versions ranges they require, which themselves may depend on still more libraries. Two libraries in the same application can easily want incompatible versions of a shared dependency. Install one, and you’ve silently broken the other.
Another challenge is when multiple applications are running on a single machine, but each application has a distinct dependecy tree.
qmamba is designed to make these problems disappear. Installing a module is a single command, run from inside a q session.
- Dependency and conflict resolution — when you ask for a package, qmamba resolves its full dependency graph and works out a consistent set of versions to install, pulling in third-party libraries from other channels (like conda-forge) automatically.
- Isolated environments — every environment is its own self-contained entity. You can have a dozen environments on the same machine, each with different, even conflicting, sets of libraries, and they will never interfere with one another.
Qmamba
qmamba is a package manager for KDB-X, built on libmamba.
It enables you to download KDB-X modules from inside a q session with a set of simple commands. The modules can be sourced from the KX channel as prebuilt packages and qmamba will automatically fetch any dependencies (such as third party libraries) from other channels such as conda-forge.
It also supports having multiple independent environments into which packages are installed and allows the user to switch between these environments.
How Do I Use It
qmamba is available as a release on the KX github
An objective we have for qmamba is to improve the user experience for users of our original KX fusion project.
With those interfaces, such as our Kafka interface, we made the user responsible for installing third party libraries such as librdkafka.
With qmamba we reduce the work involved for the user and take ownership of the dependency installation and resolution.
Installing a Module, Start to Finish
Here’s the entire experience of finding and installing a module. All KDB-X modules on anaconda.org are prefixed q-, so a search for a kafka integration on the kx channel looks like this:
q)qmamba:use `kx.qmamba
q)qmamba.search (`SPECS`CHANNEL`PRETTY)!(enlist "q-*protobuf";enlist "kx";1)
Getting repodata from channels...
kx/linux-64 Using cache
kx/noarch Using cache
q-kx-protobuf 2.0.0 hb0f4dca_0
────────────────────────────────────────
Name q-kx-protobuf
Version 2.0.0
Build hb0f4dca_0
Size 2822 kB
License
Subdir linux-64
File Name q-kx-protobuf-2.0.0-hb0f4dca_0.conda
URL https://conda.anaconda.org/kx/linux-64/q-kx-protobuf-2.0.0-hb0f4dca_0.conda
MD5 087c83e6a36eef3c8b6532c21c4f2133
SHA256 d4200c7015c1fafd8fd1ed9ea706ae1e261a2a910f5b53f45949412342e7037a
Dependencies:
- libgcc >=15
- libstdcxx >=15
- libprotobuf 3.*
- libprotobuf >=3.21.12,<3.22.0a0Create an environment for it, and install:
q)qmamba.create "myenv"
Empty environment created at prefix: /home/username/.kx/root/envs/myenv
q)qmamba.activate "myenv"
Activated environment at prefix: /home/username/.kx/root/envs/myenv
q)qmamba.install (`SPECS`CHANNEL)!(enlist "q-kx-protobuf";("kx";"conda-forge"))
kx/linux-64 Using cache
kx/noarch Using cache
conda-forge/linux-64 Using cache
conda-forge/noarch Using cache
Transaction
Prefix: /home/dwalsh/.kx/root/envs/myproto
Updating specs:
- q-kx-protobuf
Package Version Build Channel Size
────────────────────────────────────────────────────────────────
Install:
────────────────────────────────────────────────────────────────
+ _openmp_mutex 4.5 20_gnu conda-forge Cached
+ libgcc 16.2.0 ha9f2e26_4 conda-forge Cached
+ libgcc-ng 16.2.0 h69a702a_4 conda-forge Cached
+ libgomp 16.2.0 he0feb66_4 conda-forge Cached
+ libprotobuf 3.21.12 hfc55251_2 conda-forge Cached
+ libstdcxx 16.2.0 h934c35e_4 conda-forge Cached
+ libstdcxx-ng 16.2.0 hdf11a46_4 conda-forge Cached
+ libzlib 1.3.2 h25fd6f3_3 conda-forge Cached
+ q-kx-protobuf 2.0.0 hb0f4dca_0 kx Cached
Summary:
Install: 9 packages
Total download: 0 B
────────────────────────────────────────────────────────────────
Confirm changes: [Y/n]
Y
Transaction starting
Linking libzlib-1.3.2-h25fd6f3_3
Linking libgomp-16.2.0-he0feb66_4
Linking _openmp_mutex-4.5-20_gnu
Linking libgcc-16.2.0-ha9f2e26_4
Linking libgcc-ng-16.2.0-h69a702a_4
Linking libstdcxx-16.2.0-h934c35e_4
Linking libstdcxx-ng-16.2.0-hdf11a46_4
Linking libprotobuf-3.21.12-hfc55251_2
Linking q-kx-protobuf-2.0.0-hb0f4dca_0
Transaction finishedqmamba works out that q-kx-protobuf needs libprotobuf, fetches it from conda-forge, and links everything into the myenv environment without any manual dependency chasing, no version guesswork. It can now be used:
q)proto:use `kx.protobuf
q)proto.versionStr[]
"libprotobuf v3.21.12"
q)result:proto.serializeArrayFromList[`ScalarExample; (1i;2.2f;"abc")]
q)result
"\010\001\021\232\231\231\231\231\231\001@\032\003abc"
q)proto.parseArrayToList[`ScalarExample; result]
1i
2.2
"abc"How Do I Deploy This Another Environment?
Isolated environments solve library clashes on a single box, but they also unlock something more useful and this is reproducing an environment somewhere else. Once you’ve built out your application during development, you can export the full dependency list to a file and import it into a distinct environment:
q)qmamba.activate "myenv"
q)qmamba.env.export (enlist `TO_FILE)!(enlist "app_spec.yaml")which produces something like:
name: myenv
channels:
- conda-forge
- kx
dependencies:
- _openmp_mutex=4.5=20_gnu
- libgcc=16.2.0=ha9f2e26_4
- libgcc-ng=16.2.0=h69a702a_4
- libgomp=16.2.0=he0feb66_4
- libprotobuf=3.21.12=hfc55251_2
- libstdcxx=16.2.0=h934c35e_4
- libstdcxx-ng=16.2.0=hdf11a46_4
- libzlib=1.3.2=h25fd6f3_3
- q-kx-protobuf=2.0.0=hb0f4dca_0You can commit this, deploy it elsewhere to rebuild an identical environment
q)qmamba.create "qa_env"
q)qmamba.activate "qa_env"
q)qmamba.install ([FILE:enlist "app_spec.yaml"])Every package, every dependency, pinned to the exact build that was tested on your machine. The YAML file captures the environment in full, portable across machines and environments by design.
What Modules Are Available to Use
KX is hosting a number of modules on its conda channel which are available to view on our anaconda.org page
Next Steps
qmamba is available to download, with full documentation from the KX github page.
Full details on how to install qmamba and use it to install KX modules can be found there.
