Using the NAG Library for Python with kdb+ and PyQ

25 Jun 2019 | , ,
Share on:

By Christopher Brandt

This week we are re-blogging a technical paper by Christopher Brandt, an engineer at the Numerical Algorithms Group (NAG) called “Using the NAG library for Python with kdb+ and PyQ.” It is available from the NAG blog here. The NAG Library is the world’s largest collection of robust, documented, tested and maintained algorithms, and is sold by the Numerical Algorithms Group. This version of the report has been slightly edited for style.

Background

This paper provides detailed instructions on how to use the NAG Library for Python with kdb+ and PyQ. The NAG Library contains more than 1,800 mathematical and statistical routines, and is accessible by numerous programming languages (including Python, C++, Java, Fortran, etc.). PyQ is an extension to kdb+ featuring zero-copy sharing of data between Python and the q programming language. The enclosed examples will illustrate how to access routines within the NAG Library for Python using data stored in kdb+.

Setting up the workspace

Installation for both the NAG Library for Python and the PyQ extension to kdb+ may be performed using pip.

To install the NAG Library for Python:

$ python -m pip install --extra-index-url
     https://www.nag/com/downloads/py/naginterfaces_nag naginterfaces

The PyQ package is developed by Kx, the developers of kdb+. To install it:

$ python -m pip install pyq

Both the NAG Library for Python and kdb+ are commercial software packages that require active licenses for their respective usage. To obtain a temporary license for the NAG Library for Python, please contact NAG support at support@nag.com. [To obtain a free license for kdb+ for non-commercial use download a copy on-demand here].

Examples

The following three examples demonstrate how to call the NAG Library for Python routines using kdb+ and PyQ. These examples were carefully selected, as they cover techniques found in the majority of usage cases a customer will encounter across all 1,800+ routines within the Library. If your usage case falls outside of these three examples, please contact NAG support for assistance.

Example One: BLAS Routine DAXPY

Our first example demonstrates how to perform the linear algebra operation

y := αx + b.

Below is the NAG Library for Python signature for this routine.

naginterfaces.library.blas.daxpy(alpha,x,y)
   
   Parameters: alpha: float
               x: float, array-like, shape(n)
               y: float, array-like, shape(n)
               
   Returns:    y: float, ndarray, shape(n)

Within our terminal, we begin by initiating a PyQ interactive session.

$ pyq

Next, we import PyQ and the BLAS module of the NAG Library for Python.

>>> from pyq import q
>>> from naginterfaces.library import blas

We then enter a q environment and define our parameters as q objects.

>>> q()
q) alpha:0.5f
q) x:4#2 2 2 2f
q) y:4#4 4 4 4f

Finally, we exit the q environment and invoke the NAG routine.

q) \
>>> z = blas.daxpy(float(q.alpha), q.x, q.y)
>>> z # display solution: array([4., 4., 4., 4.])

Example Two: Nearest Correlation Matrix

Our second example employs a nearest correlation matrix routine which, for a given approximate correlation matrix G, computes the nearest correlation matrix X by minimizing the weighted Frobenius norm

kdb+ and Python

where W is a diagonal matrix of weights.

The NAG Library for Python signature for this routine is below.

naginterfaces.library.correg.corrmat_nearest_bounded(
  g,opt,alpha=None,w=None,errtol=0.0,maxits=0,maxit=200)

Parameters: g: float, array-like, shape(n,n)
            opt: str, length 1
            alpha: None or float, optional
            w: None or float, array-like, shape(n), optional
            errtol: float, optional
            maxits: int, optional
            maxit: int, optional

Returns:    x: float, ndarray, shape(n,n)
            itera: int
            feval: int
            nrmgrd: float

Within our interactive PyQ session, we begin by importing the Correlation and Regression Analysis module of the NAG Library for Python.

>>> from naginterfaces.library import correg

Next, we enter a q environment and define our parameters as q objects.

>>> q()
q) alpha:0.5f
q) x:4#2 2 2 2f
q) g:4 4#2 -1 0 0 -1 2 -1 0 0 -1 2 -1 0 0 -1 2f
q) opt:”B” 
q) alpha:0.02f
q) w:4#100 20 20 20f

We then exit the q environment and invoke the NAG routine.

q) \
>>> x, feval, itera, nrmgrd = correg.corrmat_nearest_bounded(
          q.g, str(q.opt), float(q.alpha), q.w)
>>> x  # display solution

Example Three: Numerical Integration

With our final example, we demonstrate how to incorporate a user-defined callback function with a NAG Library for Python routine. This example approximates the definite integral

kdb+ and Python

The NAG Library for Python signature for this routine is below.

naginterfaces.library.quad.dim1_fin_smooth(f,a,b,epsabs,epsrel,data=None)

Parameters: f: callable, result = f(x,data=None)
               Parameters:
                   x: float
                   data: arbitrary, optional, modifiable in place
            a: float
            b: float
            epsabs: float
            epsrel: float
            data: arbitrary, optional

Returns:    result: float
            abserr: float

We start by importing the Quadrature module of the NAG Library for Python.

>>> from naginterfaces.library import quad

Next, we enter a q environment and define our parameters as q objects.

>>> q()
q)  a:0f
q)  b:2f
q)  epsabs:0f
q)  epsrel:0.0001f

We then exit the q environment and define an integrable Python function. To satisfy this parameter we may use either a Python function or a lambda expression.

q) \
>>> def f(x):
      return x*x
...

With our problem now fully defined, we invoke the NAG routine to compute our solution.

>>> result, error = quad.dim1_fin_smooth(
      f, float(q.a), float(q.b), float(q.epsabs), float(q.epsrel))
>>> result  # 2.6666666666666667
>>> error   # 1.4802973661668755e-14

Additional Usage Cases 

NAG recently published the technical report “Using the NAG Library with kdb+ in a pure q environment” discussing how to call the NAG Library using the new Foreign Function Interface (FFI) from Kx. Additionally, the NAG Blog four, titled “Calling the NAG C Library from kdb+” details how to incorporate the NAG Library with kdb+ within a C++ program. We speculate that among our shared clients, a mixture of these methods will be employed.
If your desired usage case happens to fall outside of those described within our current publications, please contact NAG support at support@nag.com for assistance with your application.

References

(1) Calling the NAG C Library from kdb+, http://blog.nag.com/2013/05/ calling-nag-c-library-from-kdb.html
(2) Get Going with Kdb+, https://code.kx.com/v2/
(3) Kdb+ and Python: embedPy and PyQ, https://kx.com/blog/kdb-pythonembedpy-pyq/
(4) NAG GitHub Organisation, https://github.com/numericalalgorithmsgroup/
(5) NAG Library for Python Manual, https://www.nag.com/numeric/py/ nagdoc_latest/index.html
(6) Using Foreign Functions with kdb+ (FFI), https://code.kx.com/q/ interfaces/ffi/
(7) Using Python with kdb+ (PyQ), https://code.kx.com/q/interfaces/pyq/
(8) Using the NAG Library with kdb+ in a Pure q Environment, https: //www.nag.com/doc/techrep/pdf/tr1_18.pdf

SUGGESTED ARTICLES

A comparison of Python and q for data problem solving

8 May 2019 |

This article takes a simple, real-life problem and analyzes different solutions in Python and q. The problem leads us to discover nice areas of both programming languages, including vector operations, Einstein summation, adverbs and functional form of select statements. Each solution has lessons that deepen our IT knowledge, especially when we consider performance.

Web Scraping – A Kdb+ Use case

24 Jan 2019 | , ,

By Abin Saju Web scraping is a method through which human readable content is extracted from a web page using an automated system. The system can be implemented using a bot/web crawler which traverses through domains or through a web browser which mimics human interaction with a page. There are many use cases for the […]