Discover kdb+ 4.1’s New Features

With the announcement of kdb+ 4.1, we’ve made significant updates in performance, security, and usability, empowering developers to turbo charge workloads, fortify transmissions and improve storage efficiency. In this blog, we’ll explore these new features and demonstrate how they compare to previous versions.

Let’s begin.

Peach/Parallel Processing Enhancements

“Peach” is an important keyword in kdb+, derived from the combination of “parallel” and “each”. It enables the parallel execution of a function on multiple arguments.
In kdb+ 4.1 the ability to nest “peach” statements now exists. Additionally, it no longer prevents the use of multithreaded primitives within the “peach” operation.

It also introduces a work-stealing algorithm technique in which idle processors can intelligently acquire tasks from busy ones. This marks a departure from the previous method of pre-allocating chunks to each thread, and with it, better utilization of CPU cores, which in our own test have resulted in significant reductions in processing time.


q)\s
8i

/Before: kdb+ 4.0
q)\t (inv peach)peach 2 4 1000 1000#8000000?1.
4406

/After: kdb+ 4.1
q)\t (inv peach)peach 2 4 1000 1000#8000000?1.
2035

 

Network Improvements

For large enterprise deployments and cloud enabled workloads, unlimited network connections offer reliability and robust performance at an unprecedented scale. With kdb+ 4.1, user-defined shared libraries now extend well beyond 1024 file descriptors for event callbacks, and HTTP Persistent Connections elevate efficiency and responsiveness of data interactions, replacing the one-and-done approach previously used to reduce latency and optimize resource utilization.

Multithreaded Data Loading

The CSV load, fixed-width load (0:), and binary load (1:) functionalities are now multithreaded, signifying a significant leap in performance and efficiency, this is particularly beneficial for handling large datasets which in our own tests have resulted in a 50% reduction in ingestion times.

Socket Performance

Socket operations have also been enhanced, resulting in a five-fold increase in throughput when tested against previous versions. With kdb+ 4.1 users can expect a significant boost in overall performance and a tangible difference, when dealing with high volume connections.


q)h:hopen `:tcps://localhost:9999
q)h ".z.w"
1004i

/Before: kdb+ 4.0
q)\ts:10000 h "2+2"
1508 512

/After: kdb+ 4.1
q)\ts:10000 h "2+2"
285 512

 

Enhanced TLS Support and Updated OpenSSL

Enhanced TLS and Updated OpenSSL Support guarantee compliance and safeguard sensitive real-time data exchanges. With kdb 4.1 OpenSSL 1.0, 1.1.x, and 3.x, coupled with dynamic searching for OpenSSL libraries and TCP and UDS encryption, offers a robust solution for industries where data integrity and confidentiality are non-negotiable.

Furthermore, TLS messaging can now be utilized on threads other than the main thread. This allows for secure Inter-Process Communication (IPC) and HTTP operations in multithreaded input queue mode. Additionally, HTTP client requests and one-shot sync messages within secondary threads are facilitated through “peach”.

More Algorithms for At-Rest Compression

With the incorporation of advanced compression algorithms, kdb+ 4.1 ensures maximized storage efficiency without compromising data access speed. This provides a strategic advantage when handling extensive datasets.

New q Language Features

Kdb+ 4.1 introduces several improvements to the q language that will help to streamline your code.

  1. Dictionary Literal Syntax
    With dictionary literals, you can concisely define dictionaries. For instance, for single element dictionaries you no longer need to enlist. Compare

    
    
    q)enlist[`aaa]!enlist 123 / 4.0
    aaa| 123
    
    

    With

    
    
    q)([aaa:123]) / 4.1
    aaa| 123
    
    

    This syntax follows rules consistent with list and table literal syntax.

    
    
    q)([0;1;2]) / implicit key names assigned when none defined
    x | 0
    x1| 1
    x2| 2
    
    q)d:([a:101;b:]);d 102 / missing values create projections
    a| 101
    b| 102
    
    q)d each`AA`BB`CC
    a b
    ------
    101 AA
    101 BB
    101 CC
    
    

     

  2. Pattern Matching
    Assignment has been extended so the left-hand side of the colon (:) can now be a pattern.

    
    
    /Traditional Method
    q)a:1 / atom
    
    /Using new method to assign b as 2 and c as 3
    q)(b;c):(2;3) / list
    
    /Pattern matching on dictionary keys
    q)([four:d]):`one`two`three`four`five!1 2 3 4 5 / dictionary
    
    /Assigning e to be the third column x2
    q)([]x2:e):([]1 2;3 4;5 6) / table
    
    q)a,b,c,d,e
    1 2 3 4 5 6
    
    

    Before assigning any variables, q will ensure left and right values match.

    
    
    q)(1b;;x):(1b;`anything;1 2 3) / empty patterns match anything
    q)x
    
    1 2 3
    
    

    Failure to match will throw an error without assigning.

    
    q)(1b;y):(0b;3 2 1)
    'match
    
    q)y
    'y
    
    

     

  3. Type checking
    While we’re checking patterns, we could also check types.

    
    
    /Checks if surname is a symbol
    q)(surname:`s):`simpson
    
    /Checks if age is a short
    q)(name:`s;age:`h):(`homer;38h)
    
    /Type error triggered as float <> short
    q)(name:`s;age:`h):(`marge;36.5) / d'oh
    'type
    
    q)name,surname / woohoo!
    `homer`simpson
    
    

    And check function parameters too.

    
    
    q)fluxCapacitor:{[(src:`j;dst:`j);speed:`f]$[speed<88;src;dst]}
    
    q)fluxCapacitor[1955 1985]87.9
    1955
    
    q)fluxCapacitor[1955 1985]88
    'type
    
    q)fluxCapacitor[1955 1985]88.1 / Great Scott!
    1985
    
    

     

  4. Filter functions

    We can extend this basic type checking to define our own ‘filter functions’ to run before assignment.

    
    
    / return the value (once we're happy)
    
    q)tempCheck:{$[x<0;' "too cold";x>40;'"too hot" ;x]}
    q)c2f:{[x:tempCheck]32+1.8*x}
    q)c2f -4.5
    'too cold
    
    q)c2f 42.8
    'too hot
    
    q)c2f 20 / just right
    68f
    
    

    We can use filter functions to change the values that we’re assigning.

    
    
    q)(a;b:10+;c:100+):1 2 3
    q)a,b,c
    1 12 103
    
    

    Amend values at depth without assignment.

    
    
    q)addv:{([v:(;:x+;:(x*10)+)]):y}
    q)addv[10;([k:`hello;v:1 2 3])]
    k| `hello
    v| 1 12 103
    
    

    Or even change the types.

    
    
    q)chkVals:{$[any null x:"J"$ "," vs x;'`badData;x]}
    q)sumVals:{[x:chkVals]sum x}
    q)sumVals "1,1,2,3,5,8"
    20
    
    q)sumVals "8,4,2,1,0.5"
    'badData
    
    

 

Happy Coding

We hope you are as excited as we are about the possibilities these enhancements bring to your development toolkit. From parallel processing and network scalability to streamlined data loading, secure transmissions, and optimized storage, our commitment is to empower you with tools that make your coding life more efficient and enjoyable.

We invite you to dive in, explore, and unlock the full potential of kdb+ 4.1. Download our free Personal Edition today.

Happy coding!

Related Resources

11 Insights to Help Quants Break Through Data and Analytics Barriers

Book a Demo