With the announcement of kdb+ 4.1, we’ve made significant updates in performance, security, and usability, empowering developers to turbo charge workloads, fortify transmissions and improve storage efficiency. In this blog, we’ll explore these new features and demonstrate how they compare to previous versions.
Let’s begin.
Peach/Parallel Processing Enhancements
“Peach” is an important keyword in kdb+, derived from the combination of “parallel” and “each”. It enables the parallel execution of a function on multiple arguments.
In kdb+ 4.1 the ability to nest “peach” statements now exists. Additionally, it no longer prevents the use of multithreaded primitives within the “peach” operation.
It also introduces a work-stealing algorithm technique in which idle processors can intelligently acquire tasks from busy ones. This marks a departure from the previous method of pre-allocating chunks to each thread, and with it, better utilization of CPU cores, which in our own test have resulted in significant reductions in processing time.
q)\s
8i
/Before: kdb+ 4.0
q)\t (inv peach)peach 2 4 1000 1000#8000000?1.
4406
/After: kdb+ 4.1
q)\t (inv peach)peach 2 4 1000 1000#8000000?1.
2035
Network Improvements
For large enterprise deployments and cloud enabled workloads, unlimited network connections offer reliability and robust performance at an unprecedented scale. With kdb+ 4.1, user-defined shared libraries now extend well beyond 1024 file descriptors for event callbacks, and HTTP Persistent Connections elevate efficiency and responsiveness of data interactions, replacing the one-and-done approach previously used to reduce latency and optimize resource utilization.
Multithreaded Data Loading
The CSV load, fixed-width load (0:), and binary load (1:) functionalities are now multithreaded, signifying a significant leap in performance and efficiency, this is particularly beneficial for handling large datasets which in our own tests have resulted in a 50% reduction in ingestion times.
Socket Performance
Socket operations have also been enhanced, resulting in a five-fold increase in throughput when tested against previous versions. With kdb+ 4.1 users can expect a significant boost in overall performance and a tangible difference, when dealing with high volume connections.
q)h:hopen `:tcps://localhost:9999
q)h ".z.w"
1004i
/Before: kdb+ 4.0
q)\ts:10000 h "2+2"
1508 512
/After: kdb+ 4.1
q)\ts:10000 h "2+2"
285 512
Enhanced TLS Support and Updated OpenSSL
Enhanced TLS and Updated OpenSSL Support guarantee compliance and safeguard sensitive real-time data exchanges. With kdb 4.1 OpenSSL 1.0, 1.1.x, and 3.x, coupled with dynamic searching for OpenSSL libraries and TCP and UDS encryption, offers a robust solution for industries where data integrity and confidentiality are non-negotiable.
Furthermore, TLS messaging can now be utilized on threads other than the main thread. This allows for secure Inter-Process Communication (IPC) and HTTP operations in multithreaded input queue mode. Additionally, HTTP client requests and one-shot sync messages within secondary threads are facilitated through “peach”.
More Algorithms for At-Rest Compression
With the incorporation of advanced compression algorithms, kdb+ 4.1 ensures maximized storage efficiency without compromising data access speed. This provides a strategic advantage when handling extensive datasets.
New q Language Features
Kdb+ 4.1 introduces several improvements to the q language that will help to streamline your code.
- Dictionary Literal Syntax
With dictionary literals, you can concisely define dictionaries. For instance, for single element dictionaries you no longer need to enlist. Compareq)enlist[`aaa]!enlist 123 / 4.0 aaa| 123With
q)([aaa:123]) / 4.1 aaa| 123This syntax follows rules consistent with list and table literal syntax.
q)([0;1;2]) / implicit key names assigned when none defined x | 0 x1| 1 x2| 2 q)d:([a:101;b:]);d 102 / missing values create projections a| 101 b| 102 q)d each`AA`BB`CC a b ------ 101 AA 101 BB 101 CC - Pattern Matching
Assignment has been extended so the left-hand side of the colon (:) can now be a pattern./Traditional Method q)a:1 / atom /Using new method to assign b as 2 and c as 3 q)(b;c):(2;3) / list /Pattern matching on dictionary keys q)([four:d]):`one`two`three`four`five!1 2 3 4 5 / dictionary /Assigning e to be the third column x2 q)([]x2:e):([]1 2;3 4;5 6) / table q)a,b,c,d,e 1 2 3 4 5 6Before assigning any variables, q will ensure left and right values match.
q)(1b;;x):(1b;`anything;1 2 3) / empty patterns match anything q)x 1 2 3Failure to match will throw an error without assigning.
q)(1b;y):(0b;3 2 1) 'match q)y 'y - Type checking
While we’re checking patterns, we could also check types./Checks if surname is a symbol q)(surname:`s):`simpson /Checks if age is a short q)(name:`s;age:`h):(`homer;38h) /Type error triggered as float <> short q)(name:`s;age:`h):(`marge;36.5) / d'oh 'type q)name,surname / woohoo! `homer`simpsonAnd check function parameters too.
q)fluxCapacitor:{[(src:`j;dst:`j);speed:`f]$[speed<88;src;dst]} q)fluxCapacitor[1955 1985]87.9 1955 q)fluxCapacitor[1955 1985]88 'type q)fluxCapacitor[1955 1985]88.1 / Great Scott! 1985 - Filter functions
We can extend this basic type checking to define our own ‘filter functions’ to run before assignment.
/ return the value (once we're happy) q)tempCheck:{$[x<0;' "too cold";x>40;'"too hot" ;x]} q)c2f:{[x:tempCheck]32+1.8*x} q)c2f -4.5 'too cold q)c2f 42.8 'too hot q)c2f 20 / just right 68fWe can use filter functions to change the values that we’re assigning.
q)(a;b:10+;c:100+):1 2 3 q)a,b,c 1 12 103Amend values at depth without assignment.
q)addv:{([v:(;:x+;:(x*10)+)]):y} q)addv[10;([k:`hello;v:1 2 3])] k| `hello v| 1 12 103Or even change the types.
q)chkVals:{$[any null x:"J"$ "," vs x;'`badData;x]} q)sumVals:{[x:chkVals]sum x} q)sumVals "1,1,2,3,5,8" 20 q)sumVals "8,4,2,1,0.5" 'badData
Happy Coding
We hope you are as excited as we are about the possibilities these enhancements bring to your development toolkit. From parallel processing and network scalability to streamlined data loading, secure transmissions, and optimized storage, our commitment is to empower you with tools that make your coding life more efficient and enjoyable.
We invite you to dive in, explore, and unlock the full potential of kdb+ 4.1. Download our free Personal Edition today.
Happy coding!
