by Ferenc Bodon
STAC, the finance industry standard for testing time-series data, recently announced several world records with unprecedented query execution times in their latest report using the STAC-M3 benchmark. The results were an outcome of an Intel and KX collaboration using Intel® Optane™ Persistent Memory and KX’s kdb+ scripts.
In 10 out of 17 Antuco benchmarks, the combined solution outperformed all previous public disclosed results. Of the results, 4 benchmarks had increased speeds exceeding 2.4x. This solution outperformed all publicly disclosed results in 9 out of 10 Kanaga benchmarks on solutions configured on 2 to 3 years of data.
KX’s kdb+ provides the fastest execution for 98% of STAC-M3 queries.
KX utilizes kdb+ in products such as KX Insights, a cloud-first streaming analytics platform, to deliver unprecedented speed and scalability for demanding real-time analytics.
What is STAC-M3?
The STAC benchmarking council produces benchmark standards that lead to high- value technology research and testing software. The council is driven by end-user financial services companies.
The STAC-M3 Benchmark suite is the industry standard for testing solutions that enable high-speed analytics on time series data, such as tick-by-tick market data (aka “tick database” stacks). It is highly focussed on the business of high-speed data analytics for historical in in-memory data, as used by investment banks, hedge funds and the like.
STAC-M3 represents one of the only truly independently audited set of data analytics benchmarks available to the financial services industry. The original intent of this was to offer a valuable service to FSI business and tech team end-users, by means “cutting through” all the proprietary technical marketing benchmarks available from each technology vendor. Often these benchmarks bend the rules to the preference of their underlying technology or to one small subset of the results needed. Any of our customers and prospects are freely able to examine and download the full audited benchmarks from the STAC website.
New kdb+ scripts
KX has published performance results in STAC M3 since its inception in 2010. Due to kdb+’s unparalleled speed, KX has become the “go-to” technology for system vendors to use in proofing the performance of their new technology, especially where large memory management and massive I/O capability is key.
In 2021, KX refactored the existing kdb+ scripts to support horizontal partitioning. Kdb+ already employs map-reduce internally to achieve horizontal partitioning for a single database. The partitions are distributed in directories and disks. If the database is on a distributed, parallel filesystem then users can make use of worker processes (variable .z.pd) to leverage CPU and memory capacity of available servers. The query execution is seamless for the user.
Kdb+ also provide simple tools to implement process-level map-reduce in three ways. First, parallel each so it can invoke one-shot request to execute query on remote host. Secondly, TCP socket sharding provides a simple query distribution by the Linux kernel. Lastly, multi-threaded primitives introduced in kdb+ 4.0 helps speed up in-memory calculation and the second phase of the map-reduce. The well-known features of kdb+ including vector operations, functional programming and advanced iterators makes the code brief and easy to read.
Intel® Optane™ Technology
Intel Optane Persistent Memory redefines traditional architectures, offering a large and persistent memory tier at an affordable cost. With breakthrough performance levels in memory intensive workloads, virtual machine density, and fast storage capacity Intel Optane persistent memory accelerates IT transformation to support the demands of the data era.
- Intel Optane persistent memory provides an affordable alternative to expensive DRAM, that can deliver huge capacity and accommodate demanding workloads and emerging tools like in-memory databases (IMDB)
- Intel Optane persistent memory is propelling major infrastructure consolidation. The increase in memory size from Intel® Optane™ media provides the opportunity to consolidate workloads to concentrate them on fewer nodes, ultimately saving on more deployments and maximizing previously underutilized processors due to memory constraints
- As an entirely new memory tier that has new properties of performance and persistence, architects and developers are using Intel Optane persistent memory as a springboard for innovation taking advantage of new usages around restart and replication, ground-breaking performances from braking system bottlenecks that used to severely constrain workloads and making use of what could be considered the world’s tiniest, but fastest storage device sitting on a memory bus
The test stack involved a 2-node cluster of 4-socket Lenovo ThinkSystem SR860 V2 servers with 4 x Intel® Xeon Platinum 8380HL (Cooper Lake) 28-core CPUs @ 2.9GHz. The data was stored on Intel® Optane™ Persistent Memory (PMem) 200 Series configured in Storage Over App Direct Mode with Filesystem-DAX namespaces.
About KX
KX, the leading technology for real-time continuous intelligence, is part of FD Technologies plc, a group of data-driven businesses that unlock the value of insight, hindsight and foresight to drive organizations forward. KX Streaming Analytics, built on the kdb+ time-series database, is an industry leading high-performance, in-memory computing, streaming analytics and operational intelligence platform. It delivers the best possible performance and flexibility for high-volume, data-intensive analytics and applications across multiple industries. The Group operates from 15 offices across Europe, North America and Asia Pacific and employs more than 2,500 people worldwide. For more information visit www.kx.com
Reference
