kdb Products
Overview
KDB.AI
kdb+
kdb Insights
kdb Insights Enterprise
Capabilities
The Data Timehouse
Vector Database Explained
kdb+ Time Series Database
PyKX Python Interoperability
Services & Support
Financial Services
Quant Research
Trading Analytics
Industry & IoT
Automotive
Energy & Utilities
Healthcare & Life Sciences
Manufacturing
Telco
Learn
Overview
Featured Courses
KX Academy
KX University Partnerships
Connect
KX Community
Community Events
Developer Blog
Build
Download
Documentation
Support
About Us
Partner with Us
Become a Partner
Find a Partner
Partner Signup
Join Us
Connect with Us
By Noel Byrne
Kdb+ was the selected database component for recent STAC-M3 -benchmark tests on Dell hardware. In the tests, kdb+ 4.0 was distributed across 8 Dell PowerEdge R7525 servers, each with 2 x 64-core AMD EPYC™ 7H12 CPUs, sharing access to a Dell PowerScale F900 All-Flash Scale-Out NAS 3-node cluster with 251TiB total physical storage capacity. The database nodes were accessed over standard NFS over 100GB ethernet interface.
The results represent significant improvements on previous tests on earlier generations of Dell storage and an earlier version of kdb+. Summary figures reveal they surpassed 14 of 17 previous mean-response times including:
The benchmark results were run on the Antuco environment which uses 4.5 TB dataset residing mostly on non-volatile media to simulate performance that would be obtained with a larger real-world dataset.
When compared separately with a cloud-based configuration on 12 database servers accessing data on local NVMe SSDs it was shown to be faster in 4 of 17 mean-response benchmarks including almost 3x in NBBO (National Best Bid & Offer) and 2.4x in 1-user unpredictable interval stats. Under the Kanga benchmarks (which simulates performance on large datasets, 33-897 TB, with large numbers of concurrent requests), it was faster in all 5 mean-response times involving 100 users and faster in 3 of 5 mean-response times involving 50 users,
The test configuration provides highly scalable storage for billions of files, high read/write performance, high user-concurrency capabilities and fast interconnect between compute resources and the storage platform. Within the tests, kdb+ ensures PowerScale nodes are utilised optimally by balancing them equally to maximize I/O throughput and, where appropriate, distributing the mount points across more than one client-side network interface.
About STAC
The STAC-M3 benchmark tests were developed in 2010 by several of the world’s largest banks and trading firms to provide an independent analysis of performance improvements from emerging hardware and software innovations. Running a variety of compute-intensive operations on a large store of market data, the tests measure the end-to-end performance across the full solution stack of database software, servers, and storage.
About kdb+
kdb+ is independently benchmarked as the world’s fastest time series database and real-time analytics engine. Built for the most demanding data environments, it is trusted by the world’s leading companies for large-scale complex analytics on streaming, real-time, and historical data. It is used in capital markets for algorithmic trading, back-testing, surveillance, regulatory reporting, and research environments as well as in other industries for high-speed sensor monitoring, fault detection, predictive analytics, and machine learning. Shortly after STAC-M3 was developed, kdb+ quickly became the preferred database platform for hardware vendors running the tests because of its unparalleled performance and response times in processing time-series data.
More detail on the results and the system configuration for the Dell benchmark tests can be found in their whitepaper and STAC’s press release.
“STAC” and all STAC names are trademarks or registered trademarks of the Securities Technology Analysis Center, LLC.”