Kdb+ was the selected database component for recent STAC-M3 -benchmark tests on Dell hardware. In the tests, kdb+ 4.0 was distributed across 8 Dell PowerEdge R7525 servers, each with 2 x 64-core AMD EPYC™ 7H12 CPUs, sharing access to a Dell PowerScale F900 All-Flash Scale-Out NAS 3-node cluster with 251TiB total physical storage capacity. The database nodes were accessed over standard NFS over 100GB ethernet interface.
The results represent significant improvements on previous tests on earlier generations of Dell storage and an earlier version of kdb+. Summary figures reveal they surpassed 14 of 17 previous mean-response times including:
- Almost 17x speed-up in the 10-user market snapshot
- Over 3x speed-up in the 50-user unpredictable interval stats
- Over 4x speed-up in the 100-user unpredictable interval stats
The benchmark results were run on the Antuco environment which uses 4.5 TB dataset residing mostly on non-volatile media to simulate performance that would be obtained with a larger real-world dataset.
When compared separately with a cloud-based configuration on 12 database servers accessing data on local NVMe SSDs it was shown to be faster in 4 of 17 mean-response benchmarks including almost 3x in NBBO (National Best Bid & Offer) and 2.4x in 1-user unpredictable interval stats. Under the Kanga benchmarks (which simulates performance on large datasets, 33-897 TB, with large numbers of concurrent requests), it was faster in all 5 mean-response times involving 100 users and faster in 3 of 5 mean-response times involving 50 users,
The test configuration provides highly scalable storage for billions of files, high read/write performance, high user-concurrency capabilities and fast interconnect between compute resources and the storage platform. Within the tests, kdb+ ensures PowerScale nodes are utilised optimally by balancing them equally to maximize I/O throughput and, where appropriate, distributing the mount points across more than one client-side network interface.
About STAC
The STAC-M3 benchmark tests were developed in 2010 by several of the world’s largest banks and trading firms to provide an independent analysis of performance improvements from emerging hardware and software innovations. Running a variety of compute-intensive operations on a large store of market data, the tests measure the end-to-end performance across the full solution stack of database software, servers, and storage.
About kdb+
kdb+ is independently benchmarked as the world’s fastest time series database and real-time analytics engine. Built for the most demanding data environments, it is trusted by the world’s leading companies for large-scale complex analytics on streaming, real-time, and historical data. It is used in capital markets for algorithmic trading, back-testing, surveillance, regulatory reporting, and research environments as well as in other industries for high-speed sensor monitoring, fault detection, predictive analytics, and machine learning. Shortly after STAC-M3 was developed, kdb+ quickly became the preferred database platform for hardware vendors running the tests because of its unparalleled performance and response times in processing time-series data.
More detail on the results and the system configuration for the Dell benchmark tests can be found in their whitepaper and STAC’s press release.
“STAC” and all STAC names are trademarks or registered trademarks of the Securities Technology Analysis Center, LLC.”