WHAT IS IN-MEMORY COMPUTING?
In-Memory Computing (IMC) is a solution architecture the exploits the speed of storing application data in RAM, possibly across a number of clusters, rather than on disk in order to deliver real-time processing insights. It is complemented by on-disk or solid-state storage for recovery, archiving and accessing historical data. Gartner describes IMC as:
“An emerging paradigm enabling user organisations to develop applications that run advanced queries or perform complex transactions on very large datasets at least one order of magnitude faster — and in a more scalable way — than when using conventional architectures. It achieves this by storing application data in a computer’s main DRAM, rather than on electromagnetic disks, and without compromising the availability, consistency and integrity of that data.”
In the case of Kx this means high-speed processing of enormous volumes of high velocity, time-series data where the combination of in-memory processing and columnar design significantly outperforms traditional relational databases.
With kdb+ the powerful combination of an in-memory database, a time-series enhanced superset of SQL, tight integration to external systems, and a fully featured programming language embedded directly on the data allows Kx to deliver in a uniquely cost-effective and performant fashion.
How is the high speed achieved?
High performance is achieved in a number of ways. In the first instance it is achieved by retaining the most frequently – as opposed to the most recently – used data in memory so that it can accessed virtually instantaneously when executing queries and analytics.
In-memory performance is further improved by using columnar as opposed to row-based storage of data. The contiguous storage within a columnar design enables swifter read access to data as well as improving compression profiles and thereby further reducing memory overhead
Kx’s low footprint accelerates its IMC performance in two significant ways. Firstly, by minimizing the amount of memory space taken up by the database itself (less than 800Kb), it maximizes the amount of the critically important application data that can be stored in the remaining memory. Secondly, it enables it to operate within the L1 and L2 CPU memory caches that offer the fastest possible access and processing speeds.
The resulting high performance of Kx is reflected in its dominance of industry benchmarks published by the Securities Technology Analysis Center (STAC), an independent technology evaluation body.
How is Scalability and Resilience Achieved?
No, in-memory does not mean in-memory only. Scalability is achieved by Kx’s ability to combine in-memory and historical data from disk or other storage media. This approach is illustrated below:
- The Tickerplant receives both streaming and batch data which it logs, processes and publishes to downstream consumers
- The Real-time database (RDB) captures all intraday data and typically writes to disk at end of day (but clearly the frequency depends on the volume and available memory)
- The Historical database (HDB) processes read historic data from disk and provide to other consumers
In this way users and algorithms have access to the complete landscape of real-time and historical data. By using memory mapped tables and storing data on-disk in column arrays, in the same manner as data in memory, Kx leverages the operating system’s file cache when retrieving historical data as is automatically page faults into memory to become instantly accessible. In effect the size of tables within kdb+ is then limited only by disk space rather than by memory size.
In the event of failure, the real-time database automatically re-syncs itself with the contents of the ticker-plant’s log file before receiving further updates.
What’s the benefit of IMC and where is it used?
Kx’s IMC enables high speed analytics that traditional databases cannot match and provides the scalability to process the vast quantities of data needed for historical analytics, simulations and training of Machine Learning algorithms – see below some indications on how fast Kx can perform
- Ingest – Process and store over 30 million events /second (micro batch)
- Scan – Search in-memory tables at 4 billion records per second per core
- Analyze – Aggregate over 40 billion events in under 2 minutes
- Performance – Top performing time-series database according to STAC Research
- Latency – Sub-millisecond latency for streaming event processing
IMC enables instant detection of insights, threats and opportunities lying otherwise undiscovered within high volume, high velocity data sets. As an example of how IMC can be applied please read our paper on Gartner research describing how it enables organizations to achieve hybrid transactional/analytical processing (HTAP) where OLAP-type queries can be run in parallel with OLTP processing on operational databases without compromising performance.
IMC is used anywhere speed is critical. Examples would include cyber security and market surveillance where after-the-event detection of threats or violation is simply too late. IMC enables instant detection of insights, threats and opportunities lying otherwise undiscovered within high volume, high velocity data sets.
Please read our paper on Gartner research for more information on how it enables organizations to achieve hybrid transactional/analytical processing (HTAP) where OLAP-type queries can be run n in parallel with OLTP processing on operational databases without compromising performance.
The basis for Kx Technology is a unique integrated platform which includes a high-performance historical time-series columnar database called kdb+, an in-memory compute engine, and a real-time streaming processor all unified with an expressive query and programming language called q.
Designed from the start for extreme scale, and running on industry standard servers, the kdb+ database has been proven to solve complex problems faster than any of its competitors.LEARN MORE