By Dave Thomas
Businesses are processing exponentially more data from clicks, swipes, micropayments, cyber packets, social feeds, and meter readings today. The financial services industry has been doing this on a large scale for the past two decades.
The financial services sector has coped with steadily increasing data volumes by using a simple scalable data architecture composed of a real-time database (RDB) and a historical database (HDB). One of the major benefits of this architecture is that it is easy to scale up or out simply by adding additional RDBs and HDBs.
The in-memory RDB addresses real-time business needs while the HDB is an immutable record of all past transactions. By querying both the RDB and the HDB one can always get a consistent view of the world and the state of the business. Increasingly, RDBs are being placed in huge non-volatile memory and HDBs are being stored on fast SSDs. Working smartly with today’s memory solutions enables businesses to query all their data in near real-time providing a consistent picture directly assembled from the raw transactional data as required.
This widely used RDB/HDB architecture has lately been rediscovered by the Big Data community. Variations on the model have been described as Lambda Architecture, Gartner’s Hybrid Transaction/Analytical Processing (HTAP) and Forrester’s Translytical DB and Event Sourcing.
To read more Gartner research on HTAP and a cybersecurity use case where we use Kx technology with the best practice data architecture design, check out the report Real-Time Insights and Decision Making using Hybrid Streaming, In-Memory Computing Analytics, and Transaction Processing.