Palo Alto (1 Jun 2003) – Traditional data analytics and data warehousing cannot scale to handle the increasing volumes of data banks are required to manage, according to Janet Lustgarten, CEO of Kx Systems, the California-based data management provider. Lustgarten said that a new operational paradigm was needed if banks were to develop real-time capabilities for analyzing historical and streaming data, including stock ticker feeds and clickstream data from websites. “The biggest challenge is the volume of data,” she said. “Not only is the data volume doubling, but the regulatory requirements for managing that historical data are increasing.” Traditionally, historical data and real-time streaming data were maintained in separate databases, which generated reports at the end of the day or two to three days later. The new operational paradigm demands a real-time view of the business. However, Lustgarten said traditional relational databases were not up to the task. “They do not have the built-in capabilities to handle real-time analysis because of the volumes of data,” she said.
The next generation of database technology is faster and facilitates real-time analysis of historical, relational and streaming intraday data, which can them be used to predict future events and enhance competitiveness. In June, Kx Systems is due to roll out the next release of its kdb data management solution, kdb/plus. Lustgarten said the new release would be faster, with the ability to store two billion transactions instead of 30 million under the previous 32-bit architecture. “Thirty million transactions is being exceeded by firms that are doing real time feeds from multiple exchanges,” she stated.
Kx’s kdb series combines streaming and historical data into a single relational database and supports time-series data queries. “Ultimately there is greater transparency, and it helps reduce the exposure of an organization, especially those that are dealing with counterparties all over the world.” Its kdb/tick solution, which allows for real-time analysis of tick data feeds and end of day transfer to historical databases, is being used to track trades and quotes on European equities exchanges. Its data management systems, are also being used to distribute real-time data to customers via the web. Island ECN, one of the largest electronic marketplaces for NASDAQ Securities, placed a web front end on its kdb installation to enable subscribers to receive real-time transaction status reports on more complex trades, which it hopes will ultimately reduce call centre enquiries.