What is kdb+?
Kdb+, introduced in 2003, is the new generation of the kdb database. Like kdb, kdb+ is designed to capture, analyze, compare, and store data -- all at high speeds and on high volumes of data. But more than that, kdb+ was architected specifically to meet the emerging needs of leading-edge, real-time business.
TOP

How is kdb+ suited for real-time business?
Most data management/data analysis solutions divide the world into real-time/in-memory/front-end data and historical/on disk/back-end data. The division makes it easier for partial approaches to claim proficiency at one or the other. Having separate front-end and back-end data management worked all right until recently. Now enormous growth in the data volumes collected by business, along with the need for instant analysis of data, and real-time comparisons of in-memory to historical data, are becoming critically important to competitive differentiation. The firms that are first to market with these real-time business applications are the ones who can maintain and expand their competitive strategies.

With kdb+ there is no architectural split between the front end and the back end data management and analysis. We provide a single architecture for managing and analyzing data across the entire data management chain, maintaining exceptional performance throughout. In addition, kdb+ was designed from the outset to use 64-bit memory, because 64-bit addressability is essential to holding increasing volumes of streaming data in memory. It was also architected for extremely low latency, enabling such time-critical applications as auto-trading and real-time risk management.

Kdb+ takes full advantage of 64-bit platforms, giving you unlimited room to grow.

TOP

Why is a unified architecture so important?
It's enables leading-edge customers to rapidly develop and deploy real-time applications that deliver high-performance for business-critical applications including: operational risk management, backtesting of trading strategies, business activity monitoring, and other applications that quickly identify out-of-range patterns so that the business can respond in real-time.

The greater performance lead that kdb+ gives our customers translates to increased capability to create competitive strategies.
TOP

Which platforms does kdb+ run on?
Kdb+ is available today for industry-standard 64-bit architectures (AMD Opteron, Intel Xeon, and Sun) running Linux, Windows, Solaris, or Mac OS X.
TOP

Why did you develop a next-generation database product?
Kx was founded in 1993, and our kdb database has been in use by leading firms since 1998. In that time, we have seen customer needs evolve. A major business driver for the enterprise today is the requirement to analyze increasing volumes of data – on financial or energy trading transactions, for telecom usage analysis, for real-time CRM, in regulatory compliance/risk management, and in other high-volume areas. Firms need immediate results on these analyses, even when billions of records are involved. That’s what real-time business is all about: viewing and analyzing what is occurring in the business right now and comparing it on the fly to historical patterns. Developed for high data volume applications, kdb+ expands a firm’s ability to capture, analyze, compare, and store enormous amounts of data -- both streaming and on disk -- with analysis results in real-time.
TOP

Is kdb+ just an in-memory database?
No. Kdb+ provides a full relational database management system with time-series analysis that handles data in memory as well as stored data on disk. For advanced applications such as backtesting of auto trading strategies or operational risk management, it is essential to be able to compare streaming data against history. You must be able to understand where the business has been in order judge and act upon real-time occurrences. Approaches that handle in-memory data alone or historical data alone can’t meet the needs of today’s real-time enterprise, where accurate comparison on the fly is becoming increasingly important. Approaches that try to combine a streaming or in-memory product from one vendor with a historical product from another can't deliver the performance necessary for real-time business, because they have to cope with two separate architectures. Excess overhead is unavoidable with multiple architectures.
TOP

I've heard it's not possible to run SQL series on streaming data. Is that true?
That's untrue. Our customers have been running time-series or SQL queries on streaming data since 2001 and achieving results in real-time, even on complex queries involving millions of records.
TOP

What features contribute to the performance of kdb+?
We’ve refined the architecture in a number of ways, based on the company’s 10 years of experience:

• We expanded the data types for greater flexibility, particularly in writing time-series analytics. While other time-series companies supply a limited time-series language, kdb+ was specifically developed to let leading-edge customers go beyond limits.

• We enhanced the speed and efficiency of application development by combining our general programming, relational, and time-series languages into a single, concise programming language – q. The q language is integrated into the database, contributing to very high query performance. Q uses English-like commands and a simple syntax. C or SQL programmers typically learn q in less than a day. (See the Kdb+ Primer written by Dennis Shasha, Associate Professor of Computer Science at NYU's Courant Institute.)

• We reduced overhead and latency to maintain leadership performance even as data volumes keep rising. For example, data on many securities exchanges is doubling each year. Our product strategy has always been to maintain the lead in performance for complex data analysis, and with kdb+ we have further extended that lead for our customers.
TOP

As a relational database vendor, how do you handle streaming data?
Our product kdb+tick is a real-time tickerplant application layered on kdb+. As data streams in from a data feed or other source of streaming data, it becomes available for immediate relational analysis. In addition, the data is logged so that, in case of a system failure, you do not lose the day's data, as you would with products that support streaming or in-memory data only. Periodically, the log file is written to the historical database -- a day's worth of real-time data (easily 50 million records) can be written to the database in couple of minutes. In fact, kdb+tick is so fast at managing streaming, in-memory, and stored data that some of our customers have used it to eliminate the traditional end of day, where the database is taken off-line. Because kdb+ runs at top efficiency 24x7, it can be used to program advanced applications such as global 24x7 trading.
TOP

Is it really necessary to save all that data?
Only if your firm's strategy is to offer highly competitive, leading products. One of the reasons we developed kdb+tick originally was in response to trading departments asking us: isn't there a way we can save the streaming data so we can analyze it later? While it's true that small trading problems can be solved using a streaming data or in-memory database alone, big, strategic problems require you to be able to save data and to compare streaming or in-memory and historical data on the fly, without losing speed anywhere along the line.
TOP

Aside from kdb+tick, do you have other layered products for kdb+?
To date, we have two in addition kdb+tick:

• Kdb+tow is an application that enables traders to test sophisticated algorithms by replaying historical ticks through their models.

• Kdb+taq is a fast loader for NYSE TAQ data (distributed via CD/DVD or FTP) that enables you to create a full 10+ year history of NYSE TAQ data quickly, update it daily, and have it immediately available for relational, time-series analysis in kdb+.

TOP

Why should development teams and IT departments invest in new technologies such as kdb+, when the trend is toward standard technologies?
Doing business in real time demands new technologies and fast ROI. The volumes of data encountered in business today are like nothing the world has seen before -- and they are growing rapidly. In addition, firms need to understand how streaming data relates to historical patterns. Conventional database paradigms are floundering, because the relational databases of the 1980s are no longer able to keep up with escalating volumes of data. The old model of overnight reporting is no longer acceptable in real-time business. The business intelligence/OLAP/data warehousing structures that were built to make relational databases more efficient are also under increasing pressure to deliver faster analysis -- and they can't. Newer in-memory databases and streaming data products deliver speed as long as the data is in memory, but they don't meet the needs of real-time business, because they solve only a small part of the data volume and data analysis problem.
TOP

What if I've already invested considerable resources in developing Java, C, and .net programs?
Kdb+ provides native C and Java interfaces. In addition, to make up for Java's inability to handle large arrays, you can use our JDBC driver. To further assist you, the q language data types map directory to Java and .net.
TOP

Is it complicated to administer a kdb+ database?
Not at all. Kdb+ is remarkably simple to manage, because native operating system routines are used for much of the file management, including backup and restore.
TOP

Do you need a big server to run kdb+?
No. Most of our customers begin with a 2- or 3-CPU system and grow from there. As you build a historical database, you will need multi-terabyte storage, but kdb+ is flexible -- you can use local storage, SANs or any combination.
TOP

But what if you need a high-availability environment?
No problem -- get as big and redundant as you want to. Many Kx customers have implemented large, fully-redundant systems, including redundant tickerplants for kdb+tick. We support failover, so there is no loss of data or performance. We provide local logging as well as complete replication between data centers. Through our relationship with Cassatt, we also enable IT organizations to deploy kdb+ in distributed environments and dynamically allocate system resources to meet real-time spikes, such as unusual peaks in market data. That way you don't need to over-invest in big hardware dedicated to kdb+, but you also have extra capacity available instantly, when you need it. Contact us for a demo.
TOP

Do you think the kdb+ database will replace Oracle, DB2, SQL Server and other relational databases?
As long as another database meets your needs, use it. But for applications where you're waiting too long for reports, or you don't have the data for implementing a real-time business application, consider kdb+.
TOP