Big time-series Data
Frequently asked questions.
- 1. What is kdb+?
kdb+ from Kx Systems is a high-performance, high volume database designed from the outset in anticipation of vast increases in data volumes. kdb+ has built-in multi-threading, multi-processing, and multi-server capabilities. The same architecture is used for real-time and historical data. The database incorporates its own powerful query language, q, so that analytics can be run directly on the data.
- 2. Is the q query language needed to use kdb+?
A basic knowledge of q is needed to set up and administer a kdb+ database. Our experience is that most developers find q easy to learn and incredibly productive — even those whose prior experience is solely with traditional programming languages. Since q includes a SQL-like subset, it is very familiar to database programmers, who can be up and running within days.
The query language allows powerful ad hoc queries on-the-fly. This is important as companies are being faced with a huge number of ad hoc reporting and research requirements, putting significant pressure on their data infrastructures.
Also, while end users may access kdb+ from a GUI, and know nothing of the database back end, in practice, many actually prefer to work directly with q in a kdb+ session, making use of the ability to write their own custom queries.
- 3. How does kdb+ scale?
kdb+ is multi-threaded and multi-process, and can scale across many machines, to petabyte sizes. It will run on clusters, grids, clouds, and other large-scale distributed architectures.
- 4. Which platforms does kdb+ run on?
kdb+ runs on industry-standard 64-bit architectures running Linux, Windows, Solaris (SPARC and Intel), or Mac OS X.
- 5. Is kdb+ simply an in-memory database?
kdb+ provides a full relational database management system that handles data in memory as well as data stored on disk.
For certain applications it is essential to be able to compare real-time data against history. Approaches that handle in-memory data or historical data alone, or that try to combine a real-time or in-memory product from one vendor with a historical product from another cannot deliver the performance necessary for real-time decision-making, because they have to cope with two separate architectures. Excess overhead is unavoidable with multiple architectures.
- 6. Is it necessary to save all in-memory data?
One of the reasons kdb+ was developed was in response to customers asking: isn't there a way to save the real-time data so it can be analyzed later? While it's true that certain problems can be solved using a real-time data or in-memory database alone, big, strategic problems require the ability to save data and to compare real-time or in-memory and historical data on the fly, without losing speed anywhere along the line.
- 7. How is real-time data handled?
kdb+ supports data streams from a data feed or other source of real-time data, and makes it available for immediate relational analysis. In addition, the data can be logged so that, in case of a system failure, no data is lost. Periodically, the in-memory data is written to the historical database — a day's worth of real-time data can be written to the database in a few minutes. In fact, kdb+ is so fast at managing real-time, in-memory, and stored data that some of our customers eliminate the traditional end-of-day where the database is taken off-line. Because of this, kdb+ can be used for advanced applications requiring global 24x7 access.
- 8. Does kdb+ support a high-availability environment?
Many Kx customers have implemented large, fully-redundant systems. kdb+ supports failover, so there is no loss of data or performance. There is local logging as well as complete replication between data centers. IT departments can deploy kdb+ in distributed environments, like clouds, and dynamically allocate system resources to meet real-time spikes, such as unusual peaks in market data. There is no need to over-invest in big hardware dedicated to kdb+, while there is extra capacity immediately available when needed.
- 9. Does kdb+ support compression?
kdb+ supports two modes of compression, on-disk and over interprocess communication. On-disk compression is supported through a choice of algorithms and a broad range of settings, driving down storage requirements and latency, and optimizing CPU usage.
- 10. What API's are available for kdb+?
kdb+ has a very simple API, for easy connectivity to external graphical, reporting and legacy systems. There are interfaces to C/C++, Java, .Net, R, Matlab, Perl, Python and others. A WebSocket interface allows for a direct, bi-directional, full-duplex connection between a browser and an application, particularly useful for high-performance browser-based applications, such as visualizing real-time data. kdb+ also supports ODBC and JDBC to help with migrating data between kdb+ and traditional databases or applications like Excel.
- 11. Are there any benchmarks for kdb+?
The Securities Technology Analysis Center has a tick database benchmark, the STAC-M3. This has been run using kdb+ and typical queries on the equivalent of one year's New York Stock Exchange TAQ data, and various hardware platforms. This series of benchmarks enables users and vendors to compare the performance of their database solutions against audited, third-party measurements.
- 12. How can I try kdb+?
Kx provides a free trial version of kdb+ which you can find on our website.
Kx also offers on-site evaluations of kdb+ following preliminary discussions with a sales representative.
- 13. How can I learn to use kdb+?
There is a rich community of kdb+ users who contribute their knowledge and best practices on code.kx.com and kxcommunity.com. We recommend a helpful kdb+ tutorial on code.kx.com as well as a short intro to the q language. There is also a message board for Kx customers, called the k4 listbox.
- 14. How can I purchase kdb+?
Please reach out to one of our Kx sales representatives to see if kdb+ is the right solutions for you.
If you have already been in contact with a Kx sales representative and you are interested in discussing purchasing kdb+, you can find out how to buy on our purchase page and choose the best option for you.
- 15. What support does Kx offer?
Priority technical support is considered to be an essential part of the product, and Kx offers rapid response worldwide. When issues are reported, there is a quick response from someone who knows the code, not a scripted response from an outsourced support center.
- 16. What about training, consulting, and other support?
There is an active email user forum where clients can post questions and discuss topics of interest to the community, and an active wiki for documentation, cookbooks, tutorials, and software addons.
Kx partner companies deliver consulting, training, and installation support globally, and also provide their own add-on dashboards, feedhandlers, and development and visualization tools.