ポイント
- Market data has structural properties that most general-purpose, open-source infrastructure wasn't built for.
- General-purpose data platforms introduce compounding inefficiencies under high-frequency, high-cardinality market workloads.
- At 8% memory and 1.5% CPU, KDB-X outperformed fully-resourced open source systems in 58 of 64 benchmarks.
- Over five years, compute amplification and engineering overhead dwarf licence cost.
- Capital markets data infrastructure demands architectural precision — KDB-X was built to deliver it.
Alpha lifecycles are shorter, data volumes are higher, and regulatory scrutiny is tighter.
Firms that can validate signals quickly, measure execution accurately, and maintain a clear audit trail are at an advantage.
All three depend on the same thing: data infrastructure that was built for this workload.
Open-source infrastructure has a ceiling
Open-source data infrastructure is deeply embedded in the modern quant stack. The ecosystems are mature, the tooling is extensive, and the talent pool is deep. For certain workloads — exploratory analytics, feature stores, reporting pipelines — general-purpose systems carry less architectural cost than others.
The firms that have built on open-source foundations made a rational decision given the information available: flexibility, community support, cost visibility, and freedom from vendor dependency are all legitimate considerations in an infrastructure evaluation.
But general-purpose infrastructure was designed to serve diverse workloads across industries, and that breadth is precisely its limitation in a capital markets context. The architectural trade-offs that make these systems flexible are the same ones that generate friction under high-cardinality, high-frequency market data workloads. That friction rarely surfaces at the point of adoption. It shows up later in storage overhead, compute amplification, and widening latency distributions until it’s material enough to be a problem and embedded enough to be difficult to address.
Market data isn’t generic time-series data
The structural properties of market data are specific. It’s predominantly append-only and time-ordered, with the exception handling — late arrivals, feed corrections, out-of-order events — that any production feed handler has to absorb. It’s high cardinality across symbols, venues, and identifiers, correlated across instruments and asset classes, and queried across windows that range from intraday to multi-year. For latency-sensitive strategies, it’s measured in microseconds; across the broader capital markets stack, millisecond and second-level granularity is more common but the volume and cardinality properties hold regardless.
These properties determine how well data compresses, how efficiently it is accessed in memory, how joins perform, and how the system behaves under load.
General-purpose databases can store and query time-series data effectively. Their internal execution models, however, are optimised for flexibility and horizontal scaling across diverse industries — which carries architectural trade-offs that become visible under high-cardinality financial workloads: storage grows faster than anticipated, CPU utilisation rises disproportionately during wide scans, and query latency distributions widen exactly when you need stability.
Individually, each friction point is manageable. Cumulatively, over multi-year retention and multi-asset scale, they drive real cost.
What benchmark evidence shows
To test these differences empirically, we ran controlled benchmarks using the TSBS DevOps workload, a standardised framework that removes environment variables and isolates architectural performance.
Every system ingested the same dataset and executed identical aggregation, filtering, and group-by queries on identical hardware.
KDB-X ran in Community Edition mode: one q process, 16 GB of memory, four execution threads — roughly 1.5% of available CPU threads and 8% of system memory. Competing open-source systems ran in default configuration with full hardware access.
The results were consistent:
- KDB-X outperformed in 58 of 64 benchmark scenarios
- The closest competitor averaged 3.4× slower on geometric mean across all queries
- On worst-case queries, some systems showed order-of-magnitude latency degradation
- The performance gap held across both short-range and multi-year datasets
Consistent performance across aggregation-heavy workloads, using a fraction of the hardware, reflects an architecture designed for this data rather than tuned for a single case.
Scepticism toward vendor-produced benchmark data is reasonable. The TSBS framework is standardised and openly reproducible — we’d encourage any team to run it against their own workload profile and draw their own conclusions.
On vendor dependency: it’s a legitimate concern. KDB-X is built on decades of open standards in the kdb+ ecosystem, with documented APIs and interoperability with the tooling quant teams already use. Any specialised system carries some degree of dependency; the relevant calculation is whether the performance and cost characteristics justify it given your specific environment.
How this plays out in production economics
In production, cost is driven by how efficiently a system uses compute and memory over time.
In cloud environments, total cost scales with runtime and memory footprint. When queries take longer to complete and require more memory to execute, those costs compound directly.
The benchmark results make this visible. KDB-X delivered faster query performance while operating on a fraction of the available hardware. Competing systems were slower and required full resource allocation to complete the same workloads.
If a system is 3× slower and uses materially more memory, the cost impact is multiplicative rather than incremental. A workload that takes longer to run and consumes more memory at the same time increases total spend per query, per user, and per research cycle.
This becomes more pronounced as data volumes grow. Multi-year tick retention, wide aggregation queries, and cross-asset joins increase both execution time and memory pressure. Systems that scale inefficiently require progressively more compute to maintain acceptable latency.
Infrastructure spend follows runtime and memory usage. Systems that use more of both become more expensive as data grows. Engineering effort follows the same pattern. Time spent tuning query performance, managing storage growth, or working around execution limits is time not spent on research or strategy development.
Research throughput is infrastructure-dependent
The speed at which a signal can be validated across full history directly affects how many ideas reach production.
When aggregation or feature engineering degrades as history grows, iteration slows. When historical analytics and live production run on separate systems, reconciliation overhead accumulates. The benchmark differentials observed in aggregation, filtering, and group-by workloads translate directly into validation cycle length, experiment throughput per researcher, and compute cost per strategy.
At capital markets scale, this limits how much research a team can run.
The right evaluation frame
Most firms evaluating this aren’t starting from scratch. Open-source components are already embedded in the stack, and the useful question is where architectural mismatch is creating the most friction — and what it costs to leave it there.
Time-series infrastructure underpins strategy research, live signal generation, execution analytics, replay, and regulatory oversight. In those functions specifically, a forward-looking evaluation should consider:
- Storage growth under full tick retention over multiple years
- Compute amplification as analytical workloads widen
- Worst-case latency characteristics, not averages
- Continuity between research, production, and oversight environments
- Deterministic time alignment for replay and audit
These criteria show where architectural decisions start to affect cost and performance. A system that slows down as tick data grows will limit research throughput and increase cost as analytical demand increases.
KDB-X was designed around these constraints specifically — time-aware storage, vectorised execution across high-cardinality datasets, and a common data model and query layer across research, production, and oversight — reducing the translation overhead that typically accumulates when separate systems handle each function. That architectural continuity is what the benchmark results reflect: consistent performance across workload types, not optimisation for a narrow test case.
Open-source components that are well-matched to their workload belong in the stack. For the infrastructure carrying your highest-frequency, highest-cardinality data — whether that’s microsecond tick capture or millisecond signal generation — the evaluation question is whether the system was built for that environment or adapted to it.
KX has supported the full trade lifecycle across research, execution quality, and oversight for over three decades. If you’re evaluating time-series infrastructure for capital markets workloads, we’re happy to walk through the benchmark methodology or map it to your specific environment. Reach out for a demo here.
