Tick-X vs. the Scalable Ingestion Blueprint: Two Ways to Scale kdb+ Tick Ingestion

Tick-X is a reference architecture that splits the overloaded RDB in a classic kdb-tick stack into dedicated processes for write-down, real-time queries, and intraday serving. The scalable ingestion blueprint applies the same split at higher volume, sharding ingestion across multiple tickerplants for real NYSE TAQ-scale workloads. This piece covers how each works and when to use which.

If you’ve run a kdb+ tick stack in production, you know the shape of it by heart. A tickerplant fans out updates, the real-time database holds today in memory, a historical database serves everything else, and a gateway connects the two for callers who don’t want to care which is which. It’s a pattern the whole industry standardized on, and it works beautifully, right up until one process becomes overloaded as ingestion rates and data volumes increase.

In the classic setup, the real-time database (RDB) ingests every update, answers every real-time query, and at end-of-day writes the entire day to disk with a single .Q.hdpf. Three core responsibilities, all handled by one process. During write-down the RDB can’t serve queries, because the save and the query are competing for the same thread. Memory grows monotonically all day, so the same process that was comfortable at market open is straining by close. And a crash at 3pm loses the whole in-memory buffer – your only backstop is the tickerplant (TP) log file, and a corrupt log means data loss.

We have built two resources to highlight and address that problem. The first is a reference architecture, Tick-X, that is built on KDB-X keeping the familiar kdb-tick shape but splits the overloaded RDB into pieces that each do a specific job. The second is a blueprint for scaling the ingestion of market data that takes the same idea and pushes it out horizontally, under real NYSE TAQ volumes. The reference architecture acts as a template for system implementation; the blueprint shows how it scales in production.

This piece covers how the first was built from patterns you already know, how the second branches off it, and where the two diverge.

What Does the Base kdb-tick RDB Do Today?

Here’s the end-of-day handler from a stock kdb-tick RDB. It’s four lines and it has been mirrored across countless production systems:

.u.end:{
  t:tables`.;
  t@:where `g=attr each t@\:`sym;
  .Q.hdpf[`$":",.u.x 1;`:.;x;`sym];
  @[;`sym;`g#] each t
  }

The foundation here is solid. .Q.hdpf saves the day’s partition, clears memory, and tells the historical database (HDB) to reload. The main issues are when it runs, what it blocks while running, and how it struggles with the growing data volumes typical of production systems.

Tick-X keeps the bones of kdb-tick. The pub/sub tickerplant is unchanged, the gateway still queries across the relevant database tiers, and schemas still lead with time and sym, carry the g#sym attribute, and get enumerated against a shared sym domain on the way to disk. What changes is how the RDB tasks are now delegated to their own distinct processes.

How Does Tick-X Split the RDB?

The split is straightforward: the main RDB keeps the write-down job, but gives up everything else. It subscribes to the tickerplant, and instead of holding the whole day for a single EOD dump, it flushes slices on a timer. Every few minutes it writes the rows older than the flush window to a fresh int-partition on disk, drops them from memory, and tells an intraday database to reload. The cutoff is simply a time-of-day subtraction:

cutoff:"n"$.z.p - .rdb.flushIntv * 0D00:01

The chained RDB takes the query job. It’s a second, independent subscriber to the same tickerplant, queryable via the gateway for real-time data. But its end-of-day handler doesn’t call .Q.hdpf; it just empties its tables, as the data is already persisted by the main RDB. Queries hitting this process never wait behind a flush, because this process never flushes. But they will still be blocked by garbage collection due to q’s single-threaded nature. Set the garbage-collection thresholds with this in mind.

Lastly, the intraday database (IDB) takes a job the old design didn’t really have: serving today’s older data. Once the main RDB has flushed a slice to disk, the IDB loads those int-partitions into memory and answers queries on them. So “today” is now split across two tiers: the most recent rows still live in the chained RDB, while everything older that’s already been flushed lives in the IDB, and the gateway routes across both.

That split raises an obvious question. If the recent rows are in one process and the flushed rows are in another, how do you prevent a row from showing up in both, or falling through the gap between them?

What Is the Flushed Watermark in Tick-X?

Tick-X answers this with a single value we call the flushed watermark (W). Every row with time < W is already persisted to the IDB. The main RDB writes W to a file on each flush and, more importantly, tells the chained RDB to shed everything below it. The chained RDB drops those rows and keeps only the un-flushed tail. The two tiers stay disjoint, so a gateway querying across both the RDB and IDB can combine results without duplicate data.

The shed signal doesn’t go over a direct connection from the main RDB to the chained RDB, but instead through the tickerplant:

broadcastWatermark:{[w] if[count shedSubs; (neg shedSubs)@\:(`.rdb.shedTo;w)]}

The chained RDB receives its data from the tickerplant on one connection, in order. If the shed instruction rides that same connection, then by the time it arrives the chained RDB has already received every row the instruction is telling it to drop. Routing the signal in another way subjects it to a race condition: the “drop everything before W” message could overtake a row that’s still in flight, and that row would land after the shed and never get cleaned up. Sending it through the tickerplant instead makes this an ordered event by construction.

W does a second job that only matters when something goes wrong: it’s a floor for the flush itself. On restart, the RDB replays the tickerplant log and rebuilds the whole day in memory, including rows it had already flushed hours ago. Without a floor it would flush them again, duplicating int-partitions and doubling counts in the EOD merge. Because the RDB drops everything below W before every flush, replayed rows that are already on disk are never re-persisted. This is all meant to exist as a template for addressing some of the core pain points of base kdb-tick.

What Is the Scalable Ingestion Blueprint?

The scalable ingestion blueprint starts from the same foundation as the reference architecture, and if you line up the processes the resemblance is clear:

The blueprint doesn’t fork from Tick-X, and Tick-X isn’t necessarily a simplification of the blueprint. They’re two implementations of the same separated-write-path pattern, built for different purposes. Tick-X is the clean reference, whereas the blueprint is what happens when you point that pattern at a firehose and ask it to keep up. That difference in purpose is exactly where the design choices split.

How Does the Blueprint Shard Ingestion?

The headline implementation here is horizontal sharding. Instead of one tickerplant chain, the blueprint runs two, and routes symbols between them by first letter:

shard:{$[first[string x]in .Q.A til 13;0;1]}

Each shard exists as a full TP/RDB/WDB/IDB chain of its own. The gateway routes a query for AAPL to shard 0 and TSLA to shard 1, and fans out across both when a query doesn’t name a symbol it can route on.

The blueprint ships a “basic” mode alongside the sharded one, and it exists specifically to highlight the bottleneck before solving it. Basic mode is a single tickerplant feeding a single RDB. Push enough volume through it and the one tickerplant can’t append, log, and fan out fast enough, while the one RDB holds the entire day and eventually runs out of memory. Sharded mode cuts both in half by construction: half the symbols, half the ingest work, half the per-process footprint. This is the simplest case, meant to show the benefits of sharding. As volumes continue to increase or different exchanges are used, more complex sharding methods can be implemented, but the foundational architecture remains the same.

How Does the Blueprint Keep Its RDB and IDB Tiers in Sync?

Recall that Tick-X keeps its real-time and intraday tiers disjoint by broadcasting a watermark through the tickerplant. The blueprint doesn’t broadcast a watermark at all. Its write-down database (WDB) signals the query RDB directly, and it gates the signal on sequence:

if[any flushed;
  if[.wdb.sigIDB[]; .wdb.sigRDB[cutoff]]];

In order: the WDB flushed some rows to disk, it signals the IDB to reload, and only if the IDB confirms it has reloaded does the WDB then tell the RDB to purge those same rows. The rows are guaranteed to be queryable in the IDB before they’re dropped from the RDB. There’s no window where they’re gone from one tier and not yet visible in the other. Tick-X guarantees no-gap through ordering on a shared connection, whereas the blueprint guarantees it through an explicit two-step handshake with a confirmation in the middle.

Why the different call? The blueprint’s handshake is simpler to follow and doesn’t need the tickerplant to carry control messages, which matters more when you’re already running two tickerplants and reasoning about ordering across both. Tick-X’s watermark also enforces the replay floor that makes restarts idempotent, making the implementation a bit more complex. The reference architecture optimizes a system you can prove correct and restart safely; the blueprint refines a data path you can scale out and observe throughput.

There’s one other difference that is embedded in the intraday cutoff. Tick-X determines it from .z.p, whereas the blueprint computes it against the data:

mx:max raze {value[x]`time} each dt;
cutoff:mx-.wdb.intv*0D00:01;

This takes the latest event time actually sitting in memory and subtracts the window from that. For a live feed, the two approaches land in nearly the same place. But when replaying a historical NYSE TAQ session at 100× speed, they don’t land anywhere near each other, and the event-time version is the only one that makes sense. The flush cadence has to follow the data’s clock, not the replay machine’s.

Watching It Run

The final layer bundled in the blueprint is the thing that turns a demo into something you can operate: instrumentation. Every feed publishes a batch_sent metric immediately before it publishes the matching data batch, both on the same connection. The RDB, on the receiving end, stamps a batch_arrived when the data lands and pairs it back to the announcement by batch_id. Because both messages travel along the same ordered connection, the announcement always arrives first, and the pairing is exact. That gives you true end-to-end transit lag, per table, in microseconds, queryable through the gateway.

The same instrumentation layer carries a soft memory cap when running in basic mode. Start an RDB with -maxmb and its upd handler checks the workspace size before every insert:

if[(.met.maxBytes>0) and (t in `bbo`trade) and .met.maxBytes < first system"w";
  ...
  :()];

At the cap, it drops the incoming batch and stays alive and queryable, rather than growing continuously until the OS kills it. The logging statement printed when it starts dropping directly describes the fix: either lower the volume, or switch to sharded mode. This is essentially the bottleneck telling you what it was built to demonstrate.

When Should You Use Tick-X vs. the Scalable Ingestion Blueprint?

If you’re standing up a tick stack and you want the query path separated from the write-down path without assembling more than you need, the reference architecture is the one to build from. It’s a smaller footprint, recovers cleanly from a crash via the replay floor, and every design decision in it is there to be understood rather than to survive a specific load.

If your problem is volume – more symbols, higher rates, a longer trading day than one tickerplant can absorb – the blueprint illustrates the path. Shard the ingest, split out the RDB, and instrument the path so you can see where it bends. It’s more resource intensive to run as it has more moving parts, but that’s the trade you make when the single tickerplant version has stopped keeping up.

What’s Next

The release of these assets is the first step. Each currently covers a single-user setup running the core ingestion path. Future iterations will build on this foundation, adding more functionality as they progress.

Immediately on the horizon is a third level for our Tick-X architecture – from base tick, to our Tick-X configuration, finally into a fully scalable version of this updated architecture. The current architecture caters only to a single user. The next step is to refactor the gateway and databases to integrate deferred synchronous communication that will allow for multiple concurrent users, as well as query processors that can scale up or down depending on the amount of traffic.

As for our blueprint, we set out to build a working system that addresses the friction points hit as data ingestion volumes continually increase. Now that this has been established across basic and sharded system configurations, the next phase is to integrate the KDB-X Database Service. This is a more out-of-the-box, self-contained cloud deployment for addressing the same problem areas that our sharded infrastructure fixes.

These are the first release in a series on scaling kdb+ tick ingestion. Future updates will cover the iterations described above. Contact devrel@kx.com with feedback, ideas, or suggestions. We would love to hear from you.