Key Takeaways
- Dynamic Time Warping (DTW) flexes the time axis to identify similarities, even when events occur at different times or speeds.
- Powerful sequences with tunable parameters help detect anomalies, align signals, and uncover hidden patterns.
- DTW is widely applicable across industries including Finance, Manufacturing, Aerospace and Defense.
If you’ve ever tried to compare two pieces of time series data, you’ll know it’s not always as simple as checking if the lines overlap. Markets shift, sensors drift, and heart rates spike at different times. However, with Dynamic time warping (DTW), analysts can now identify similar shapes in sequences, even when they’re stretched or misaligned.
In this article, we’ll get hands-on with DTW using the KDB-X AI libs module. We’ll analyze NYSE stock data for our example and also discuss how it can be applied in other industries.
Why DTW?
Traditional similarity metrics, such as Euclidean distance, assume that sequences line up neatly in time, but that’s rarely the case:
- A stock rally for one ticker might happen hours before another
- A turbine sensor may vibrate just before a fault, while a “normal” unit indicates the same pattern with a delay
- An ECG trace could show the same heartbeat pattern at slightly different speeds
DTW flexes the time axis to find alignments, measuring similarity based on shape rather than rigid time steps.
Getting started
The KDB-X AI libs module provides DTW with simple functions, such as .ai.dtw.search, .ai.dtw.searchRange, and .ai.dtw.filterSearch. In this tutorial, we will run a small subset, but you can also explore the entire notebook via our GitHub tutorial.
Load the AI libs module:
.ai:use`kx.aiLet’s explore our dataset:
first trademsgType | 220
sequenceNo| 58765
time | 0D07:00:00.105862814
sym | `TMF
tradeId | 24476
price | 4.74
volume | 4000We can see a single trade message for the symbol TMF executed at $4.74 for 4,000 shares at 7:00:00.
Next, we will create a query vector that will define the pattern we wish to match within our NYSE dataset, specifically in the price column.
vector:10*abs sums neg[0.5]+25?1fIn the above:
- 25?1f: Generates 25 random float numbers between 0 and 1
- neg[0.5]: Returns -0.5
- sums: Computes the cumulative sum
- abs: Tables the absolute value
- Finally, we multiply by 10*
The result is a random pattern that fluctuates around zero, mimicking price fluctuations we may want to detect in our time-series data.
DTW search
We run our first query using .ai.dtw.search to perform a DTW search on the price column of the trade table.
The .ai.dtw.search function compares the query pattern against the time series data, identifying the top five most similar subsequences based on DTW distance. The fourth parameter, 0.1, represents the window or the ratio of the query size allowed during warping. The larger the window, the more flexible the search becomes, but also the more time-consuming.
`distances`indexes!.ai.dtw.search[;vector;5;0.1;::] trade`pricedistances| 1.519624 1.766677 1.773423 1.78443 1.794284
indexes | 3802228 24797 2314123 2746995 4067262Let’s set the window to a higher value of 0.8, to get greater time warping into our search.
`distances`indexes!.ai.dtw.search[;vector;5;0.8;::] trade`pricedistances| 1.265552 1.354349 1.446201 1.468039 1.515346
indexes | 2303698 632384 632383 2303699 3388567 Tuning parameters
The DTW search functions .ai.dtw.searchRange and ai.dtw.filterSearch allows for some additional fine-tuning:
- k: The maximum number of best matches to return (ai.dtw.filterSearch, ai.dtw.Search)
- cutoff: Threshold similarity, reducing weak matches
- returnMatches: Lets you decide whether to bring back the aligned subsequences for inspection.
The DTW searchRange query provides additional flexibility in setting a maximum distance. In this example, we set it to 1.6, ensuring the search will only return matches within those parameters.
`distances`indexes!.ai.dtw.searchRange[;vector;0.2;1.6;::] trade`pricedistances| 1.265552 1.468039 1.507729 1.519624 1.520065 1.592965
indexes | 2303698 2303699 632384 3802228 3388567 2620562The DTW filteredSearch query sets a maximum results cap. Here we limit the results to 3, and ensure returned results are <1.6.
`distances`indexes!.ai.dtw.filterSearch[;vector;3;0.2;1.6;::] trade`pricedistances| 1.265552 1.468039 1.507729
indexes | 2303698 2303699 632384Beyond Wall Street
While this tutorial relies on NYSE data, DTW is a universal pattern matcher that can be applied across various industries.
For example:
- Manufacturing: Detect anomalies in machine vibration before failure
- Aerospace & Defense: Identify repeat signal patterns in radar/sonar streams or match flight telemetry against known mission profiles
- Healthcare: Compare patient ECG traces or gait cycles over time
- IoT: Spot recurring usage patterns in smart meter data
- Speech recognition: DTW has long been used to align spoken words against templates.
Dynamic time warping fills a gap that traditional similarity measures leave behind. By stretching and compressing the time axis, it surfaces meaningful patterns hidden in noisy, misaligned data.
If you enjoyed this blog and would like to explore other examples, you can visit our GitHub repository. You can also begin your journey with KDB-X by signing up for the KDB-X Community Edition Public Preview.
