Patterns in data tell a story. They correlate to real-life events, changing sentiments, and unexpected scenarios. Let’s take, for example, something we see quite often — an Elon Musk tweet:
This tweet, in particular, had an immediate and significant impact on the TSLA stock price, spiking it in seconds. But this is just one example: events in the market happen continually.
Monitoring anomalies and patterns of interest in data is difficult. Doing so with live data adds layers of complexity. In this article, we will explore two variations using KDB.AI temporal similarity search.
Temporal similarity search (TSS)
Temporal similarity search (TSS) enables the identification of patterns, trends, and anomalies in time-series data. By doing so, companies can take advantage of accelerated similarity search across massive datasets.
TSS consists of two search methods: Transformed and non-transformed similarity search.
- Transformed TSS enables highly efficient vector search across large-scale time series datasets (historical or reference data)
- Non-transformed TSS focuses on near real-time similarity search for rapidly changing time series data (live data)
Together, they enable traders to search and analyze time series data quickly and at scale.
In this demonstration, we will use non-transformed TSS, which is more suited to real-time pattern matching on stock price and volume. Non-transformed TSS can scan data while ingested without attaching to a search index. This is key in real-time scenarios, where data must be ingested and searched quickly.
In non-transformed TSS, similarity search is computed based on the shape of the data, not the underlying value; this enables it to detect similarities, even when the data differs in specific numerical values.
We will use one week of TSLA price and volume data in our demonstration. Price data will be bucketed by the second and trade volume by the minute. The first four days will be used as historical data, and the fifth to mimic a live stream.
Here are the first four loaded into KDB.AI:
As you can see, no vector embeddings are present in the table. And because non-transformed TSS searches directly on the time-series data, we won’t need to adhere to a fixed vector length.
Search Method: Anomaly Detection in KDB.AI
Now, we will detect anomalies in live incoming data (simulated). TSS achieves this by searching incoming patterns against the historical dataset. If a new incoming pattern is not similar to any historical patterns, it will be flagged.
Let’s take a look at the results of the TSS search on incoming ‘live’ data
The top graph represents the live pricing feed; the bottom graph identifies similar patterns (nearest neighbors) against historical data (the previous four days).
But then…Elon…
As you can see, we experienced a sudden spike in the TSLA price (about 15 seconds after Elon’s tweet), turning the graph red to identify an anomaly.
As we continue to monitor the stock price, we see another anomaly, this time due to the market halting TSLA trading due to high volatility. Since this phenomenon was not previously recorded, it is flagged.
Search method: Identify predetermined patterns in KDB.AI
In another example, we will identify known patterns in inbound data to gain intelligence on what might happen next. Using live TSLA data streamed into KDB.AI; we are able to compare it against predetermined patterns.
The predetermined patterns, alongside their nearest neighbors, are then identified in the similarity search. Once detected, we can see historically what happens next (highlighted in yellow)
Non-transformed TSS offers powerful capabilities for analyzing time-series data, particularly in fast-paced environments like the stock market.
Through the demonstration presented in this blog, we’ve seen how TSS can be applied in anomaly detection and pattern identification. These methods allow for real-time analysis of incoming data, enabling quick identification of unusual events or predefined patterns that could signal important market movements.
- Learn the Basics: Learn more about TSS and get hands-on with sample notebooks on GitHub or directly via Google Colab
- Join the Slack Channel: A direct line of access to the KX team for questions and to share learnings with the broader community
