DATA, DATA, DATA, EVERYWHERE
The digital universe of our connected world continues to expand exponentially every year, driven by human and machine-generated data. It’s not surprising in this age of IoT, with sensors everywhere reporting thousands of measurements each second. Autonomous vehicles, market trading platforms, smart meters, machine tools and retail systems amongst many others are providing a constant stream of time-series data that we can now capture and analyze faster than ever before.
Over the last ten years we have experienced the growth in the volume and velocity of data, an expanding cloud infrastructure, the emergence of 5G and the rapid adoption of AI and machine learning and an explosion in the use of IoT applications. Real-time data has progressed from minutes/seconds to milliseconds with a requirement to take action on how behaviors, processes and systems are changing over time.
The challenges for business and organizations include:
- how to use this time-series data to make better informed decisions to drive real business benefits
- evaluating what platforms and analytic tools can help deliver this
- how to store and query the data efficiently
What do you mean by time-series?
Time-series databases are increasingly being recognized as a powerful way to manage big data. They can be used to instrument, learn and automate applications and systems, enable real-time and predictive analytics across manufacturing processes and provide operational intelligence to make improved and faster decisions based on what is occurring now, what has occurred in the past, and what is predicted to take place in the future .
As an early leader in high-performance time-series database analytics we still see a lot of misconceptions about exactly what time-series means. Many businesses still look at their big data applications purely through the lens of the architecture and miss the vital point that the unifying factor across all data is its dependency on time.
The ability to capture and factor in time when ordering events can be the key to unlocking real cost efficiencies for an organization’s big data applications. Data falls into one of three categories — historical, recent, or real-time and firms need to access each of these data types for different reasons. For example, customer reporting requires firms to get a handle on recent data from earlier in the day, while various surveillance activities demand a combination of recent and real-time insight. Market pricing-related decisions rely on a combination of real-time and historical data analytics. Manufacturing plants generate thousands to millions of data points collected by sensors, RFIDs and many more every second of every day that can be used to predict machinery and material failure.
Each piece of data exists and changes in real time, earlier today, or further in the past. However, unless they are all linked together in a way that firms can analyze, there is no way of providing a meaningful overview of the business at any specific point in time.
Some firms have struggled to introduce this due to the limitations of their technology. Some architectures simply don’t allow for time-based analysis and struggle with both recording it and querying it in a highly efficient way. It’s a common issue that can result from opting for a seemingly safe technology choice. It’s also one of the reasons why over the past 24 months that time-series databases (TSDBs) have emerged from software developer usage patterns as the fastest growing category of databases.
Trend of the last 24 months
Why not use a traditional database instead of a time-series platform?
You can and some people do. Taking this approach though often involves trying to combine multi-layered solutions using dissimilar tools, adds complexity and cost, and falls short on scale and usability.
In many cases, the different collections of data will live on three different platforms — such as a traditional database for historical data, in-memory for recent data, and event processing software for real-time data. This type of technology legacy stack can offer a wide variety of options. However, it is rare for constituent applications to span more than one of the three domains effectively – and even rarer for them to do so with sufficient performance.
We believe the answer is one unifying piece of architecture that spans all time boundaries. Many firms will recognize that all their data is, in some way, time-dependent. By rethinking and unifying the underlying technology, they can move beyond simply storing data, or even analyzing only part of it, to truly understanding all of their data and making correlations across the business.
The cost benefit is huge — and that’s crucial. We also recognise that you need approaches and techniques for preserving your investment in your software and RDBMS technology while addressing the performance, scale, and cost challenges when working with high volumes of data. For more detail on how you can implement Kx alongside existing systems and processes, their advantages and disadvantages, and case studies of these patterns in action read our Supercharging Your Legacy Systems whitepaper.
Time really does mean money, and firms that invest in embedding time into their technology in the right way stand to realize the greatest savings.
Why should I use kdb+ time-series database?
Kdb+ is a time-series database optimized for Big Data analytics. The columnar design of kdb+ means it offers greater speed and efficiency than typical relational databases and its native support for time-series operations vastly improves both the speed and performance of queries, aggregation, and analysis of structured data.
Kdb+ is also different from other popular databases because it has a built-in proprietary array language, q, allowing it to operate directly on the data in the database, removing the need to ship data to other applications for analysis. Kdb+ and q make full use of the intrinsic power of modern multi-core hardware architectures. Kdb+ supports a number of different interfaces with a very simple API, for easy connectivity to external graphical, reporting and legacy systems.
If however you currently find that you are dealing with simple queries, have scant or mainly unstructured data then there are other database solutions that would better suit your requirements. But if you think that Kx time-series database platform could help you deliver an improved solution you can learn more by downloading our software or requesting a demo to find out more.
The basis for Kx Technology is a unique integrated platform which includes a high-performance historical time-series columnar database called kdb+, an in-memory compute engine, and a real-time streaming processor all unified with an expressive query and programming language called q. Designed from the start for extreme scale, and running on industry standard servers, the kdb+ database has been proven to solve complex problems faster than any of its competitors.LEARN MORE