Stop Stuffing Your Time Series Data into Your Data Warehouse Like it’s Your Sock Drawer

21 April 2023 | 6 minutes

By Michael Benjamin

Stuffing time series data into a data warehouse is a bit like stuffing socks into a messy sock draw. Once they’re in, you won’t find the ones you need again in a timely manner. Bring order to your data with the Data Timehouse. Read on to find out more as we digest the key takeaways from the Gartner Data & Analytics Summit, Orlando.

The enterprise value of data and analytics is critical to modern organizations. Leading companies in all industries not only maintain continuous, situational awareness of their activities but model their business digitally. Though it may seem strange, digital leaders run digital twins of their business, constantly re-evaluating their world given changing assumptions.

In uncertain times, we may not know what comes next, but the machine that’s modeled everything has a pretty good idea of what to do when unexpected events occur. When this insight is coupled with our own knowledge and intuition — well, we’re ready to leap into these new realities like a Marvel movie superhero.

So, what’s holding modern companies back from this high-value promise? We’ll examine four pitfalls effecting mindshare (and potentially market share) according to analysts and attendees at the Gartner Data & Analytics Summit in Orlando:

  • An enterprise-wide technology approach that traps analytics within an ivory tower. According to Gartner, the true impact of most organizational data programs is commonly sequestered to a central analytics team. Perhaps it’s unsurprising then that two-thirds of data and analytics leaders still struggle to deliver measurable ROI (Gartner Chief Data Officer Agenda Survey for 2023). At the conference, Rita Sallam, a Distinguished VP Analyst and Gartner Fellow in the Data and Analytics team, and Kurt Schlegel, a Research Vice President at Gartner, suggested a franchise model for growing analytics across an organization. No matter the method though, scale and skill are necessary to ensure all corners of a business are able to define, calculate and use data in such a way that will drive impact. And if done correctly, the results speak for themselves. According to KX research (Speed to Business Value), companies who had unlocked more efficient processing and managing of data realized firm-level gains that resulted in an uplift of $7.4 billion in gross value added, with a further potential increase of $3.6 billion.
  • More rapid AI & ML advances and a deluge of new AI-driven products and services. The power and potential for AI and ML is growing, with reports suggesting that generative AI technology is already beginning to move deeper into more specialized business functions (Wall Street Journal). But with the rate of change, are companies realistically equipped to operationalize it effectively? At Gartner, it was suggested that two or three analytical workloads will soon need to run simultaneously against a current production model for companies to gain the ideal accuracy and optimization necessary to, as Daryl Plummer (Gartner VP, Distinguished Analyst & Gartner Fellow) put in his keynote, transition from ‘catch up’ and ‘keep up’ to a position where they can truly lead and innovate. In its own 2023 trend report, Gartner identified adaptive AI as a key capability, saying that given the engineering complexity and the faster time to market demanded today, it is critical to develop less rigid AI engineering models or build AI models that can self-adapt.
  • Technology investments that drain productivity and purpose from the workforce. More than ever, employees are demanding better work/life balance post-Covid — one indicator of this is the increasing popularity of the four-day workweek. According to a ZDNet report, one UK company who successfully transitioned to a shorter workweek “saw a 500% increase in applications for job vacancies, and in August 2022 the organization reported that productivity had increased by an incredible 92%. An employee survey found that 91% of staff were able to accomplish everything they need to in four days.” These potential benefits are hard to ignore, and they’re motivating IT departments to seek technologies that perform faster and are easier to set up. By reducing the time it takes to set up popular initiatives such as data science, while also minimizing the time required to process data, IT can get ahead of this trend by finding best-of-breed technology stacks that were built for high performance and rapid deployment.
  • New pressure on conventional data stores due to digital transformation. In his Tuesday keynote, Gartner analyst Daryl Plummer talked about new pressures on conventional data stores due to digital transformation: “We must have data that allows us to do digital things effectively. That includes being able to evaluate things over sequences of moments. Time series data is now critical to what we do, and if digital demands time, we have to take our data warehouses and not just force fit time series data into them.” The processing and analysis of time series and machine data generated by digital transformation is a critical business requirement — such as market data in financial services, network data in the telecoms industry, patient data in healthcare, and sensor data from factory equipment and smart energy meters. Many businesses already leverage storage provided by a data lake or a data warehouse, but this is only a part of the story; a data timehouse infuses storage fuel with an analytics engine to create gold, and of course a more orderly sock drawer.

Looking Ahead

Leaders who hope to not only do data a lot but, in the words of Mr. Plummer, “do data well” must anticipate and adapt to trending technology and business demands; unfortunately, too many today are hampered by out-of-touch practices and non-performant technology stacks that act like tar pits for digital transformation.

To meet this moment like a true Marvel movie superhero, business leaders must leverage partners that help them harness the true power of their data and analytics strategies, respond to emerging trends and optimize decision-making.

At KX, our highly performant time series data and real-time analytics solution — empowered by the industry’s first Data Timehouse, a new data and AI management platform — accelerates data and analytics delivery for less cost, faster performance, and increased efficiency.

Sign up for a Free Trial today to learn more, or visit us at Gartner London in May!

Demo kdb, the fastest time-series data analytics engine in the cloud

    For information on how we collect and use your data, please see our privacy notice. By clicking “Download Now” you understand and accept the terms of the License Agreement and the Acceptable Use Policy.