Democratizing fast access to Big Data

26 Apr 2016 | , , ,
Share on:

For many years business users have had both limited and slow access to the massive data sets their organizations own. Their data has been locked up in databases and data warehouses (now called data lakes).  Typical users have never seen their actual data. Rather they have had to wait for limited summaries provided by either BI tools or expert coded data pipelines using multiple technologies and tools. These complex ETL data pipelines are written in complex queries and programs often using multiple tools.  In most cases the cost of accessing the whole dataset is such that analyses must be run as batch jobs.

Meanwhile kdb+ empowered analysts have been able to concisely express their entire tasks in a few statements of q, enabling them to obtain results in seconds or minutes that previously took hours or days.  Q is a powerful language used extensively in finance, for IoT and for other time-series applications, which is executed by the underlying k data language. Q is a simple functional data language which both simplifies and extends SQL.  Both the queries and the analytics are in the same language and execute inside the database.

It is natural to ask “Can’t we bring the power of kdb+ to users who are non-expert programmers?”  The answer is “Yes, of course we can.”  This is now being done through visual and textual DSLs which are easier to learn and designed for the data analyst and end user.  They can have their information expressed in visual analytics such as heat maps or in operational dashboard charts.

These new UI’s are democratizing the use of massive data sets, giving all business users and analysts access to their complete datasets, as well as giving them a much more powerful means for visualizing and analyzing their data.

Dave Thomas is chief scientist at Kx Labs and was cofounder of Bedarra Research Labs. Prior to developing complex commercial applications in kdb+ Dave was known for his contributions to Object Technology including IBM VisualAge and Eclipse IDEs, Smalltalk and Java virtual machines. He is a thought leader in large-scale software engineering and a founding director of the Agile Alliance.

© 2018 Kx Systems
Kx® and kdb+ are registered trademarks of Kx Systems, Inc., a subsidiary of First Derivatives plc.


kdb+ for sensor analytics

Kx Insights: Benefits of Utility Predictive Maintenance Analytics

17 Oct 2017 | , , , ,

Utilities are going through significant modernization, with the adoption of smart grid technologies such as advanced metering, advanced distribution management, outage management, customer engagement and analytics. This modernization is creating a wealth of diverse data about assets, operations, and customers. At the same time the job of utilities is becoming more challenging with pressure to reduce costs, competition from new forms of technologies and energy providers as well as the need to integrate renewable energy resources. These new challenges, competition and pressures are leading to innovation and transformation in the utilities industry.

Kdb+ Mastermind Challenge

Kdb+ Mastermind Challenge

4 Oct 2017 | , , , , ,

The Kx Community NYC Meetup recently competed in a Mastermind coding competition moderated by expert kdb+ programmer Nick Psaris, author of the book “Q Tips: Fast, Scalable and Maintainable Kdb+.” The top four solutions were notable for the completely different approaches they took. Mastermind is a popular code-breaking game introduced in 1975 that pits a “codemaker” against a “codebreaker.” The “code,” represented by four pegs chosen from a universe of 6 colors, is picked by the “codemaker.” The “codebreaker” continually makes 4-peg guesses and is told by the “codemaker” how many are the correct color in the correct position, and how many are the correct color in the wrong position.

Kdb+ for Factom Blockchain application

Kdb+ and Factom: Securing data on the blockchain with qfactom

13 Sep 2017 | , , , ,

Blockchain applications are proliferating in the financial services industry and Kx technology is increasingly being used in these systems to provide high-speed data analytics, like by the Bitcoin Mercantile Exchange (BitMEX), a cryptocurrency derivatives exchange and by financial technology firm Cobalt DL. qfactom, is a kdb+ wrapper library designed to interact with the Factom blockchain via its client and wallet applications. The library provides convenient functions to create, publish data to, and read data from, Factom chains for the purpose of creating and validating secure audit trails.