Kx powers Data Mart for Ontario’s Smart Meter Data

20 Mar 2017 | , , , , ,
Share on:

By Przemek Tomczak


For over ten years Ontario, Canada has been a leader in the reduction of greenhouse gas emissions by closing coal-fired plants, diversifying its energy supply mix, promoting conservation, implementing time-of-use rates for electricity and by installing smart meters across the province.

The data from smart meters is processed by a government agency called the Independent Electricity System Operator (IESO) that manages Ontario’s bulk-power system. The IESO’s responsibilities include ensuring that there is enough power to meet the province’s energy needs in real time while also planning and securing energy for the future. The Smart Metering Entity within the IESO, operates the provincial smart meter data processing service (the MDM/R) for 67 distribution system operators and over 4 million customers.

The volume of data generated by the province’s smart meters exceeds 250 billion meter reads, and grows by 100 million reads per day. The administrator of this data is required to process and provide billing quality data to the utilities in the province in a timely manner and has set up a meter data management system (MDM/R) to do so.

However, the existing transaction and relational database system within the MDM/R was not originally designed to handle the increasing requirements of the fast growing data set and its users. That’s where Kx came in with its high-performance database system kdb+, and the suite of powerful data access and analytics tools built on top of it. The IESO chose Kx after evaluating a number of other technologies, and performing over 25 tests on five years of simulated data from 5 million smart meters. They found that kdb+ delivered very high levels of performance at the lowest total cost of ownership.

The IESO implemented a Data Mart using Kx technology to support the current volumes of data access requests and anticipated growth with 24x7x365 availability, as well as current and future demands for advanced analytics in Ontario’s energy sector. The Data Mart delivers low-latency simultaneous data ingestion, transformation, data retrieval requests, and is synchronized in near real-time with the MDM/R system. For example, with kdb+ the IESO is able to aggregate an entire year of data for all meters in a few minutes, which supports analysis of load profiles, consumption behavior and different rate models.

Ontario is taking the lead again, in adopting advanced tools for retrieval and analysis of the province’s smart meter data using Kx technology. To read more about the IESO use case, click here.


Przemek Tomczak is Senior Vice-President Internet of Things and Utilities at Kx Systems. For over twenty years, Kx has been providing the world’s fastest database technology and business intelligence solutions for high velocity and large data sets. Previously, Przemek held senior roles at the Independent Electricity System Operator in Ontario, Canada and top-tier consulting firms and systems integrators. Przemek has a CPA and a background in business, technology and risk management.

© 2017 Kx Systems
Kx® and kdb+ are registered trademarks of Kx Systems, Inc., a subsidiary of First Derivatives plc.


kdb+ for sensor analytics

Kx Insights: Benefits of Utility Predictive Maintenance Analytics

17 Oct 2017 | , , ,

Utilities are going through significant modernization, with the adoption of smart grid technologies such as advanced metering, advanced distribution management, outage management, customer engagement and analytics. This modernization is creating a wealth of diverse data about assets, operations, and customers. At the same time the job of utilities is becoming more challenging with pressure to reduce costs, competition from new forms of technologies and energy providers as well as the need to integrate renewable energy resources. These new challenges, competition and pressures are leading to innovation and transformation in the utilities industry.

Satellite Earth Observation Data with kdb+

Kx for Earth Observation and Astronomy Big Data challenges

11 Oct 2017 | , , , ,

The aerospace industry is increasingly becoming aligned with companies that can take advantage of high-value geospatial and global Earth observation data. In the current era of Big Data and IoT analytics, businesses in industries ranging from energy to civil engineering to facilities management that can commercially exploit non-traditional sources of data, such as data from satellites, can gain a critical edge. Kx technology has already been adopted in a number of such initiatives.

Kdb+ Mastermind Challenge

Kdb+ Mastermind Challenge

4 Oct 2017 | , , , , ,

The Kx Community NYC Meetup recently competed in a Mastermind coding competition moderated by expert kdb+ programmer Nick Psaris, author of the book “Q Tips: Fast, Scalable and Maintainable Kdb+.” The top four solutions were notable for the completely different approaches they took. Mastermind is a popular code-breaking game introduced in 1975 that pits a “codemaker” against a “codebreaker.” The “code,” represented by four pegs chosen from a universe of 6 colors, is picked by the “codemaker.” The “codebreaker” continually makes 4-peg guesses and is told by the “codemaker” how many are the correct color in the correct position, and how many are the correct color in the wrong position.