distributed computing with kdb+/q

Distributed Computing in kdb+

21 Dec 2016 | , , ,
Share on:

Kx financial engineer, Connor Gervin, gave a talk at a Kx Community NYC Meetup in 2015 that is worth revisiting. Connor described kdb+’s built-in multithreading and multiprocessing capabilities, which are an essential part of every serious kdb+ programmer’s’ toolkit. With these features, programmers can make the best use of multicore hardware when solving increasingly complex problems over ever-expanding datasets.

In a guided overview with real code examples, Connor outlines how the multicore and multithreading aspects of parallel execution operations; multithreading input queues; multithreaded access to the HDB; segmentation and MapReduce all work together to remove memory limits and single-thread computing constraints.

Connor demonstrates a kdb+ solution to the complex problems posed by massive datasets that shows programmers how to significantly improve vector operations, disk performance and optimize queries.

Watch Connor’s presentation on the Kx Systems’ YouTube channel (here) and while you’re there, subscribe so you never miss new videos about Kx and kdb+.

 

© 2017 Kx Systems
Kx® and kdb+ are registered trademarks of Kx Systems, Inc., a subsidiary of First Derivatives plc.

SUGGESTED ARTICLES

kdb+ for sensor analytics

Kx Insights: Benefits of Utility Predictive Maintenance Analytics

17 Oct 2017 | , , ,

Utilities are going through significant modernization, with the adoption of smart grid technologies such as advanced metering, advanced distribution management, outage management, customer engagement and analytics. This modernization is creating a wealth of diverse data about assets, operations, and customers. At the same time the job of utilities is becoming more challenging with pressure to reduce costs, competition from new forms of technologies and energy providers as well as the need to integrate renewable energy resources. These new challenges, competition and pressures are leading to innovation and transformation in the utilities industry.

Satellite Earth Observation Data with kdb+

Kx for Earth Observation and Astronomy Big Data challenges

11 Oct 2017 | , , , ,

The aerospace industry is increasingly becoming aligned with companies that can take advantage of high-value geospatial and global Earth observation data. In the current era of Big Data and IoT analytics, businesses in industries ranging from energy to civil engineering to facilities management that can commercially exploit non-traditional sources of data, such as data from satellites, can gain a critical edge. Kx technology has already been adopted in a number of such initiatives.

Kdb+ Mastermind Challenge

Kdb+ Mastermind Challenge

4 Oct 2017 | , , , , ,

The Kx Community NYC Meetup recently competed in a Mastermind coding competition moderated by expert kdb+ programmer Nick Psaris, author of the book “Q Tips: Fast, Scalable and Maintainable Kdb+.” The top four solutions were notable for the completely different approaches they took. Mastermind is a popular code-breaking game introduced in 1975 that pits a “codemaker” against a “codebreaker.” The “code,” represented by four pegs chosen from a universe of 6 colors, is picked by the “codemaker.” The “codebreaker” continually makes 4-peg guesses and is told by the “codemaker” how many are the correct color in the correct position, and how many are the correct color in the wrong position.