Kdb+ and Observational Astronomy Analytics

31 Jan 2017 | ,
Share on:

As Kx expands into industries beyond financial services, data-driven fields of study, like astronomy, are emerging as use cases for the kdb+ database, with its built-in programming language q.

Kdb+ developers Andrew Magowan and James Neill recently wrote a whitepaper that explores the opportunities for kdb+ in observational astronomy, inspired in part by the construction of new infrastructure that has the potential to record volumes of data that have never before been seen in the field.

The technological advancements that have made new projects like the Large Synoptic Survey Telescope (LSST) in northern Chile and the Square Kilometer Array (SKA) in South Africa possible are causing something of a paradigm shift, and – according to experts – a bit of a headache. Scientists are concerned about their ability to make sense of this newly available data, purely due to its sheer size.

Andrew and James write:

“There is no single choice of programming language in astronomy, but C is used for many astronomical applications. C is one of the most popular and commonly used programming languages in the world with a wide range of uses varying from powering operating systems to building application software. Kdb+ has the ability to extend its functionality through dynamically loaded C/C++ modules, so we have the ability both to make use of existing utilities, and to create our own.

We believe that due to the amount of data collected, its time series nature, and the potential need for both real-time and historical based analysis, kdb+ would be very well suited to the data collected in the astronomy industry, and would be an ideal fit for many future astronomy projects.”

To read the rest of their whitepaper, with code examples, please follow the link to the left.

Andrew Magowan and Jame Neill are expert kdb+ programmers and Kx consultants who have worked on large-scale, complex kdb+/q applications at major investment banks around the world. James Neill wrote a previous paper in this series called “An Introduction to Neural Networks with kdb+” available here.

SUGGESTED ARTICLES

Satellite Earth Observation Data with kdb+

Kx for Earth Observation and Astronomy Big Data challenges

11 Oct 2017 | , ,

The aerospace industry is increasingly becoming aligned with companies that can take advantage of high-value geospatial and global Earth observation data. In the current era of Big Data and IoT analytics, businesses in industries ranging from energy to civil engineering to facilities management that can commercially exploit non-traditional sources of data, such as data from satellites, can gain a critical edge. Kx technology has already been adopted in a number of such initiatives.