Satellite Earth Observation Data with kdb+

Kx for Earth Observation and Astronomy Big Data challenges

11 Oct 2017 | , , , ,
Share on:

By Robert Hill

The aerospace industry is increasingly becoming aligned with companies that can take advantage of high-value geospatial and global Earth observation data. In the current era of Big Data and IoT analytics, businesses in industries ranging from energy to civil engineering to facilities management that can commercially exploit non-traditional sources of data, such as data from satellites, can gain a critical edge. Kx technology has already been adopted in a number of such initiatives.

Airbus Defence and Space

In March, Kx began a collaboration with Airbus Defence and Space to develop an innovative approach for large-scale processing of geospatial data using Kx technology. Under the agreement, Airbus is contributing historic and future satellite imagery which Kx technology will process, interpret and manage.

“There are many valuable applications of satellite imagery across a range of industries, many of which are time sensitive and require powerful analytic processing. By combining our data and Kx technology we expect to be able to provide new and existing customers with unique and valuable insights,” said Dr. Anthony Denniss, Head of Solutions & Imagery Portfolio at Airbus Defence and Space, commented at the time of the March announcement.

NASA Frontier Development Lab and Observational Astronomy

The technological advancements that have made new astrophysical research projects like the Large Synoptic Survey Telescope (LSST) in northern Chile and the Square Kilometer Array (SKA) in South Africa possible are causing a paradigm shift due to the scale of data they are able to collect. Scientists are concerned about their ability to make sense of this newly available data, purely due to its sheer size.

In June Kx joined with Lockheed Martin, IBM and other technology leaders to support space research at the NASA Frontier Development Lab. This year the lab’s teams tackled planetary defense, space resources and space weather. Kx supported the Solar Team, which explored whether machine learning and sophisticated data processing with kdb+ could be useful in predicting and defending our planet from asteroids and solar weather. A Kx data scientist working at the lab was able to load and analyze an extremely large dataset from the USGS Geomagnetism Program and run an analysis in nanoseconds to start getting real insights into the trends and patterns behind solar storms.

The global market for Earth observation data and value-added services is expected to grow to $43 billion in the next ten years according to Northern Sky Research. The need for innovative ways to capture, analyze, store and disseminate data as a meaningful information source will rapidly increase.

In the field of astronomy, Kx believes that due to the amount of data collected for astronomical research, including observational astronomy data, across the entire electromagnetic spectrum, coupled with other forms of data, there is a compelling need for Kx technology because of its time series nature, and its powerful real-time and historical based analytics.

© 2018 Kx Systems
Kx® and kdb+ are registered trademarks of Kx Systems, Inc., a subsidiary of First Derivatives plc.


Head of Products, Solutions and Innovation at Kx on Product Design and the Vision for the Future

16 Mar 2018 | , , ,

As the SVP of Products, Solutions and Innovation at Kx Systems, James Corcoran is part of a new chapter in software development at Kx. Since joining Kx parent First Derivatives as a financial engineer in 2009, James has worked around the world building enterprise systems at top global investment banks before moving to the Kx product team in London. James sat down with us recently to discuss his perspective on product design and our technology strategy for the future.

Kdb+ Utilities: Essential utility for identifying performance problems

28 Feb 2018 | ,

If you are a kdb+/q developer, you will find the utilities created by Kx Managing Director and Senior Solution Architect Leslie Goldsmith to be a valuable resource. The “Kdb+ Utilities” series of blog posts gives a quick introduction to the utilities, available at Leslie Goldsmith’s GitHub. In this third part of the series we look at Leslie’s qprof, which allows a programmer to drill down into q functions or applications to inspect performance and CPU usage in a fine-grained fashion.

kdb+ utility to search codebase

Kdb+ Utilities: Q code Workspace Utilities

6 Feb 2018 | , ,

If you are a kdb+/q developer, you will find the workspace utilities created by Kx Managing Director and Senior Solution Architect Leslie Goldsmith to be a valuable resource. This is the first in a series of blog posts that give a quick introduction to several utilities available at Leslie Goldsmith’s GitHub. In this part of the series we look at an essential tool which contains routines for summarizing and searching the contents of a workspace, ws.