Kx for Earth Observation and Astronomy Big Data challenges

11 Oct 2017 | , , , ,
Share on:

By Robert Hill

The aerospace industry is increasingly becoming aligned with companies that can take advantage of high-value geospatial and global Earth observation data. In the current era of Big Data and IoT analytics, businesses in industries ranging from energy to civil engineering to facilities management that can commercially exploit non-traditional sources of data, such as data from satellites, can gain a critical edge. Kx technology has already been adopted in a number of such initiatives.

Airbus Defence and Space

In March, Kx began a collaboration with Airbus Defence and Space to develop an innovative approach for large-scale processing of geospatial data using Kx technology. Under the agreement, Airbus is contributing historic and future satellite imagery which Kx technology will process, interpret and manage.

“There are many valuable applications of satellite imagery across a range of industries, many of which are time sensitive and require powerful analytic processing. By combining our data and Kx technology we expect to be able to provide new and existing customers with unique and valuable insights,” said Dr. Anthony Denniss, Head of Solutions & Imagery Portfolio at Airbus Defence and Space, commented at the time of the March announcement.

NASA Frontier Development Lab and Observational Astronomy

The technological advancements that have made new astrophysical research projects like the Large Synoptic Survey Telescope (LSST) in northern Chile and the Square Kilometer Array (SKA) in South Africa possible are causing a paradigm shift due to the scale of data they are able to collect. Scientists are concerned about their ability to make sense of this newly available data, purely due to its sheer size.

In June Kx joined with Lockheed Martin, IBM and other technology leaders to support space research at the NASA Frontier Development Lab. This year the lab’s teams tackled planetary defense, space resources and space weather. Kx supported the Solar Team, which explored whether machine learning and sophisticated data processing with kdb+ could be useful in predicting and defending our planet from asteroids and solar weather. A Kx data scientist working at the lab was able to load and analyze an extremely large dataset from the USGS Geomagnetism Program and run an analysis in nanoseconds to start getting real insights into the trends and patterns behind solar storms.

The global market for Earth observation data and value-added services is expected to grow to $43 billion in the next ten years according to Northern Sky Research. The need for innovative ways to capture, analyze, store and disseminate data as a meaningful information source will rapidly increase.

In the field of astronomy, Kx believes that due to the amount of data collected for astronomical research, including observational astronomy data, across the entire electromagnetic spectrum, coupled with other forms of data, there is a compelling need for Kx technology because of its time series nature, and its powerful real-time and historical based analytics.


A comparison of Python and q for data problem solving

8 May 2019 | ,

This article takes a simple, real-life problem and analyzes different solutions in Python and q. The problem leads us to discover nice areas of both programming languages, including vector operations, Einstein summation, adverbs and functional form of select statements. Each solution has lessons that deepen our IT knowledge, especially when we consider performance.