Satellite Earth Observation Data with kdb+

Kx for Earth Observation and Astronomy Big Data challenges

11 Oct 2017 | , , , ,
Share on:

By Robert Hill

The aerospace industry is increasingly becoming aligned with companies that can take advantage of high-value geospatial and global Earth observation data. In the current era of Big Data and IoT analytics, businesses in industries ranging from energy to civil engineering to facilities management that can commercially exploit non-traditional sources of data, such as data from satellites, can gain a critical edge. Kx technology has already been adopted in a number of such initiatives.

Airbus Defence and Space

In March, Kx began a collaboration with Airbus Defence and Space to develop an innovative approach for large-scale processing of geospatial data using Kx technology. Under the agreement, Airbus is contributing historic and future satellite imagery which Kx technology will process, interpret and manage.

“There are many valuable applications of satellite imagery across a range of industries, many of which are time sensitive and require powerful analytic processing. By combining our data and Kx technology we expect to be able to provide new and existing customers with unique and valuable insights,” said Dr. Anthony Denniss, Head of Solutions & Imagery Portfolio at Airbus Defence and Space, commented at the time of the March announcement.

NASA Frontier Development Lab and Observational Astronomy

The technological advancements that have made new astrophysical research projects like the Large Synoptic Survey Telescope (LSST) in northern Chile and the Square Kilometer Array (SKA) in South Africa possible are causing a paradigm shift due to the scale of data they are able to collect. Scientists are concerned about their ability to make sense of this newly available data, purely due to its sheer size.

In June Kx joined with Lockheed Martin, IBM and other technology leaders to support space research at the NASA Frontier Development Lab. This year the lab’s teams tackled planetary defense, space resources and space weather. Kx supported the Solar Team, which explored whether machine learning and sophisticated data processing with kdb+ could be useful in predicting and defending our planet from asteroids and solar weather. A Kx data scientist working at the lab was able to load and analyze an extremely large dataset from the USGS Geomagnetism Program and run an analysis in nanoseconds to start getting real insights into the trends and patterns behind solar storms.

The global market for Earth observation data and value-added services is expected to grow to $43 billion in the next ten years according to Northern Sky Research. The need for innovative ways to capture, analyze, store and disseminate data as a meaningful information source will rapidly increase.

In the field of astronomy, Kx believes that due to the amount of data collected for astronomical research, including observational astronomy data, across the entire electromagnetic spectrum, coupled with other forms of data, there is a compelling need for Kx technology because of its time series nature, and its powerful real-time and historical based analytics.


Kx Insights: Machine learning subject matter experts in semiconductor manufacturing

9 Jul 2018 | , ,

Subject matter experts are needed for ML projects since generalist data scientists cannot be expected to be fully conversant with the context, details, and specifics of problems across all industries. The challenges are often domain-specific and require considerable industry background to fully contextualize and address. For that reason, successful projects are typically those that adopt a teamwork approach bringing together the strengths of data scientists and subject matter experts. Where data scientists bring generic analytics and coding capabilities, Subject matter experts provide specialized insights in three crucial areas: identifying the right problem, using the right data, and getting the right answers.

Transitive Comparison

Kdb+ Transitive Comparisons

6 Jun 2018 | , ,

By Hugh Hyndman, Director, Industrial IoT Solutions. A direct comparison of the performance of kdb+ against InfluxData and, by transitivity, against Cassandra, ElasticSearch, MongoDB, and OpenTSDB