Satellite Earth Observation Data with kdb+

Kx for Earth Observation and Astronomy Big Data challenges

11 Oct 2017 | , , , ,
Share on:

By Robert Hill

The aerospace industry is increasingly becoming aligned with companies that can take advantage of high-value geospatial and global Earth observation data. In the current era of Big Data and IoT analytics, businesses in industries ranging from energy to civil engineering to facilities management that can commercially exploit non-traditional sources of data, such as data from satellites, can gain a critical edge. Kx technology has already been adopted in a number of such initiatives.

Airbus Defence and Space

In March, Kx began a collaboration with Airbus Defence and Space to develop an innovative approach for large-scale processing of geospatial data using Kx technology. Under the agreement, Airbus is contributing historic and future satellite imagery which Kx technology will process, interpret and manage.

“There are many valuable applications of satellite imagery across a range of industries, many of which are time sensitive and require powerful analytic processing. By combining our data and Kx technology we expect to be able to provide new and existing customers with unique and valuable insights,” said Dr. Anthony Denniss, Head of Solutions & Imagery Portfolio at Airbus Defence and Space, commented at the time of the March announcement.

NASA Frontier Development Lab and Observational Astronomy

The technological advancements that have made new astrophysical research projects like the Large Synoptic Survey Telescope (LSST) in northern Chile and the Square Kilometer Array (SKA) in South Africa possible are causing a paradigm shift due to the scale of data they are able to collect. Scientists are concerned about their ability to make sense of this newly available data, purely due to its sheer size.

In June Kx joined with Lockheed Martin, IBM and other technology leaders to support space research at the NASA Frontier Development Lab. This year the lab’s teams tackled planetary defense, space resources and space weather. Kx supported the Solar Team, which explored whether machine learning and sophisticated data processing with kdb+ could be useful in predicting and defending our planet from asteroids and solar weather. A Kx data scientist working at the lab was able to load and analyze an extremely large dataset from the USGS Geomagnetism Program and run an analysis in nanoseconds to start getting real insights into the trends and patterns behind solar storms.

The global market for Earth observation data and value-added services is expected to grow to $43 billion in the next ten years according to Northern Sky Research. The need for innovative ways to capture, analyze, store and disseminate data as a meaningful information source will rapidly increase.

In the field of astronomy, Kx believes that due to the amount of data collected for astronomical research, including observational astronomy data, across the entire electromagnetic spectrum, coupled with other forms of data, there is a compelling need for Kx technology because of its time series nature, and its powerful real-time and historical based analytics.

© 2017 Kx Systems
Kx® and kdb+ are registered trademarks of Kx Systems, Inc., a subsidiary of First Derivatives plc.


Kx collaborating with Fintech startup chartiq

Collaboration: The Dominant Trend in Finance

13 Dec 2017 | , , , ,

In December we are re-blogging some of our favorite content from Kx partners and affiliated companies, starting with this article on the ChartIQ blog. ChartIQ is an agile FinTech company that sells an advanced HTML5 charting library used in technical data analysis, trading configurations and for charting in the capital markets industry. Kx offers a ChartIQ integration as an addition to our Dashboards. In “Collaboration: The Dominant Trend in Finance,” ChartIQ’s Hanni Chehak writes about the rise of FinTech companies, and the role collaboration plays as FinTech companies are increasingly disrupting the traditional banking sector.

Water system workers with kdb+ historical database

Kdb+ Use Case: Machine Learning Water System Maintenance Application

6 Dec 2017 | , , , ,

Kdb+ is being used much more widely in machine learning applications today. Its ability to quickly ingest and process data, particularly large, fragmented datasets, is one way that developers are adding kdb+ to their technology stack of artificial intelligence and machine learning tools.
For Australian kdb+ developer Sherief Khorshid, who also develops machine learning systems, incorporating kdb+ into a predictive maintenance application gave him the edge in a hackathon win that landed him a cash prize and a contract with the Water Corporation of Western Australia.

kdb+ FFI

Kdb+ FFI: Access external libraries more easily from q

22 Nov 2017 | , , ,

Following on from the hugely popular Python library and interface embedPy and PyQ, Kx has released an FFI as part of the Fusion for kdb+ interfaces. As with embedPy and PyQ, this FFI is open-sourced under the Apache 2 license.
The kdb+ FFI is a foreign function interface library for loading and calling dynamic libraries from q code. It has been adapted and expanded upon from a library originally written by Alex Belopolsky of Enlightenment Research. With the kdb+ FFI you can now call your favorite C/C++ libraries directly from q without the overhead of having to compile shared objects and load into q using the 2: command.