The Exploration of Space Weather at NASA FDL with kdb+

4 Dec 2018 | , , , ,
Share on:

By Deanna Morgan

As part of Kx’s continuing relationship with the NASA Frontier Development Lab (FDL), I recently had the opportunity to work as a visiting data scientist at the NASA FDL, in Mountain View, California. The lab is an applied artificial intelligence (AI) research accelerator, hosted by the Search for Extraterrestrial Intelligence Institute (SETI), in partnership with the NASA Ames Research Center. The program brings commercial and private partners together with researchers to solve challenges facing the space science community using new AI technologies.

NASA FDL 2018 focused on four areas of research – Astrobiology, Exoplanets, Space Resources and Space Weather – each with their own separate challenges. I worked with the Space Weather 1 team on the first of the Space Weather challenges, which aimed to forecast Global Navigation Satellite System (GNSS) disruptions.

A GNSS is a network of satellites providing geospatial positioning with global coverage. The most famous example is the United States’ Global Positioning System (GPS). Such a network relies upon radio communications between satellites and ground-based receivers, which can be subject to interruptions in the presence of extreme space weather events.

Space weather refers to changes in radiation emitted by the Sun, leading to fluctuations in the Earth’s ionosphere. Changes to the electron density in the ionosphere cause fluctuations in the amplitude and phase of radio signals, referred to as phase scintillation. Radio signals propagating between GNSS satellites and ground-based receivers are affected by these scintillation events and can become inaccurate or even lost.

In a society that has become dependent on GNSS services for navigation in everyday life, it is critically important to know when signal disruptions might occur. Given that space weather events occurring between the Sun and the Earth have a nonlinear relationship, physical models have struggled to predict scintillation events. One method for making more accurate predictions is to use machine learning (ML) techniques.

Throughout the project, we examined how to to train ML models to predict scintillation events. We showed how to harness the power of the kdb+/q time-series database platform when performing data analytics and additionally, how to use embedPy for importing the necessary Python ML libraries. Publicly available historical GNSS data was used throughout.

The full paper and JupyterQ notebooks explaining the data origins, data pre-processing, feature engineering and the ML models used can be found here on the Kx Developer’s site.

Additional information about Kx at NASA FDL is below:

Case study: Kdb+ Used at NASA Frontier Development Lab in Predictive AI tool

The Exploration of Solar Storm Data Using JupyterQ

VIDEO: The Exploration of Solar Storms at NASA FDL


Deanna and Kx gratefully acknowledge the FDL Space Weather 1 team, Danny Kumar, Karthik Venkataramani, Kibrom Ebuy Abraha and Laura Hayes, for their contributions and support.


How to Avoid a Goat in Monte Carlo – Elegantly

19 Feb 2019 | ,

by PJ O’Kane and Rian O’Cuinneagain We are all encouraged to be problem solvers. Solutions, however, depend not only on the complexity of the problem but also on the skillset of the solver and the tools at their disposal. Moreover, some can be faced head-on whereas others are best addressed indirectly. If, for example, you […]

Kx Product Insights: Kx for Love!!

14 Feb 2019 | , ,

By Declan Fallontine This month’s Insights post falls on Valentine’s day so today’s article will feature a dashboard visualizing data centered on romance.  As a source we will use a blog post by dating site, Zoosk, which investigating generational differences in dating[1].   Zoosk has over 750K dating profiles and its members have generated nearly 250 million messages […]