by Oisín Creaner and Eoin Carley
For centuries, humans have looked to the skies in wonder at what’s out there. Since the mid-twentieth century, we’ve been able to look with new eyes as technology allows us to detect forms of light which our eyes cannot see. Radio astronomy is one of these windows on the universe, and the Dublin Institute for Advanced Studies (DIAS) is working with KX to maximize what we can see through that window.
With radio telescopes, astronomers can observe and investigate a huge variety of phenomena. This can range from phenomena closer to home like lightning strikes all the way to extra-terrestrial forces such as solar flares and eruptions from the Sun which could threaten satellites and communications (no more iPhones or Google Maps!). We can even take images out into the farthest reaches of space as seen recently with the Event Horizon Telescope (EHT) which took the first direct image of a black hole. Nowadays, we can stretch this even further by listening for the first signs of alien life as we contribute to programs such as the Search for Extra Terrestrial Intelligence (SETI).
I-LOFAR
Clockwise from top-left: aerial image of the Irish LOFAR station I-LOFAR in Birr, Co. Offaly. The inside of the data processing cabin on-site at I-LOFAR. A map showing the network of telescopes across Europe, which together form an enormous Big Data challenge. (image from www.lofar.ie)
I-LOFAR (the Irish Low Frequency Array, lofar.ie) is a very modern radio telescope and is one of a new breed of fixed-mount radio telescopes. Often described as a “software telescope,” I-LOFAR lacks the distinctive moveable dishes that are so iconic of traditional radio telescopes. These dishes are usually used to focus radio waves from one direction onto the detectors and allow the telescopes to be steered from one source to the next. Instead, I-LOFAR antennas are very simple in their construction, being fixed to the ground and constructed out of basic materials. The sophistication of I-LOFAR comes through the software needed to process the signals it receives.
Because it lacks a focussing dish, I-LOFAR receives signals from all directions at once. This is both the greatest asset and the greatest challenge facing the science teams. Signals from all directions reach the antennas producing a huge stream of data. Sophisticated software is required to combine the signals to focus in on the object that is being studied. In addition, since LOFAR operates at the same frequencies as FM and Air Traffic Control Radio, finding astronomical signals amongst the channels used by humans is difficult. Additional software is needed to filter out known sources of radio interference to allow scientists to study events in space without them being swamped by local news, music or talk radio! All of this requires high-speed data processing to allow for real-time results to be generated. KX are the leading specialists in real-time data processing, and with their support, DIAS scientists hope to greatly improve the speed and performance of their analyses.
As shown in the image above, I-LOFAR forms part of the Europe-spanning International LOFAR Telescope (ILT). Headquartered in the Netherlands, the ILT faces even more extreme data challenges in combining the data from each of its stations across Europe, with data rates of up to 35TB/hr.
Data Processing Software
Existing data processing techniques in the astronomical domain are dominated by bespoke software with highly specialized applications. Modern approaches in industrial applications of Big Data rely on rapid, repeatable analyses which can be controlled by a domain expert. Astrophysicists at DIAS have worked closely with KX developers to create a next-generation system that harnesses the power of kdb+, the world’s fastest time-series database, for rapid, real-time data ingestion and processing.
As soon as data is ingested into kdb+, it is instantly available for advanced analytics and visualisation on the easy-to-use KX Dashboards. All of the data is available at a glance and can be drilled-down or rolled-up easily as needed. The manner and speed at which researchers gain access to this data allows them to focus more on extracting meaning from the information presented. This was previously unattainable with legacy data analysis software platforms.
Prototype Dashboard from KX (image from Paul Lynch, Matteo Cozzi, Connor Gervin, Paul Duffy, Peter Clarke, Sean Lang & Rob Hill)
Initial results in applying these techniques to I-LOFAR data analysis challenges show a greater than 10x improvement in data ingestion speeds. Speeding data ingestion frees processing time for other tasks, and even has the potential to allow for multiple, parallel data processing streams to make better use of LOFAR’s all-sky capability. Increasing the speed and efficiency of data ingestion and processing makes it possible to perform real-time analysis on streaming datasets, with just-in-time calculations enabling response to transient events in milliseconds!
This is essential for many astronomical applications as the ability to respond to sudden events allows for instruments to be pointed at a source before the event has faded. With archival processing, it may even be possible to “travel back in time” and find the effects of something that has happened in the past.
Upcoming Developments
Based on this initial prototype, DIAS and KX have commenced an ambitious project to develop a high-speed analytics platform for radio astronomy. Together, we expect to provide a suite of software capable of addressing multiple challenges in this field. The system will be able to ingest data from both streaming and large-scale historical archives into the kdb+ architecture before harnessing the power of its analytics and dashboards to provide rapid, repeatable analyses in an accessible format.
This new approach is essential for large-scale astronomical projects currently in development, each of which faces enormous data processing challenges which DIAS and KX hope to address. One example of this is the ambitious Breakthrough Listen project which aims to detect signs of civilisations beyond our own. Another is the enormous Square Kilometer Array (SKA), which promises to generate data at a volume greater than the combined traffic of the entire modern-day internet. It is expected that these projects and others like them will require novel solutions and extreme data processing speed to succeed. Our system has the potential to revolutionize the world of astronomical data analysis and pave the way for a deeper understanding of the universe.
About the Author
Dr. Oisín Creaner is a postdoctoral researcher and software developer at the Astronomy and Astrophysics Section of the School of Cosmic Physics at DIAS. His work has included data mining of archival data to support the search for Extrasolar planets and the development of novel tools to assess the performance of radio telescope beam models. As part of that remit he is leading the research element of this innovative and exciting project with KX as an industry partner.
About DIAS
DIAS (Dublin Institute for Advanced Studies) is a centre for advanced, fundamental research in Ireland. The School of Cosmic Physics includes researchers in a variety of fields from Solar Physics to Star Formation and more. DIAS is a member of the I-LOFAR consortium which is responsible for maintenance of the Irish station of LOFAR.
About KX
KX is a division of First Derivatives (FDP.L), a global technology provider with more than 20 years of experience working with some of the world’s largest finance, technology, retail, pharma, manufacturing and energy institutions. KX technology, incorporating the kdb+ time-series database, is a leader in high-performance, in-memory computing, streaming analytics and operational intelligence. KX delivers the best possible performance and flexibility for high-volume, data-intensive analytics and applications across multiple industries. The Group operates from 15 offices across Europe, North America and Asia Pacific, including its headquarters in Newry, and employs more than 2,400 people worldwide.
