Deal to solve SKA's big data conundrum

The move signals the unlocking of the hidden secrets in the immense amount of data generated by the SKA.
The move signals the unlocking of the hidden secrets in the immense amount of data generated by the SKA.

South Africa and the Netherlands are set to sign a memorandum of understanding (MOU) which seeks to address the big data conundrum at the Square Kilometre Array (SKA).

President Jacob Zuma and Dutch prime minister Mark Rutte will meet during Rutte's visit from 17 to 18 November "to elevate and reaffirm the longstanding and warm bilateral relations" between the countries.

Zuma invited Rutte to SA when the two met at Davos at the World Economic Forum in January.

Rutte's visit will include a pivotal South African-Dutch data science partnership between key institutions from both countries to understand the volume of data generated by the SKA.

Hidden secrets

SKA SA says this signals the unlocking of the hidden secrets in the immense amount of data generated by the SKA - the world's biggest radio telescope. The agreement is part of the visit to SA by the prime minister of the Netherlands and his trade delegation of 75 companies.

SKA SA and the University of Cape Town, through the newly established Inter-University Institute for Data Intensive Astronomy (IDIA), will sign a MOU with fellow research institutions in the Netherlands, IBM and Astron, to collaborate in a ground-breaking research project entitled Precursor Regional Science Data Centres for the SKA.

The MOU will be celebrated as part of 'House of the Future' - a programme of workshops, seminars, presentations and round tables with South African and Dutch stakeholders, taking place from 16 to 20 November in Turbine Hall, Johannesburg.

The SA-Dutch agreement on data science aims to establish national and regional data centres - to tackle one of the most significant challenges presented by the SKA: how to manage, process, and make accessible the immense amount of data the telescope will generate.

The data centres will provide astronomers around the world with access to the large-scale data infrastructures and associated high performance computing needed to make sense of the data.

"We assume there will be at least two astronomy-focused sites, one each in SA and Netherlands," says professor Russ Taylor, IDIA founding director and joint University of Cape Town/University of the Western Cape SKA research chair.

"The initial focus of the centres will be to service the current and future data archiving, distribution and science exploration needs of the MeerKAT and Lofar radio telescopes in SA and the Netherlands, respectively. The activity, combining both operational and research components, is an important step on the path towards being able to efficiently extract major science value from the massive astronomical datasets which will be collected by the SKA," says Dr Jasper Horrell, general manager of science computing and innovation at SKA SA.

The techniques developed can, in turn, be applied in other fields such as big data analytics, high performance computing, green computing, and visualisation analytics.

Understanding the universe

The SKA will be the world's largest radio telescope - a hundred times bigger than any current radio telescope; it will revolutionise the world's understanding of the universe.

It will be built in two phases - SKA1 and SKA2 - starting in 2018. SKA 1 will include two instruments - SKA1 MID (to be built in SA) and SKA1 LOW (to be built in Australia); they will observe the universe at different frequencies.

With SA's MeerKAT and the Netherlands' Apertif telescopes both expected to come online in 2016, the scale of such data collection is poised to increase significantly.

The large scale of the datasets and the requirements of the astronomers to perform complex scientific analyses, which are often compute-intensive, demand innovative approaches. Data at these scales present unique challenges not only for managing the collection, but also for how researchers extract their science.

IBM and Astron have been working together since 2012 in a five-year collaboration totalling EUR32.9 million to research exascale computer systems that will be needed by the SKA.

See also