Innovative SingAREN Solutions Facilitated Speedier Transfer of Large and Complex Datasets between Australia and Singapore

23 March 2023 – In July 2022, National Environment Agency of Singapore (NEA) worked with SingAREN to assist for the transfer of over 4PBytes of climate simulation and prediction data as part of a collaboration between the National Supercomputing Centre (NSCC) Singapore and the National Computational Infrastructure (NCI) Australia.

The key challenge faced was in managing the large transfer of data and datasets with complex folder structures comprising hundreds of thousands of files and folders. SingAREN devised a simple and effective strategy to break these large datasets into smaller datasets before performing parallel data transfer between Gadi and ASPIRE 2A.

Tapping on an existing 100Gbps dedicated research network link between SingAREN and Australia, SingAREN swiftly executed the climate data transfer between the NCI Gadi and NSCC ASPIRE 2A supercomputers with an appropriate application. In comparison to the original transfer speed of 300Mbps that NEA achieved previously, SingAREN was able to achieve transfer speeds of 5-8Gbps, which are 20-30 times faster. Thus, the data transfer time was reduced to just 76 days (including interruption delays and data integrity verification ) as opposed to over a year in NEA’s plans , which were based on their original transfer speed estimations.

SingAREN engineers were able to achieve this significant increase in transfer speed through a synergistic combination of network tuning and tapping on the appropriate applications to parallelise the transfer in a reliable and easy-to-manage manner while working within tight constraints to overcome reliability issues on the source and destination. The resultant network reliability was also greatly increased.

At the end of the project, SingAREN conducted a knowledge transfer with NEA, sharing the techniques and tools used to complete the data transfer. SingAREN would like to thank NSCC Singapore, NCI Australia and NEA for their support and co-operation that contributed to the success of this project.

This article was written by Linda Lim and edited by Eugene Low

Image credit: Image by <a href=”https://www.freepik.com/free-vector/transfer-files-concept-landing-page_5749406.htm#query=data%20transfer&position=14&from_view=search&track=ais”>Freepik</a>