Co-design and Software Architecture

During the next decade we expect novel radio astronomy instruments to come online and start producing datasets of high volume and complexity at a rate that will be orders of magnitude above any existing radio telescope. This development necessitates the adoption of new data processing approaches that take advantage of latest-generation computing facilities and analysis algorithms. In this chapter, we consider associated challenges from the viewpoint of both software architecture and scientific data processing workflow. We advocate taking a co-design approach, whereby hardware and software are developed and validated in close coordination, to ensure that HPC-based workflows meet performance, cost, and reliability goals as necessitated by the overwhelming data flow expected by different observational experiments. We consider software stack development following on the principle of separation of concerns, and with reproducibility and performance portability as primary design goals, for systems built on top of heterogeneous hardware platforms. Finally, motivated by the particular case of the LOFAR radio interferometer, we present a study on how to exploit large-scale parallel architectures for efficiently solving the gridding processing step.