From the Data Generation to the Archive
摘要
In this chapter, we focus on the journey of the data from the telescope into the archive. The chapter begins with a discussion on irreversible data reduction and the storage of derived data products. We discuss dynamic filtering and dynamic archiving that can be applied to reduce the amount of data to be stored to acceptable proportions. The use of parallelisation for processing is then discussed. We continue with reflections on the needs for reproducibility, dynamic curation, and the further development of smart green computing. The next section covers several data formats as they are used in the context of data intensive radio astronomy (FITS, Casacore Measurement Set, ADIOS/HDF5) and a discussion on methods of lossy compression that can be applied to the data. In the final section of this chapter, we introduce the ways several instruments (LOFAR, Aperitif, MeerKAT, ASKAP, ALMA and MWA) ingest data into their archives.