Data processing, information retrieval and classification of atmospheric measurements

The product of this research work takes raw atmospheric science data as input and generates clean, standardized, and redundancy-free data as output. There are two major tasks involved in the work: data processing and information retrieval. Data processing involves removing inaccuracies and resolving inconsistencies among data. Information retrieval involves identifying the information needed, extracting the data, and consolidating similar entries. There are four challenges encountered in this study: the inaccuracy of the raw data, the inconsistency of records, the need for multiple criteria for validating a record, and the large number of alternatives for certain records. Some existing research dealt with some of the complexities involved in this study. Unfortunately, a solution that targets all these challenges at once was not found in the literature. The contribution of this work is to find a comprehensive approach that works with all four problems and produces a reliable outcome. In particular, fuzzy matching and fuzzy rule-based inference systems have been used for removing inconsistencies among data entries, retrieving information from certain sections of data files, and consolidating information from different sources. A rule-based system is chosen to represent the factors that are associated with the type of measurement as well as their interrelationships, and then make decisions on the category of the measurement. The retrieval of instrument information is aided by a structural-based approach using the natural language processing (NLP) technique. The output of NLP is used to match an entry in the instrument dictionary using a fuzzy rule-based inference system. A multi-criteria decision-making algorithm is used to aggregate information from different sources and select the instrument classification by factoring in the significance of each source. A software package based on the algorithms presented in this paper has been developed; the package has been deployed for real-world applications.