Spark-Based Virtual Data Warehouse Design

Command and Control Systems are complex integrated information systems composed of multiple subsystems. Each subsystem has its own data storage methods and data formats, which makes data communication between subsystems extremely difficult. This results in data resource waste. Early methods of obtaining required data from the databases of the subsystems by programming and then analyzing them were time-consuming and labor-intensive. Therefore, in practical applications, it is necessary to integrate the data of these subsystems and provide an effective method to query these data, to help combat personnel perform analysis and decision-making. Existing data integration methods face two challenges: (1) when the source database is updated, the integrated data cannot be updated in real-time; (2) due to the inconsistent data storage types of each subsystem, a mapping and transformation mechanism needs to be manually established between subsystems data. To address these challenges, we propose a Spark-based virtual data warehouse design scheme. It accesses Spark via JDBC and operates Hive through Spark to build a virtual data warehouse. We use an Equipment Management System and Combat Mission Planning System as examples to verify the feasibility of this scheme. This scheme can effectively handle data from different subsystems, thus improving the efficiency of the Command and Control Systems.