Digital Elevation Models (DEMs) are essential to advance earth observation applications such as city model development and monitoring of natural hazards including earthquakes, landslides and floods. However, for prediction and monitoring purposes, DEMs additionally have to be represented as objects with a timeline. As this leads to a large amount of spatio-temporal data, handling and analysing them makes an efficient geodatabase indispensable. In this paper we first introduce a database analysis framework for spatio-temporal DEMs. Second, extending the system architecture, a deep learning-based analysis service is presented that integrates semantic classification into the database analysis framework. Initial results based on the U-Net architecture with combined inputs, coming from airborne laser scanning and OpenStreetMap, demonstrate a significantly improved building height estimation and semantic separation of buildings and trees. Overall, the approach aims to enhance the semantic, i.e. non-pixel-oriented spatio-temporal analysis of DEM datasets on object- and feature-level. Finally, conclusions are drawn from our previous experiences and an outlook is given on future research advancing AI-based analysis of spatio-temporal Digital Elevation Models.