Knowledge graphs (KGs) play a vital role in various artificial intelligence applications, including question answering, recommender systems, etc. However, the facts stored in KGs could easily become obsolete and thereby compromise the knowledge freshness and the performance of KG-based applications. Existing approaches for outdated fact detection in KGs rely either on external textual sources or human involvements, which suffer from limited coverage or high monetary cost and thus are not suitable for large-scale KGs. In this work, we propose GTS2, a novel framework for end-to-end outdated fact detection from time-evolving KG sequences, which is capable of capturing both graph temporal-structural dependencies and textual semantics. Specifically, we design a temporal-structural module, which consists of a sequence of temporal relation-aware graph convolutional network blocks and a temporal self-attention block, to capture the temporal-structural dependencies. A textual semantic module is applied to model the fact-level textual semantics based on descriptions of entities and relations. Finally, we employ a fusion module based on cross-attention to integrate the temporal-structural dependencies with the textual semantics, considering their inter-modality relationships. Experimental results on real dynamic KG sequences demonstrate that our proposed framework achieves noticeable improvements over the state-of-the-art baselines.