<p>Fake news video detection is a challenge for social media and content platforms. There are two limitations in current method. (1) they only analyze video frames or video objects separately, which lacks information integration; and (2) the key role of semantic knowledge in recognition is ignored, and the video context information is not fully utilized. To solve these problems, we propose the Visual-Semantic Fake News Detection (VS-FND) framework, which aims to achieve deep relational inference of video content. Two types of graph memory modules are designed: (a) visual graph memory module, which focuses on mining key cues in video visual information. (b) Semantic graph memory module, which can use the semantic knowledge of videos to construct a rich semantic space and identify the semantic features related to fake news. Through the collaboration of these two modules, VS-FND build a hierarchical framework to enable visual-semantic relational reasoning from object level to frame level. We experiment on two benchmark datasets and achieve competitive performance compared to state-of-the-art methods, while also achieving significant advantages in the number of parameters and inference speed.</p>