Globally, approximately 375 million CT scans are performed annually, with Japan performing around 30 million scans, leading in scan density. Radiologists have to face a significant workload, which results in delays and reduced diagnostic quality due to understaffing. To mitigate these issues, artificial intelligence (AI) is being explored for generating radiology reports, aiming to reduce workload and minimize human error. This study focuses on the use of vision language models (VLMs) to integrate textual and visual data. Most VLMs are designed for 2D images, while the few existing 3D VLMs either lack longitudinal volume integration or clinical histories, and none are tailored for Japanese report generation, which is further complicated by the complexity of the language. In this research, we introduce SIGMA, designed to automatically generate Japanese radiology findings reports from longitudinal 3D CT volumes, guided by medical expert instruction. Our model integrates the state-of-the-art language model, Gemma 2, with the Swin Transformer technique to enhance the efficiency of 3D image processing. We trained SIGMA using a large-scale longitudinal CT dataset, quantitatively and qualitatively evaluated its performance. The results are promising and establish the first baseline for future research in Japanese radiology report generation.