There are high expectations for autonomous personal mobility vehicles (PMVs) in outdoor environments to support the daily lives of older people. These autonomous PMVs are required to adhere to traffic rules and exhibit safe obstacle avoidance. Existing methods extract regions recommended for travel based on stereo camera images with semantic segmentation and navigate through them to meet the requirements. Currently, there is a problem where the recommended area is incorrectly extracted due to changes in surface brightness caused by shadows, leading to failures in autonomous navigation. Therefore, this study focused on 3D point clouds obtained with a stereo camera and proposed a robust autonomous navigation method that addresses shadows. Finally, the effectiveness of the proposed method was evaluated through autonomous navigation experiments conducted in environments that include shadows.