With the advent of passive methods, the scope of applications for visible light positioning (VLP) systems has expanded. However, the diverse light sources present in various rooms necessitate improvements to ensure consistent deployment of passive VLP (PVLP) technology. In this paper, we propose an approach to passive VLP based on solar irradiation, which incorporates a pre-processing algorithm and a semi-supervised positioning model. The algorithm refines the Hilbert transform envelope using a dual threshold, taking into consideration both weather conditions and human activity patterns. The model employs supervised learning for solar irradiance in one room, while unsupervised learning enhances it in another. The experiments were conducted in two buildings with similar lighting conditions. The results demonstrate that our model effectively mitigates environmental and activity-related noise, achieving a mean of 86% positioning accuracy. Furthermore, the experiments indicate precise positioning in different rooms with a margin of error below 14.9 cm and maintain a pre-labeling accuracy of 84%.