3D Visual Grounding (3DVG) involves identifying corresponding objects within a three-dimensional point cloud based on natural language descriptions. Most existing approaches focus on two-stage methods, but their performance is heavily dependent on the quality of the object detector. In contrast, single-stage methods directly perform cross-modal inference from the point cloud, bypassing the need for object detectors and preserving surrounding environmental information during the point-cloud filtering process. However, single-stage methods remain relatively underexplored. The main challenges include: (1) the difficulty in aligning point cloud features with language features across modalities, and (2) the inability of traditional Transformer structures to effectively model the local relationships between objects of different sizes within the 3D scene. To address these challenges, we propose 3D Multi-scale Local Voting (3D-MLV), a single-stage 3DVG method that employs a multi-scale local voting mechanism. This method leverages the encoder of a 3D object detector for deep feature encoding. To achieve effective cross-modal alignment, we design an optimization framework that incrementally incorporates language information into the point cloud feature vector. Additionally, we introduce a Transformer-based multi-scale local voting mechanism for seed point selection. Unlike traditional global attention, this mechanism focuses on local information around key points and encodes multi-scale contextual features through multi-head attention. Experimental results demonstrate that the 3D-MLV approach significantly enhances the performance of single-stage 3DVG.