The continuous advancement and rapid iteration of autonomous driving technology have made LiDAR-based 3D object detection a critical area of research in both industry and academia. Currently, two widely used approaches are voxel-based and point-based two-stage detection frameworks, while more advanced methods effectively fuse voxel and point feature representations. However, existing voxel-point fusion methods still face challenges such as poor keypoint sampling performance, inadequate multi-scale feature fusion, and low computational efficiency. To address these issues, we propose a novel 3D object detection framework, adaptive sectorized points sampling network (ASPSnet), which adapts scene encoding for objects of varying scales and achieves efficient voxel-point feature aggregation, resulting in superior detection performance with reduced resource consumption. Experiments on the KITTI dataset show that ASPSnet achieves 3D mAP of 82.26%, 54.78% and 69.32% for the car, pedestrian and cyclist categories in moderate difficulty. Experiments on the Waymo Open Dataset show that ASPSnet achieves 3D mAPH of 70.20%, 77.21% and 73.75% for the vehicle, pedestrian and cyclist categories in LEVEL2 difficulty.