Camouflaged Object Detection (COD) remains highly challenging due to the strong visual similarity between objects and their surroundings. Most existing methods rely on dense pixel-level annotations under fully supervised settings, which are expensive and difficult to scale. In contrast, current weakly supervised COD approaches often exhibit inferior performance and still suffer from the high labeling cost of certain weak supervision signals. In this work, we propose Point-to-box SAM (PB-SAM), a novel point-supervised COD framework. We first introduce a Pseudo-label Collaborative Optimization Strategy (PCOS) that generates high-quality pseudo labels by combining sparse point annotations with the Segment Anything Model (SAM). To enhance feature representation, we further design a Global-Local Interaction Module (GLIM) to capture long-range dependencies and local spatial details, and a Bidirectional Cross-layer Fusion Module (BCF) for effective multi-scale feature aggregation. Extensive experiments on multiple benchmark datasets demonstrate that PB-SAM not only significantly outperforms existing point-supervised methods, but also achieves competitive performance compared to fully supervised counterparts.