Unmanned retail stores are gaining popularity for their efficiency and cost-effectiveness. However, the absence of on-site supervision poses challenges in preventing excessive item pickup, potentially leading to financial losses. This underscores the need for accurate recognition of item pickup and return actions, a task complicated by limited labeled data, complex environments, and real-time constraints. To address these challenges, we introduce the UnmannedRetail-PickReturn Dataset (UR-PRD), featuring synchronized RGB and pose-estimated skeleton data collected under diverse real-world conditions, including occlusions, small object interactions, and false pickups or returns. We further propose a vision-based two-stage multimodal framework designed for real-time recognition of item pickup and return actions. Stage 1 detects key frames using skeleton data, while Stage 2 analyzes hand states within these frames to determine item possession. This design avoids full-frame RGB processing while retaining critical visual cues, achieving a balance between robustness and efficiency. Experimental results demonstrate that our base model achieves 90.6% accuracy and maintains over 90% accuracy across most scenarios. Meanwhile, the lightweight model (0.92 million parameters) processes a 64-frame item pickup or return sample in just 33.14ms on a CPU. These results highlight the effectiveness of our proposed two-stage framework, which significantly outperforms existing skeleton-based methods in both speed and accuracy, demonstrating its feasibility for real-world edge deployment scenarios. Our dataset is available at https://github.com/chelsea9067/UR-PRD .