Camouflaged object detection (COD) is a challenging task that aims to identify and segment objects visually similar to their surroundings. Despite remarkable advances, existing COD methods often face issues with inaccurate localization and detail loss, primarily due to uniform encoder-decoder processing and suboptimal feature integration. Additionally, most methods require significant parameters and high computational complexity. To address these challenges, we propose a lightweight inter-group and intra-group mutual learning network (IGML-Net) that explores both distinctive properties and intrinsic commonalities of deep semantic and shallow detail features through a tailored learning strategy. Specifically, we divide backbone features into two groups to separately learn semantic and detail representations. The Hierarchical Semantic Attention Integration (HSAI) module enhances deep features consistency while modeling long-range dependencies, generating reliable semantic guidance. Guided by this semantic information, the Detail-Enhanced Adaptive Fusion (DEAF) module improves detail capture in target regions. Finally, the Semantic-Detail Cooperative Refinement (SDCR) module fuses complementary features and suppresses background noise for sharper boundaries. Extensive experiments demonstrate that IGML-Net achieves superior performance on three COD datasets with only 11.43M parameters and 9.25G FLOPs, outperforming 14 advanced methods in both accuracy and efficiency.