Accurate and computationally efficient segmentation of rice panicles from high-resolution UAV imagery remains essential for high-throughput phenotyping yet challenging because of onboard computational limits and the fine-grained structure of panicles. To balance segmentation quality and resource efficiency, we propose EdgeUNetV2, a lightweight encoder–decoder network built on EfficientNetV2 that retains only the skip connections of mid-level and deep-level feature maps and incorporates an edge-aware refinement block to sharpen boundaries. A structured-pruning procedure followed by post-pruning fine-tuning cuts parameters by more than 60% and FLOPs by over 57% with negligible loss of accuracy. On the public Paddy Rice Panicle Segmentation dataset, EdgeUNetV2 attains 77.89% mIoU and 85.12% F1 while processing 512 × 512-pixel tiles at 100.3 fps. These results show that EdgeUNetV2 offers an excellent trade-off between accuracy and real-time throughput, making it well-suited to resource-constrained precision-agriculture applications on edge UAVs.