With the continuous development of deep learning, various new object detection models continue to emerge, among which the YOLOv8 model has attracted a large number of researchers in this field due to its excellent performance. The proposal of various new models has become particularly important for the application of object detection in various complex scenes. This article mainly discusses the introduction of a more focused bounding box loss function Focaler-IoU in the YOLOv8 model to improve its detection ability in Earth observation scenarios. The dataset of Earth observation scenes usually faces problems such as small targets, uneven distribution, and difficulty in annotation. These issues result in low accuracy of the trained model in practical detection tasks, making it difficult to meet the needs of practical applications. To address these issues, we chose to replace the original loss function of YOLOv8 with the Focaler-IoU loss function. The Focaler-IoU loss function improves the model’s performance in small object detection and imbalanced data distribution by better focusing on the accuracy of bounding boxes, enhancing the model’s learning ability for samples. The experimental results show that our proposed method improves mAP and mAP50:95 by about 3.5% compared to traditional methods, and both Precision and Recall are improved. This indicates that the improved model can be more effectively applied to object detection tasks in Earth observation scenes.