With its characteristics of unmanned, intelligent and good maneuverability, the surface intelligent unmanned boat has a broad application prospect in the field of port patrol and suspicious vessel verification. Aiming at the tracking and recognition of dim targets on the surface of the unmanned boat in complex sea conditions, an image defogging enhancement algorithm based on guided filtering and atmospheric scattering model is adopted to pre-process the recognized target images. On the base of the study, a YOLOv8 network structure combining ACMix model and multiple detection head mechanism is proposed. After testing and verification, the research in this paper has improved the recognition accuracy and real-time performance to a certain extent, which can better achieve the recognition of dim targets on the sea surface under real complex conditions such as frequent rain and fog on the sea and salt precipitation crystallization attached to the photoelectric detection head.