Purpose <p>Monitoring large-scale benthic habitats using visual imagery requires extensive in-water surveying, significant human effort for manual data analysis, and the integration of artificial intelligence (AI)/machine learning (ML) to enhance efficiency for time-series monitoring. Achieving accurate and measurable results from this process remains challenging due to biases and errors in the data capture, annotation, analysis, and automated reporting stages.</p> Methods <p>For these reasons, we examine the process of achieving full automation in the classification of submerged macroalgal forests in benthic survey imagery. We adapt a framework from social sciences for capturing the total error of human judgment and apply it to benthic image analysis, exploring the potential extent of bias and error at each stage of the pipeline and analyzing how these issues may affect the downstream reporting of macroalgal forest condition.</p> Results <p>We demonstrate that various sources of error can arise in the image analysis pipeline, spanning biases arising from the method (image-based versus manual diver-based transects), and noise arising from different annotators applying their judgement during the image labelling process. Our findings indicate current methods of automated benthic image classification are likely to be biased and noisy with the potential to result in inaccurate and uncertain estimates of benthic cover.</p> Conclusion <p>We outline some key solutions for minimising errors that include quality assurance during the design and application of image sampling, calibration of label criteria among human annotators, using multiple human annotators to unpack and minimize inter-annotator noise effects on final estimates, and the use of an ensemble of ML models. Together, these approaches will minimize errors and limit the propagation of human-induced noise into estimates of ecological condition in seascapes over space and time.</p>