<p>Advances in machine learning have enhanced researchers’ ability to detect vaccine hesitancy on social media using Natural Language Processing. Our objective in this study was to evaluate, through a systematic review of studies that employed machine learning to analyze sentiment and stance regarding COVID-19 vaccines on Twitter in assessing their methodological quality and consistency. We searched for papers published between 1 January 2020 and 31 December 2023 in PubMed, Web of Science, and Scopus. The inclusion criteria were the use of supervised machine learning to assess COVID-19 vaccine hesitancy through stance detection or sentiment analysis on Twitter/X. We categorized the studies according to a taxonomy of five dimensions: tweet sample selection approach, self-reported study type, classification typology, annotation codebook definitions, and interpretation of results. We analyzed the risk of bias in the included studies, examining whether stance detection was used to report different hesitancy trends compared to those using sentiment analysis. We identified 51 papers that were published in 36 journals. Our review found that measurement bias is widely prevalent in studies employing supervised machine learning to analyze sentiment and stance toward COVID-19 vaccines and vaccination. The reporting errors are sufficiently serious to hinder the generalizability and interpretation of these studies, making it difficult to determine whether discursive content communicates reluctance to vaccinate against SARS-CoV-2. Our findings underscore the importance of more transparent reporting of NLP methods in vaccine discourse studies. Addressing methodological shortcomings is essential to improving our understanding of vaccine hesitancy on social media.</p>