Binary code analysis serves as the foundation for research in vulnerability discovery, software protection, and malicious code analysis. However, analyzing binary files is challenging due to the lack of high-level semantic information, leading to heavy dependence on analysts’ expertise and significantly impacting the efficiency of binary code analysis. Recent years has witnessed the blossom of machine learning models for binary analysis, but few researches address the problem of binary code datasets. In this paper, we review all the existing and available datasets, and make classification according to their application. We set up experiments to illustrate how dataset quality could affect the performance of machine learning models in binary function recognition. Based on the experimental evaluation, we present a discussion on the ground truth as well as quality evaluation problems for binary code datasets.