Artificial intelligence (AI) and machine learning (ML) have great potential to enhance digital forensic investigation, but progress is impeded by challenges in building datasets that meet technical accuracy and legal requirements. We herein compile findings from the latest scholarly literature to identify potential key aspects that are required for building forensic datasets that can effectively support AI-based investigative tools. We examine current practices in dataset building, ranging from representativeness of data, quality of annotation, chain-of-custody documentation, and metadata standardization, and consider their effects carefully on training robust AI models. Results point to key shortcomings that impede advanced AI implementations in digital forensics, which form a strong baseline for developing a standard workflow for building forensic datasets. This work, therefore, forms a stepping stone for future projects to enhance investigation capabilities through a better-structured and legally sound process of dataset building.