<p>Machine Learning models trained on code and artifacts extracted from them (e.g., version control histories, code differences, etc.), provide invaluable assistance for software engineering tasks. Despite their good performance, there exists a lack of understanding about the quality of code representations used to train these models and how different representations affect their learning process. This gap limits our ability to systematically evaluate and improve representation quality in ML-based software engineering. As such, we present DeepCodeProbe, an approach for analyzing the quality of code representations and their impact on model learning. Our study of current ML models for code shows that high-quality structured representations enable effective learning of task-specific patterns without needing to understand complete programming syntax. We demonstrate how the quality of code structure representation is more important for model performance than increasing model size. Further analysis shows that selectively scaling specific parts of a model works better than expanding the entire architecture when using high-quality representations. Based on our results, we outline practical steps for improving representation quality in ML models for code-related tasks. To support future work, we release a replication package that allows for comparative analysis of representation quality across diverse code-focused ML models. This work enhances our understanding of representation quality in ML models trained on code, laying a foundation for the development of more efficient and robust tools across software engineering tasks.</p>