With machine learning (ML) models becoming ubiquitous, including in high-stakes domains, ensuring their dependability is crucial. We propose Model-Cart, a comprehensive open-source meta-framework that integrates explainability methods, human intervention, and reproducibility. By combining state-of-the-art approaches such as SHAP for model explainability, MLflow for reproducibility, and human-in-the-loop approach, this work contributes to addressing the challenges of transparency, trustworthiness, and reliability in end-to-end model training. Model-Cart incorporates explanations of model predictions, seamless integration with popular ML libraries, and a custom built module based on a novel method to quantify model explanation similarities between ML models with an intuitive UI to facilitate model training and evaluation with human-in-the-loop. We evaluate Model-Cart on a popular dataset and demonstrate its advantages over existing approaches.