Codecard: Leveraging LLMs to Evaluate AI Model Code Development with the System Cards Framework
摘要
As AI systems are increasingly deployed in high-stakes domains such as finance, healthcare, and education, their responsible use requires scrutiny of how they are engineered, not just how they perform. Although there are now documentation standards for datasets and models, the source code that governs data handling, training, and deployment is rarely audited systematically. To address this gap, this paper introduces Codecard as a part of the System Cards Framework. Codecard is a pipeline that evaluates AI codebases against five criteria: reproducibility, design transparency, documentation quality, privacy, and testing practices. It parses each repository, segments its artifacts, and queries a large language model with structured prompts to produce a scorecard containing numeric ratings, supporting evidence, and targeted recommendations. Codecard’s evaluation of 12 public machine learning repositories revealed that only a minority achieved strong reproducibility and documentation, while testing and modular design were consistently weak. These findings show that code-level audits complement existing dataset and model documentation and can guide concrete engineering improvements.