As AI systems are increasingly deployed in high-stakes domains such as finance, healthcare, and education, their responsible use requires scrutiny of how they are engineered, not just how they perform. Although there are now documentation standards for datasets and models, the source code that governs data handling, training, and deployment is rarely audited systematically. To address this gap, this paper introduces Codecard as a part of the System Cards Framework. Codecard is a pipeline that evaluates AI codebases against five criteria: reproducibility, design transparency, documentation quality, privacy, and testing practices. It parses each repository, segments its artifacts, and queries a large language model with structured prompts to produce a scorecard containing numeric ratings, supporting evidence, and targeted recommendations. Codecard’s evaluation of 12 public machine learning repositories revealed that only a minority achieved strong reproducibility and documentation, while testing and modular design were consistently weak. These findings show that code-level audits complement existing dataset and model documentation and can guide concrete engineering improvements.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Codecard: Leveraging LLMs to Evaluate AI Model Code Development with the System Cards Framework

  • Tadesse K. Bahiru,
  • Ioannis A. Kakadiaris

摘要

As AI systems are increasingly deployed in high-stakes domains such as finance, healthcare, and education, their responsible use requires scrutiny of how they are engineered, not just how they perform. Although there are now documentation standards for datasets and models, the source code that governs data handling, training, and deployment is rarely audited systematically. To address this gap, this paper introduces Codecard as a part of the System Cards Framework. Codecard is a pipeline that evaluates AI codebases against five criteria: reproducibility, design transparency, documentation quality, privacy, and testing practices. It parses each repository, segments its artifacts, and queries a large language model with structured prompts to produce a scorecard containing numeric ratings, supporting evidence, and targeted recommendations. Codecard’s evaluation of 12 public machine learning repositories revealed that only a minority achieved strong reproducibility and documentation, while testing and modular design were consistently weak. These findings show that code-level audits complement existing dataset and model documentation and can guide concrete engineering improvements.