Generating high-quality descriptions for web archives remains a time-consuming bottleneck in digital preservation workflows. This paper explores the use of large language models (LLMs) to automate this task, focusing on fine-tuning the Qwen3-8B model on a curated corpus of human-written summaries. The resulting system produces semantically accurate and context-aware descriptions of HTML records extracted from WARC files. We integrate the model into a full-stack pipeline that handles ingestion, parsing, and AI-driven analysis. Performance is evaluated using BERTScore and MoverScore, alongside manual assessments by archivists, librarians, and archival science students. Results show low Composite Edit Rates (CER) and high user satisfaction, validating both the reliability and utility of LLMs for metadata generation. These findings highlight the potential of AI to enhance scalability, consistency, and quality in archival description practices.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Towards a Digital Archivist: Applications of LLMs in Automated Web Archive Description

  • Hao Zhang

摘要

Generating high-quality descriptions for web archives remains a time-consuming bottleneck in digital preservation workflows. This paper explores the use of large language models (LLMs) to automate this task, focusing on fine-tuning the Qwen3-8B model on a curated corpus of human-written summaries. The resulting system produces semantically accurate and context-aware descriptions of HTML records extracted from WARC files. We integrate the model into a full-stack pipeline that handles ingestion, parsing, and AI-driven analysis. Performance is evaluated using BERTScore and MoverScore, alongside manual assessments by archivists, librarians, and archival science students. Results show low Composite Edit Rates (CER) and high user satisfaction, validating both the reliability and utility of LLMs for metadata generation. These findings highlight the potential of AI to enhance scalability, consistency, and quality in archival description practices.