Secure and Intelligent Data Deduplication for Large Scale Data Storage in Language Model

The rapid expansion of large-scale AI systems and power-grid metrology services has intensified redundancy in training corpora and massive measurement archives, significantly increasing storage pressure and reducing data quality. In real-world grid metering workflows, large volumes of near-duplicate and semantically overlapping entries are frequently occurred. Traditional storage-oriented deduplication cannot handle these fine-grained redundancies, while existing secure schemes struggle to scale in multi-tenant utility environments. To address the above problems, this paper proposes a secure and intelligent data deduplication (SAID for short) for large scale data storage in language model. Specifically, SAID integrates strengthened encryption, randomized message-locked coordination, and a semantic near-duplicate detection pipeline with hierarchical filtering. Complemented by a parallel suffix-array architecture, the system efficiently identifies exact and near-duplicate substrings across distributed shards while preserving confidentiality. Experiments on real and synthetic corpora show that SAID achieves higher storage reduction, improved deduplication accuracy, and lower verification overhead compared with state-of-the-art baselines, enabling scalable and secure data management for large AI systems.