Instability of Retraining as a Benchmark for Machine Unlearning Evaluation
摘要
Machine unlearning has become a pivotal paradigm for mitigating the impact of unwanted data points incorporated during model training. Despite recent progress in unlearning algorithms, rigorously assessing their effectiveness remains challenging. Retraining a model from scratch—though the very process many algorithms aim to avoid—continues to serve as a widely adopted benchmark because it inherently excludes undesirable data. This chapter critically examines the role of retraining as a benchmark for evaluating machine unlearning. Through extensive experiments spanning diverse metrics, datasets, and model architectures, it reveals significant instability in retrained models. In particular, retraining can produce both the strongest and weakest unlearning performance while maintaining comparable utility. This pattern persists across multiple membership inference attacks and distance-based metrics. Although these findings do not diminish retraining’s value as an unlearning method that guarantees complete data removal, they highlight the importance of applying it with caution and nuance as an evaluation standard.