NILM is instrumental for unleashing the energy savings potential for many smart appliances as well as demand side management for Smart Grid. Despite the good results brought about by the advancement in applying various deep learning techniques in NILM, a significant research gap lies in comparative studies that systematically evaluate techniques and “tricks” across models. Addressing this gap requires identifying the unique challenges posed by NILM, such as the necessity of conducting experiments under consistent settings. It also involves exploring appliance-specific characteristics that influence tuning strategies, notably the appropriateness of regularization techniques and the suitability of different loss functions for various appliance types. The contributions of this paper in NILM are twofold: i) an exploratory study on regularization strategy and appliance-specific loss function and a resulting deep dive into their impacts on performance of deep learning architectures and appliances, ii) a comprehensive evaluation and benchmark of NILM under the same pipeline guided by two evaluation metrics(MAE and MRE) and of three proven deep learning architectures (Seq2Seq, Seq2Point and BERT4NILM).