The majority research studies on Intrusion and DDoS attack detection are just limited to the performance evaluation of detection techniques which is accomplished by using IDS datasets. But these studies cannot be considered reliable as the datasets considered do not satisfy the necessary criteria of recent IDS evaluation framework proposed by Gharib et al. in 2017. Hence, it is a high time to rethink on the quality of IDS datasets used for designing intrusion detection systems (IDSs) in the past research studies. The research insights of the present research study receives high significance because of three main reasons (a) Gharib’s 2017 evaluation framework is considered for evaluating IDS datasets (b) Benchmark IDS datasets used in various research studies are considered from 1998 to 2023 (c) We have computed reliability scores for various benchmark IDS datasets by considering the most recent 2023 NETSCOUT DDoS threat report; which has the most recent information on DDoS attack diversity, the number of DDoS attacks that have been recorded globally. After dataset evaluation and assessment, the performance of various machine learning algorithms are evaluated by using the subset CICDDoS2019 dataset which we have generated from PCAP files in one of our previous studies. Our dataset consisted of 4 Lakh training and 39998 testing network traffic flows. We have also applied non-parametric Wilcoxon signed rank test to evaluate the performance of various machine learning algorithms to gauge their suitability for DDoS attack detection.