In this connecting world of IoT, communication is in danger from insider and outsider attacks. The network nodes are continuously threatened to be attacked and fooled by the one with whom they are communicating. For this problem, a trust value evaluation and monitoring model for smart home-based IoT networks is proposed in this work. For accurate evaluation of trust values, the communication-specific features are taken and selected as compared to the random selection of features. The feature set shown helps evaluate the correct trust value of a node in an IoT network. Then, the membership value for each feature in trustworthy set is estimated, based on which the final trust value using min–max composition is evaluated. Also, using machine learning based K means clustering approach, malicious nodes with similar communication characteristics are grouped in one group and normal nodes in another group with similar characteristic values. The proposed model is a fuzzy logic based trust evaluation FLBTE and clustering-based malicious node detection model during DoS or DDoS attack, showing 97% detection accuracy.