Nowadays blockchain technology is widely adopted by different sectors, not just in cryptocurrencies it is also adopted for tasks such as smart contracts and decentralized finance (DeFi), etc. With the increased complexity and utilization of blockchain networks, new security challenges have emerged, requiring advanced protective measures against attacks. This paper examines the security challenges of blockchain technology, including reentrancy, Sybil, and Distributed Denial of Service (DDoS). This research delves into the mechanisms of blockchain security threats and introduces strategies to augment resilience and detection capabilities. Here a hybrid model integrating digital signatures and SHA-256 hashing algorithms to mitigate vulnerabilities is proposed. By conducting a thorough examination including a systematic review of existing literature, creating architectural models, and analyzing empirical data on the Ethereum blockchain, the paper shows how the hybrid model is successful. The findings show a significant security improvement, with an 88% accuracy rate in identifying and addressing malicious activities. It indicates the potential of integrating cryptographic algorithms within blockchain’s consensus mechanisms to enhance transaction security.