Polar codes have aroused extensive attention due to their capacity-achieving property and low encoding and decoding complexity. With the increasing demand for real-time and high-quality applications, achieving low-latency communication in resource-constrained scenarios such as on Internet of Things (IoT) devices has become essential. This paper proposes a modified semi-parallel decoder for 5G IoT communication, with low decoding latency and high efficiency of hardware resources. 4-bit decoding algorithm and look-ahead approach are used in this work to reduce latency caused by conventional semi-parallel architecture. For a code length of \(\hbox {N} = 2^{10}\) , the proposed decoder improves latency by 48.64% and 75.19% than the conventional semi-parallel decoder and 2-bit decoder, separately. The significant improvement in hardware utilization rate of processing elements by 68.42% and 119.35% leads to high efficiency of hardware resources.