We aim to classify terrain into different ground covers, such as urban, crops, forests, water, etc., from polarimetric SAR (PolSAR) images. State-of-the-art classification approaches relish the advantage of deep learning techniques. However, conventional techniques, such as convolutional neural networks (CNN), developed for optical images are not quite suitable for complex-valued PolSAR images. Further, CNN focuses mainly on the spatial relationship within local receptive fields. However, the process entangles the channel correlation with spatial information. To address this issue, we propose a complex-valued squeeze-excitation network (CV-SENet), where the complex-valued CNN encodes the spatial relationship, and the SENet considers channel-wise important information. Thus, we utilize spatial as well as channel relationships in our work. This, in turn, helps in reducing the speckle noise in the images. The experimental results on several datasets justify the importance of spatial information and inter-channel correlation in classifying PolSAR images.