Obstacle detection is important for keeping safety of Unmanned Surface Vehicles (USVs). Currently, obstacles are usually detected by semantic segmentation using state-of-the-art encoder-decoder based deep learning models, like CNN (Convolutional Neural Networks). However, current CNN models face challenges when applied in marine environments due to their dynamic nature. Many existing studies have used Fully Convolutional Networks for the task of image segmentation. However, in these networks, the decoder path lacks sufficient low-level features from the encoder path for the proper reconstruction of the image, because large marine dataset for USVs are needed to train complex neural networks. By considering these issues, a hybrid encoder-decoder based CNN model, called HybridNet, is proposed for improving semantic image segmentation for USVs. The proposed encoder network uses pretrained model weight for faster learning and extracts multiscale features as in many SegNet models which are memory efficient. Then, some changes have been made in the decoder layer, which adds Atrous Convolution to increase the receptive field introducing dilation rate. Besides, we have added residual block connection as in U-Net models to compensate losing spatial information in the encoder layer during down sampling operation. The proposed model is tested and cross validated on the MaSTr1325 maritime datasets. Results show that HybridNet achieves higher precision, recall rate and mean Intersection over Union (mIoU 0.98 compared with baseline models like SegNet (Segmentation Network) and U-Net.