This work proposes an optimized fusion of object detection and image segmentation network for on-board chip deployable road scene perception. Firstly, the adaptive kernel convolution (AKConv) and C2F module are introduced into end-to-end network model to achieve lightweight and efficient feature extraction. Meanwhile, a novel lightweight multi-scale segmentation head module (RPFseg) based on the fusion of YOLO and PSPNet is proposed to enhance the model so as to reduce the size and parameters of the network and improve the computation speed. With the fusion of the improved modules, an optimized network structure (named FYPnet) is designed to focus on discriminative features and improve the quality of feature fusion. Experimental results show that the proposed network owns excellent performances in terms of mean intersection over union (mIoU) and frames per second (FPS).