A sketch recognition method based on bi-modal model using cooperative learning paradigm

Static image is an important form of displaying a sketch, representing the appearance information of the sketch. And a stroke sequence composed of several points can also express the shape and contour information of the sketch. Therefore, it is very reasonable to treat a sketch as point-modal data and image-modal data simultaneously. In this paper, a method based on bi-modal model using cooperative learning paradigm is proposed for the sketch recognition task. Specifically, in the point-modal branch, a structural point convolution block is developed by properly dividing local regions to preserve the structural information. In the image-modal branch, the hierarchical residual structure is used to fully extract image-modal features. To reduce the negative impact of noisy samples on the recognition performance, a cooperative learning paradigm is designed based on different perceptual abilities of two modal branches on noisy samples, that is, when training the two branches, the noisy samples can be filtered out through information exchanges and mutual learning. Extensive experiments on the sketch datasets TU-Berlin and QuickDraw show that the proposed method outperforms most baseline methods and has many advantages such as no dependence on additional data and stroke information.