Research on type-aware fashion compatibility prediction based on a hybrid attention mechanism

In recent years, the vigorous development of e-commerce has also greatly promoted the development of the fashion multimedia field. Fashion compatibility modeling plays a very important role in a series of applications, such as fashion retrieval and personalized recommendation. Because fashion compatibility prediction usually contains multiple modal information, in this paper, we present a type-aware fashion compatibility prediction model based on a hybrid attention mechanism that is centered on using a priori category information for the fusion of multimodal information. Specifically, the model first designs a hybrid attention mechanism to fully explore the inner dependency of the visual modality while enriching with textual modalities to obtain a joint feature representation of multimodal information. Furthermore, the model matches fashion items as a distance measurement problem, and the joint feature representation of multimodal information realizes type-aware characteristics by the design of a distance measurement function with category-aware embedding. The experimental results obtained on the publicly available Polyvore and Polyvore-D datasets demonstrate the effectiveness of the model.