Shopee Price Match Guarantee Algorithm based on multimodal learning

Shopee has been a popular online shopping website in the Southeast Asia. Customers appreciate its easy, secure, and fast online shopping experience tailored to their region. At the same time, it allows customers to choose the one with the lower price of the same product. It relies on the product matching, that is the same product with the same description image must be removed. The base technology to achieve this function is multimodal learning, in which we focus on the images and text. In our article, we proposed a new multimodal learning model mainly based on transformer and BERT. For image matching, we use NFNet, Swin_Transformer and Efficientnet to get image embeddings. For text matching, we use Distil-Bert, Albert, Multilingual Bert and TF-IDF to get text embeddings. After we get the embedding vector, we choose KNN to classify. We use cosine and distance to measure the similarity of the different models. It is worth mentioning that the loss function is Arcface, not the traditional Softmax, which improve the difficulty of training to ensure the final performance in the test periods. In addition, 7 models vote for the final results ensuring the effect of prediction. To avoid the bad matching result, we add some postprocessing process.