Co-attention enabled content-based image retrieval

TY - JOUR

T1 - Co-attention enabled content-based image retrieval

AU - Hu, Zechao

AU - Bors, Adrian Gheorghe

N1 - © 2023 The Author(s).

PY - 2023/7/1

Y1 - 2023/7/1

N2 - Content-based image retrieval (CBIR) aims to provide the most similar images to a given query. Feature extraction plays an essential role in retrieval performance within a CBIR pipeline. Current CBIR studies would either uniformly extract feature information from the input image and use them directly or employ some trainable spatial weighting module which is then used for similarity comparison between pairs of query and candidate matching images. These spatial weighting modules are normally query non-sensitive and only based on the knowledge learned during the training stage. They may focus towards incorrect regions, especially when the target image is not salient or is surrounded by distractors. This paper proposes an efficient query sensitive co-attention\footnote{``Co-attention'' in this paper refers to spatial attention conditioned on the query content.} mechanism for large-scale CBIR tasks. In order to reduce the extra computation cost required by the query sensitivity to the co-attention mechanism, the proposed method employs clustering of the selected local features. Experimental results indicate that the co-attention maps can provide the best retrieval results on benchmark datasets under challenging situations, such as having completely different image acquisition conditions between the query and its match image.

AB - Content-based image retrieval (CBIR) aims to provide the most similar images to a given query. Feature extraction plays an essential role in retrieval performance within a CBIR pipeline. Current CBIR studies would either uniformly extract feature information from the input image and use them directly or employ some trainable spatial weighting module which is then used for similarity comparison between pairs of query and candidate matching images. These spatial weighting modules are normally query non-sensitive and only based on the knowledge learned during the training stage. They may focus towards incorrect regions, especially when the target image is not salient or is surrounded by distractors. This paper proposes an efficient query sensitive co-attention\footnote{``Co-attention'' in this paper refers to spatial attention conditioned on the query content.} mechanism for large-scale CBIR tasks. In order to reduce the extra computation cost required by the query sensitivity to the co-attention mechanism, the proposed method employs clustering of the selected local features. Experimental results indicate that the co-attention maps can provide the best retrieval results on benchmark datasets under challenging situations, such as having completely different image acquisition conditions between the query and its match image.

U2 - 10.1016/j.neunet.2023.04.009

DO - 10.1016/j.neunet.2023.04.009

M3 - Article

SN - 0893-6080

VL - 164

SP - 245

EP - 263

JO - Neural Networks

JF - Neural Networks

IS - 7

ER -