Logo image
Back
Relationship detection based on object semantic inference and attention mechanisms
Conference paper   Open access

Relationship detection based on object semantic inference and attention mechanisms

L. Zhang, S. Zhang, P. Shen, G. Zhu, S.A.A. Shah and M. Bennamoun
Proceedings of the 2019 on International Conference on Multimedia Retrieval - ICMR '19
2019 on International Conference on Multimedia Retrieval (ICMR) 2019 (Ottawa ON, Canada, 10/06/2019–13/06/2019)
2019
DOI: https://doi.org/10.1145/3323873.3325025
pdf
Semantic Inference.pdfDownloadView
Open Access
url
Link to Published Version *Subscription may be requiredView

Abstract

Detecting relations among objects is a crucial task for image understanding. However, each relationship involves different objects pair combinations, and different objects pair combinations express diverse interactions. This makes the relationships, based just on visual features, a challenging task. In this paper, we propose a simple yet effective relationship detection model, which is based on object semantic inference and attention mechanisms. Our model is trained to detect relation triples, such as <man ride horse>, <horse, carry, bag>. To overcome the high diversity of visual appearances, the semantic inference module and the visual features are combined to complement each others. We also introduce two different attention mechanisms for object feature refinement and phrase feature refinement. In order to derive a more detailed and comprehensive representation for each object, the object feature refinement module refines the representation of each object by querying over all the other objects in the image. The phrase feature refinement module is proposed in order to make the phrase feature more effective, and to automatically focus on relative parts, to improve the visual relationship detection task. We validate our model on Visual Genome Relationship dataset. Our proposed model achieves competitive results compared to the state-of-the-art method MOTIFNET.

Details

Metrics

201 File views/ downloads
75 Record Views
Logo image