Logo image
Back
Leveraging Linguistically-aware Object Relations and NASNet for Image Captioning
Conference paper

Leveraging Linguistically-aware Object Relations and NASNet for Image Captioning

N. Sharif, M.A.A.K. Jalwana, M. Bennamoun, W. Liu and S.A.A. Shah
2020 35th International Conference on Image and Vision Computing New Zealand (IVCNZ)
35th International Conference on Image and Vision Computing New Zealand (IVCNZ) (Wellington, New Zealand, 25/11/2020–27/11/2020)
2020
DOI: https://doi.org/10.1109/IVCNZ51579.2020.9290719
url
Link to Published Version *Subscription may be requiredView

Abstract

Image captioning is a challenging vision-to-language task, which has garnered a lot of attention over the past decade. The introduction of Encoder-Decoder based architectures expedited the research in this area and provided the backbone of the most recent systems. Moreover, leveraging relationships between objects for holistic scene understanding, which in turn improves captioning, has recently sparked interest among researchers. Our proposed model encodes the spatial and semantic proximity of object pairs into linguistically-aware relationship embeddings. Moreover, it captures the global semantics of the image using NASNet. This way, true semantic relations that are not apparent in visual content of an image can be learned, such that the decoder can attend to the most relevant object relations and visual features to generate more semantically-meaningful captions. Our experiments highlight the usefulness of linguistically-aware object relations as well as NASNet visual features for image captioning.

Details

Metrics

64 Record Views
Logo image