Files and links (1)
Open Access CC BY V4.0
Journal article
Egocentric Image Captioning for Privacy-Preserved Passive Dietary Intake Monitoring
IEEE transactions on cybernetics, Vol.54(2), pp.679-692
2024
PMID: 37028043
Abstract
Camera-based passive dietary intake monitoring is able to continuously capture the eating episodes of a subject, recording rich visual information, such as the type and volume of food being consumed, as well as the eating behaviours of the subject. However, there currently is no method that is able to incorporate these visual clues and provide a comprehensive context of dietary intake from passive recording (e.g., is the subject sharing food with others, what food the subject is eating, and how much food is left in the bowl). On the other hand, privacy is a major concern while egocentric wearable cameras are used for capturing. In this paper, we propose a privacy-preserved secure solution (i.e., egocentric image captioning) for dietary assessment with passive monitoring, which unifies food recognition, volume estimation, and scene understanding. By converting images into rich text descriptions, nutritionists can assess individual dietary intake based on the captions instead of the original images, reducing the risk of privacy leakage from images. To this end, an egocentric dietary image captioning dataset has been built, which consists of in-the-wild images captured by head-worn and chest-worn cameras in field studies in Ghana. A novel transformer-based architecture is designed to caption egocentric dietary images. Comprehensive experiments have been conducted to evaluate the effectiveness and to justify the design of the proposed architecture for egocentric dietary image captioning. To the best of our knowledge, this is the first work that applies image captioning for dietary intake assessment in real life settings.
Details
- Title
- Egocentric Image Captioning for Privacy-Preserved Passive Dietary Intake Monitoring
- Authors/Creators
- Jianing Qiu - Imperial College LondonFrank P. -W. Lo - Imperial College LondonXiao Gu - Imperial College LondonModou L. Jobarteh - Imperial College LondonWenyan Jia - University of PittsburghTom Baranowski - Children's Nutrition Research Center at Baylor College of MedicineMatilda Steiner-Asiedu - University of GhanaAlex K. Anderson - University of GeorgiaMegan A. McCrory - Boston UniversityEdward Sazonov - University of AlabamaMingui Sun - University of PittsburghGary Frost - Imperial College LondonBenny Lo - Imperial College London
- Publication Details
- IEEE transactions on cybernetics, Vol.54(2), pp.679-692
- Publisher
- IEEE
- Number of pages
- 14
- Grant note
- OPP1171395 / Bill and Melinda Gates Foundation; Bill & Melinda Gates Foundation OPP1171395 / Bill & Melinda Gates Foundation; Bill & Melinda Gates Foundation Grand Challenges Explorations Initiative; CGIAR Innovative Passive Dietary Monitoring
- Identifiers
- 991005884853007891
- Copyright
- © 2023 The Authors.
- Murdoch Affiliation
- Centre for Computational and Systems Medicine
- Language
- English
- Resource Type
- Journal article
UN Sustainable Development Goals (SDGs)
This output has contributed to the advancement of the following goals:
Source: SDGs in the Output
Metrics
1 Record Views
InCites Highlights
These are selected metrics from InCites Benchmarking & Analytics tool, related to this output
- Collaboration types
- Domestic collaboration
- International collaboration
- Citation topics
- 4 Electrical Engineering, Electronics & Computer Science
- 4.48 Information Retrieval & Knowledge Systems
- 4.48.672 Natural Language Processing
- Web Of Science research areas
- Automation & Control Systems
- Computer Science, Artificial Intelligence
- Computer Science, Cybernetics
- ESI research areas
- Computer Science