Logo image
Back
Dietary Assessment With Multimodal ChatGPT: A Systematic Analysis
Journal article   Open access   Peer reviewed

Dietary Assessment With Multimodal ChatGPT: A Systematic Analysis

Frank P.-W. Lo, Jianing Qiu, Zeyu Wang, Junhong Chen, Bo Xiao, Wu Yuan, Stamatia Giannarou, Gary Frost and Benny Lo
IEEE journal of biomedical and health informatics, Vol.28(12), pp.7577-7587
2024
DOI: https://doi.org/10.1109/JBHI.2024.3417280
PMID: 38900623
pdf
Published5.22 MBDownloadView
Open Access CC BY V4.0

Abstract

Accuracy Artificial intelligence Chatbots ChatGPT deep learning dietary assessment Estimation food recognition foundation model GPT-4V Monitoring passive monitoring Task analysis Visualization
Conventional approaches to dietary assessment are primarily grounded in self-reporting methods or structured interviews conducted under the supervision of dietitians. These methods, however, are often subjective, inaccurate, and time-intensive. Although artificial intelligence (AI)-based solutions have been devised to automate the dietary assessment process, prior AI methodologies tackle dietary assessment in a fragmented landscape (e.g., merely recognizing food types or estimating portion size) and encounter challenges in their ability to generalize across a diverse range of food categories, dietary behaviors, and cultural contexts. Recently, the emergence of multimodal foundation models, such as GPT-4V, has exhibited transformative potential across a wide range of tasks in various research domains. These models have demonstrated remarkable generalist intelligence and accuracy, owing to their large-scale pre-training on broad datasets and substantially scaled model size. In this study, we explore the application of GPT-4V powering multimodal ChatGPT for dietary assessment, along with prompt engineering and passive monitoring techniques. We evaluated the proposed pipeline using a self-collected, semi free-living dietary intake dataset, captured through wearable cameras. Our findings reveal that GPT-4V excels in food detection under challenging conditions without any fine-tuning or adaptation using food-specific datasets. By guiding the model with specific language prompts (e.g., African cuisine), it shifts from recognizing common staples like rice and bread to accurately identifying regional dishes like banku and ugali. Another standout feature of GPT-4V is its contextual awareness. GPT-4V can leverage surrounding objects as scale references to deduce the portion sizes of food items, further facilitating the process of dietary assessment.

Details

Metrics

1 Record Views
Logo image