Abstract
Medical Visual Question Answering (VQA) requires models to reason over medical images and generate clinically meaningful responses. While recent approaches increasingly rely on multimodal foundation models and full-parameter fine-tuning, such methods impose substantial computational and memory costs. In this work, we present MedVisualLLM, a lightweight generative framework for medical VQA that balances efficiency and performance. The proposed architecture integrates a biomedical vision encoder with a question-aware Mixture-of-Experts (MoE) projection module and a parameter-efficient large language model. Visual features are aligned with the language embedding space and incorporated as contextual tokens, enabling multimodal reasoning without requiring specialized cross-modal transformer architectures. The training follows a two-stage strategy: domain alignment on PMC-VQA and task-specific fine-tuning with Low Rank Adaptation. Experiments on both closed-ended and open-ended questions on Path-VQA and SLAKE demonstrate that MedVisualLLM obtained 91.97% 91.97% closed accuracy and 35.81% 35.81% open accuracy on Path-VQA, along with 88.22% 88.22% and 82.79% 82.79% on SLAKE, respectively. These results are obtained while updating only 2–4% of the total model parameters. Although large fully fine-tuned multimodal Vision Language Models achieve higher open-ended accuracy, especially on Path-VQA, the proposed framework maintains competitive closed accuracy while substantially reducing training overhead. These results suggest that efficient domain alignment and parameter-efficient fine-tuning may reduce reliance on full-parameter training in medical VQA under resource-constrained settings.