Abstract
Post-hoc local explanations are widely used to improve the transparency of AI predictions. However, their reliability varies across methods and data settings. This study investigates whether Large Language Models (LLMs), using structured prompting and in-context learning (ICL), can function as local explainers for tabular models and compares their performance with Local Interpretable Model-agnostic Explanations (LIME). We introduce a reproducible benchmarking framework to evaluate explanation quality by assessing each method’s ability to recover instance-level ground-truth feature contributions from linear additive models. This serves as a sanity check of local explanation validity. We further examine the influence of key factors, including the number of numerical and categorical features and the number of in-context examples (shots). Faithfulness is evaluated using the Model-Intrinsic Additive Scores (MIAS) metric, while robustness is assessed through standard insertion–deletion metrics across multiple public tabular datasets. Empirical results indicate that LLM-based local explanations do not consistently recover instance-level ground-truth contributions. While they perform comparably in low-shot settings, their performance plateaus or declines as additional contextual information is introduced.