What do you see? An XAI approach for VLM-generated map descriptions

Abstract

Over the last decades, significant progress has been made in enabling diverse communities to create and share cartographic maps. However, advancements in map accessibility, for blind and visually impaired users in particular, still lag behind. A critical challenge remains in generating effective and efficient text descriptions that are supported by screen-readers. Vision Language Models (VLMs) offer a promising solution, as they can produce image descriptions quickly. However, their outputs depend heavily on network architecture and prompt engineering. Further, VLMs usually are complex and outputs are difficult to interpret. To address the interpretation of outputs in particular, we propose an Explainable AI (XAI) approach using Shapley Explanations to analyze and understand the contributions of specific map regions to the text outputs generated by a VLM. Our contribution lies in applying XAI techniques to spatial data, providing a workflow to evaluate and improve the interpretability of VLM-generated map descriptions. Data and further information can be found on a corresponding GitHub repository: https://github.com/grndng/CartoXAI