ArtInsight: Enabling AI-Powered Artwork Engagement for Mixed Visual-Ability Families

2025-02-26

AI Summary

Blind or low-vision (BLV) parents face significant challenges in interpreting their children’s hand-drawn artwork, hindering intergenerational artistic engagement within mixed-visual-ability families. Method: We propose an AI-augmented cross-sensory interaction framework featuring: (1) the first AI description quality assessment scale tailored for children’s hand-drawings; (2) a full-cycle AI intervention model spanning pre-drawing, drawing, and post-drawing phases; and (3) a tripartite synergistic mechanism integrating large language model (LLM)-based generation, child-initiated voice input, and context-aware question-answering. Contribution/Results: Evaluated with five mixed-visual-ability families and one BLV child therapist, our system significantly outperformed existing BLV tools (p < 0.05). Users particularly valued its creative, audio-rich descriptions for “animating” visual content and centering children’s narratives—demonstrating its effectiveness and innovation in fostering inclusive, equitable art dialogue.

Technology Category

• Humans and AI: AI for Accessibility
• Computer Vision: Language and Vision
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

We introduce ArtInsight, a novel AI-powered system to facilitate deeper engagement with child-created artwork in mixed visual-ability families. ArtInsight leverages large language models (LLMs) to craft a respectful and thorough initial description of a child's artwork, and provides: creative AI-generated descriptions for a vivid overview, audio recording to capture the child's own description of their artwork, and a set of AI-generated questions to facilitate discussion between blind or low-vision (BLV) family members and their children. Alongside ArtInsight, we also contribute a new rubric to score AI-generated descriptions of child-created artwork and an assessment of state-of-the-art LLMs. We evaluated ArtInsight with five groups of BLV family members and their children, and as a case study with one BLV child therapist. Our findings highlight a preference for ArtInsight's longer, artistically-tailored descriptions over those generated by existing BLV AI tools. Participants highlighted the creative description and audio recording components as most beneficial, with the former helping ``bring a picture to life'' and the latter centering the child's narrative to generate context-aware AI responses. Our findings reveal different ways that AI can be used to support art engagement, including before, during, and after interaction with the child artist, as well as expectations that BLV adults and their sighted children have about AI-powered tools.

Problem

• Facilitates art engagement in mixed visual-ability families.
• Generates AI descriptions and questions for BLV members.
• Enhances interaction between BLV adults and sighted children.

Innovation

• AI-powered artwork engagement system
• Utilizes large language models
• Generates creative descriptions and questions