Visual Lifelog Retrieval through Captioning-Enhanced Interpretation

2025-10-04

AI Summary

To address the challenge of retrieving fine-grained memory details from first-person visual lifelogs, this paper proposes a cross-modal retrieval method grounded in image–text matching. The core insight is to generate subjectively grounded image captions—reflecting the wearer’s experiential perspective—rather than merely objective scene descriptions, and to introduce the first annotated caption dataset specifically designed for lifelog retrieval. Methodologically, the approach integrates vision-language captioning models with text embedding techniques to project images and queries into a shared semantic space, and employs contrastive learning to achieve precise query–image alignment. Experiments demonstrate substantial improvements in text-driven retrieval accuracy for lifelog images, enabling effective memory reconstruction from personal experiences. Key contributions include: (1) three first-person-aware caption generation strategies; (2) the first lifelog-specific caption dataset; and (3) an end-to-end trainable cross-modal retrieval framework.

Technology Category

• Computer Vision: Image and Video Retrieval
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

People often struggle to remember specific details of past experiences, which can lead to the need to revisit these memories. Consequently, lifelog retrieval has emerged as a crucial application. Various studies have explored methods to facilitate rapid access to personal lifelogs for memory recall assistance. In this paper, we propose a Captioning-Integrated Visual Lifelog (CIVIL) Retrieval System for extracting specific images from a user's visual lifelog based on textual queries. Unlike traditional embedding-based methods, our system first generates captions for visual lifelogs and then utilizes a text embedding model to project both the captions and user queries into a shared vector space. Visual lifelogs, captured through wearable cameras, provide a first-person viewpoint, necessitating the interpretation of the activities of the individual behind the camera rather than merely describing the scene. To address this, we introduce three distinct approaches: the single caption method, the collective caption method, and the merged caption method, each designed to interpret the life experiences of lifeloggers. Experimental results show that our method effectively describes first-person visual images, enhancing the outcomes of lifelog retrieval. Furthermore, we construct a textual dataset that converts visual lifelogs into captions, thereby reconstructing personal life experiences.

Problem

• Extracting specific images from visual lifelogs using text queries
• Interpreting first-person activities in lifelog images through captioning
• Enhancing lifelog retrieval by converting visual data into descriptive captions

Innovation

• Generates captions for visual lifelog images
• Uses text embedding model for shared vector space
• Introduces three distinct caption interpretation methods