Assisting for Open-Ended Tasks: Goal-Oriented Shared Autonomy as a Particle Filter

2026-09-26

AI Summary

Existing shared autonomy methods rely on predefined, fixed goal sets, limiting their adaptability in open, dynamic environments. This work proposes a particle filter-based dynamic goal proposal mechanism that formulates goal-directed shared autonomy as a particle filtering process. By integrating visual grounding with large language models, the approach enables semantic generation of candidate goals and real-time belief updates, effectively capturing continuously evolving user intent during interaction and overcoming the limitations of static goal assumptions. User studies demonstrate that, compared to existing baselines, the proposed method significantly reduces teleoperation time and enhances user satisfaction, thereby achieving efficient assistance for open-ended tasks.

Abstract

A common approach for shared autonomy blends human inputs with autonomous assistance based on the human's likely goal. However, most existing approaches assume that a static set of possible goals is known a priori, which limits the use of such methods in unstructured assistive settings. We instead investigate how to enable shared autonomy with open-ended and dynamically changing goals. We formulate goal-oriented shared autonomy as a particle filter in which particles represent candidate human goals. Unlike conventional approaches with a fixed goal set, our transition model dynamically proposes new candidate goals as the interaction evolves, and human actions update the belief over these goals in real time. We instantiate this framework with foundation models (e.g., vision grounding and large language models) that propose context-relevant semantic goals, generate goal-conditioned assistance from low-level skill primitives, and refine those skills from human corrections. We assess our approach through a user study where 12 participants perform a variety of tabletop manipulation tasks with our method and state-of-the-art shared autonomy baselines. The results show that our particle filter-based approach reduces the amount of time users spend teleoperating the system and improves user satisfaction. User study videos: https://youtu.be/Ii26XuRqm9c

Problem

• Shared Autonomy
• Open-Ended Tasks
• Dynamic Goals
• Assistive Robotics

Innovation

• Shared Autonomy
• Particle Filter
• Foundation Models
• Open-Ended Goals
• Large Language Models