Just-In-Time Objectives: A General Approach for Specialized AI Interactions

2025-10-16

AI Summary

Large language models (LLMs) often produce generic, suboptimal outputs—such as templated emails—when no explicit objective is provided, limiting their ability to satisfy personalized user needs. To address this, we propose the “Prompt Objective Inference and Dynamic Optimization” framework, which passively infers users’ transient intentions in real time from implicit interaction behaviors and dynamically regulates both the LLM’s generation process and output evaluation accordingly. Our approach integrates behavioral modeling, goal inference, controllable generation, and human-in-the-loop automated assessment, enabling zero-shot generation of personalized tools. Experiments demonstrate that our method achieves 66%–86% user preference win rates over strong baselines across multiple tasks. Crucially, it generates functionally and semantically distinct domain-specific tools for each individual user, significantly enhancing response professionalism, precision, and interactive efficiency.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large language models promise a broad set of functions, but when not given a specific objective, they default to milquetoast results such as drafting emails littered with cliches. We demonstrate that inferring the user's in-the-moment objective, then rapidly optimizing for that singular objective, enables LLMs to produce tools, interfaces, and responses that are more responsive and desired. We contribute an architecture for automatically inducing just-in-time objectives by passively observing user behavior, then steering downstream AI systems through generation and evaluation against this objective. Inducing just-in-time objectives (e.g., "Clarify the abstract's research contribution") enables automatic generation of tools, e.g., those that critique a draft based on relevant HCI methodologies, anticipate related researchers' reactions, or surface ambiguous terminology. In a series of experiments (N=14, N=205) on participants' own tasks, JIT objectives enable LLM outputs that achieve 66-86% win rates over typical LLMs, and in-person use sessions (N=17) confirm that JIT objectives produce specialized tools unique to each participant.

Problem

• Inferring user objectives to optimize LLM responses
• Automatically generating specialized tools from observed behavior
• Improving LLM output relevance through just-in-time objective induction

Innovation

• Inferring user objectives from passive behavior observation
• Optimizing LLMs for singular inferred objectives
• Generating specialized tools through objective-driven evaluation