instruction crafting

This work addresses the problem of language model behavior elicitation—i.e., systematically triggering targeted model responses (e.g., harmful outputs or hallucinations) for safety evaluation. We propose an interpretable and diverse adversarial prompt generation method based on automated search. Our core methodological innovation is a Frank-Wolfe–based iterative training objective that jointly integrates supervised fine-tuning and DPO-based reinforcement learning, augmented by amortized Bayesian inference to model the distribution over effective prompting strategies. This enables an investigator agent to efficiently synthesize semantically rich, human-understandable triggers. On the AdvBench subset, our approach achieves 100% attack success rate for eliciting harmful outputs and an 85% hallucination induction rate—substantially outperforming existing heuristic- or gradient-based optimization methods. The framework establishes a new paradigm for rigorous model safety assessment and alignment mechanism analysis.

Large language models (LLMs) frequently generate factual hallucinations due to ambiguous or incomplete user prompts, undermining output reliability. This work is the first to systematically identify low-quality user-side prompting as a primary cause of hallucination. To address this, we propose a plug-and-play prompt optimization framework that employs lightweight fine-tuning of a small language model to perform prompt cleaning and intent alignment, while automatically generating structured, information-complete supplementary task descriptions—entirely without external knowledge bases. Our method significantly enhances semantic clarity and task solvability of input prompts. Extensive evaluation across multiple LLMs demonstrates over 90% win rate against baselines, substantially mitigating hallucination and improving output accuracy. The framework exhibits strong generalizability across diverse tasks and models, and enables seamless deployment with minimal computational overhead.

This work addresses the heavy reliance of current large language models on prompt engineering for generating task planning explanations, a limitation exacerbated by the lack of systematic understanding of how users construct and refine prompts. To bridge this gap, the authors propose COMPASS, a novel method that formalizes users’ cognitive states—such as attention, comprehension, and uncertainty—and integrates implicit cognitive signals with explicit interaction cues through a Partially Observable Markov Decision Process (POMDP). This framework enables adaptive prompt synthesis tailored to task planning explanations. Experimental evaluations on two cyber-physical system case studies demonstrate that COMPASS effectively leverages user cognition and feedback to significantly enhance both the quality and personalization of generated explanations.

This study addresses the frequent inefficiencies in human-AI collaboration caused by incomplete contextual information, which often leads to excessive iteration and suboptimal output quality. To mitigate this, the authors propose a structured context construction framework that integrates a five-role context package—comprising authority, exemplars, constraints, evaluation criteria, and metadata—within a four-stage workflow encompassing review, design, construction, and audit. Notably, this work pioneers the incorporation of information theory and reliability engineering principles into context quality assessment, yielding a reusable and auditable collaboration framework. Empirical results from 200 interaction trials demonstrate that the approach reduces the average number of iterations from 3.8 to 2.0, increases first-pass success rates from 32% to 55%, and achieves a final task success rate of 91.5%.

This paper investigates how developers build generative AI software incorporating natural language prompts, focusing on “prompt programming”—an emerging paradigm distinct from traditional programming in its foundational principles. Method: Applying Straussian grounded theory, we conducted in-depth interviews with 20 cross-domain developers to systematically characterize prompt programming as a novel programming phenomenon. Contribution/Results: We formally define prompt programming as centered on constructing a *mental model of foundation model behavior*, rather than a logical model of code execution. The study identifies 15 key practice characteristics—such as high iteration frequency, inherent ambiguity, and dependence on opaque model internals—revealing fundamental regularities unique to prompt development. These findings provide empirical grounding and conceptual scaffolding for designing prompt-specific tools, debugging methodologies, evaluation frameworks, and advancing software engineering theory.

Large language models (LLMs) often exhibit low alignment with human expert judgments when identifying theory-driven psychological constructs in textual data. Method: This study proposes an empirically grounded prompting framework that integrates codebook-guided instruction with automated prompt generation, systematically evaluating how construct definitions, task formulations, and exemplar selection affect prompting efficacy. We compare five prompting strategies—codebook-guided selection, automated prompt engineering, role prompting, chain-of-thought, and explanatory prompting—under zero-shot and few-shot settings. Contribution/Results: The few-shot “codebook-guided + automated engineering” strategy achieves statistically significant improvements in human–model agreement across multiple psychological constructs and mainstream LLMs, yielding an average increase of 0.23 in Krippendorff’s α. This work establishes a reproducible, theory-embedded prompting paradigm for construct-driven psychological text analysis.

This study addresses the instability of large language models (LLMs) across different prompting styles, a phenomenon whose underlying mechanisms remain poorly understood. By systematically comparing instruction-based and exemplar-based prompts through attention head analysis, representational probing, and behavioral experiments, the work identifies for the first time a shared “lexical task attention head” that operates consistently across prompting paradigms. The activation strength of this attention head strongly correlates with model performance and reliably triggers task-relevant answer generation. Furthermore, the research reveals that competition among internal task representations is a key driver of prompt sensitivity. These findings offer an interpretable framework for understanding LLM internal dynamics and suggest novel directions for enhancing prompt robustness.

This work addresses the lack of standardized documentation and evaluation methodologies in prompt engineering, which hinders the reproducibility and interpretability of complex prompts. To remedy this, the authors propose “Prompt Cards,” a novel framework that adapts the model card concept to prompt engineering by introducing a structured template to explicitly document a prompt’s design objectives, contextual strategies, evaluation protocols, and ethical considerations. Demonstrated through a “wordification” task, the approach integrates natural language generation with qualitative assessment to enable systematic recording and analysis of the entire prompting pipeline. Prompt Cards substantially enhance transparency, reproducibility, and methodological rigor, offering the research community a scalable standard for prompt documentation and a new paradigm for benchmarking beyond conventional metrics.

This study investigates how users collaborate with generative AI to accomplish 3D modeling tasks when bypassing traditional tutorials. Through qualitative observations of 26 participants—comprising 14 non-experts and 12 professionals—the research examines their natural language–driven learning trajectories, help-seeking strategies, and approaches to evaluating outputs. Three key findings emerge: (1) a novel onboarding mechanism wherein the prompt input field itself serves as the primary entry point; (2) a new practice of recursively refining prompts using external large language models; and (3) a pronounced divergence between expert and non-expert users in interpreting and filtering AI-generated results. The findings indicate that users predominantly rely on trial-and-error and external AI assistance rather than formal instruction, highlighting an emergent paradigm of human-AI collaborative learning in generative environments.

This study addresses the limitations of large language models (LLMs) in reasoning tasks involving implicit physical constraints, such as the “car wash problem,” where models struggle to infer unstated conditions. Through controlled experiments, the authors systematically evaluate the impact of prompt architectures on reasoning performance and present the first quantitative validation of the STAR structured reasoning framework’s critical role in explicitly articulating task objectives. Using Claude 3.5 Sonnet as the base model, the study integrates user-profile retrieval via a vector database and retrieval-augmented generation (RAG) for contextual enrichment. Results show that STAR alone achieves 85% accuracy, which further improves to 100% when combined with user profiling and RAG—substantially outperforming baseline approaches and underscoring the decisive contribution of explicit goal formulation to reasoning efficacy.