behavioral evaluation

Conventional correlational analyses fail to establish causal links between training data and language model (LM) behavior. Method: We propose a “rewriting history” intervention framework that systematically identifies, modifies, and re-trains on training documents containing target knowledge—leveraging co-occurrence statistics and information retrieval for precise document matching—and quantifies behavioral changes via standardized benchmarks. Contribution/Results: This work introduces the first controlled, causal data intervention at the training stage, moving beyond observational studies to enable rigorous causal testing of data effects on LM behavior. Experiments demonstrate that localized data rewriting significantly alters model knowledge expression; however, current matching strategies remain insufficient to fully account for knowledge acquisition, revealing the inherent complexity of the mapping between training data and emergent model knowledge.

This work addresses the failure of probes in probe-guided fine-tuning due to Goodhart’s Law—where probes, when optimized as objectives, lose reliability. We propose an alignment method that jointly suppresses toxicity and preserves probe fidelity. Our core innovation lies in synergistically integrating supervised fine-tuning (SFT) and direct preference optimization (DPO) for probe-guided training: DPO substantially outperforms SFT, achieving both effective toxicity reduction and high probe accuracy in detecting harmful representations. Crucially, only a lightweight probe retraining post-fine-tuning is required to restore >95% of the original probe accuracy—eliminating the need for complex probe ensembling. Experiments across multiple toxicity benchmarks demonstrate significant toxicity reduction while maintaining robust internal monitoring capability, thereby validating the feasibility and practicality of probe-guided alignment.

Evaluating large language models’ (LLMs) capability to generate accurate causal loop diagrams (CLDs) for system dynamics (SD) modeling remains an open challenge due to the lack of standardized, domain-specific benchmarks. Method: This work introduces the first dual-dimensional SD-oriented evaluation framework, assessing both *technical correctness* (accuracy of causal translation from textual descriptions to CLDs) and *instruction fidelity* (adherence to structured prompting requirements). We develop the open-source sd-ai evaluation engine and a standardized test suite, integrating multi-LLM causal reasoning assessment (e.g., GPT-4.5-preview, o1, GPT-4o), structured prompt engineering, and human-in-the-loop validation. Contribution/Results: Empirical evaluation across 11 state-of-the-art LLMs reveals substantial performance variance—e.g., o1 achieves 100% causal translation accuracy, while GPT-4.5-preview attains a composite score of 92.9%. The framework demonstrates high reproducibility and community extensibility, establishing the first standardized benchmark for AI-driven SD modeling and enabling rigorous, comparable assessment of dynamic systems reasoning capabilities.

Current AI research often treats models as static artifacts, overlooking the fundamental influence of training dynamics on critical properties such as capability, bias, robustness, and safety. This work proposes shifting the focus toward the training process itself to establish a science of AI centered on training dynamics. By analyzing the interactions among data, objectives, architectures, and optimizers, the paper develops a theoretical framework that is predictive, intervenable, and design-oriented. Integrating approaches from mechanistic interpretability, fairness, memory mechanisms, and simplicity biases, it uncovers causal links between early-training signals and final model behavior. The study systematically outlines key challenges and open problems, offering both theoretical pathways and practical foundations for extending scaling laws beyond performance to encompass multidimensional model attributes.

Causal Loop Diagram (CLD) construction in system dynamics suffers from low efficiency and high entry barriers for novices. Method: This paper proposes the first stepwise prompt engineering framework tailored for CLD generation, leveraging large language models (LLMs) to automatically map textual dynamic hypotheses into structured CLDs. The approach integrates chain-of-thought reasoning, role-guided prompting, and domain-specific constraints, representing CLDs as standard directed graphs; it is fine-tuned and evaluated on a textbook-based system dynamics dataset. Contribution/Results: Experiments show that the automatically generated CLDs achieve 89% agreement with expert-built diagrams on simple dynamic structures, substantially reducing modeling time. This work establishes the first end-to-end, accurate, interpretable, and domain-aligned natural-language-to-CLD generation pipeline, empirically validating the feasibility and practical utility of LLMs in automating system modeling.

Access to real-world Applied Behavior Analysis (ABA) session data is severely limited by privacy constraints, hindering the training of AI models in this domain. To address this challenge, this work proposes a deterministic synthetic data generation method grounded in authoritative ABA taxonomies, enabling—for the first time—the construction of fully traceable instruction-tuning datasets. The approach supports two core tasks: instructional program generation and multi-session behavioral trajectory interpretation, while integrating standard ABA paradigms such as Discrete Trial Teaching and Natural Environment Teaching. The resulting TRACE dataset comprises 2,999 structurally transparent and content-compliant samples, partitioned into training, validation, test, and reasonableness-check splits according to predefined ratios. Both code and data are publicly released under CC BY-NC 4.0 and MIT licenses.

Traditional A/B testing struggles to automatically distill actionable insights from historical experiments to optimize subsequent interventions. This work proposes a tool-augmented AI agent that, for the first time, integrates the DIKW (Data-Information-Knowledge-Wisdom) reasoning framework with transparent chains of evidence for generating behavioral interventions. Leveraging field experiment data from the domain of medical prescription messaging, the approach employs a two-stage learning and optimization process. Results demonstrate that the AI-generated optimal intervention message achieves a click-through rate of 69.8%, outperforming both general-purpose large language models and theory-driven baselines by 6.5 percentage points. These findings underscore the critical role of domain-specific experimental data in enhancing the efficacy of behavioral interventions.

This study addresses a critical limitation of large language models (LLMs) in behavioral simulation: while LLMs effectively reproduce descriptive patterns such as attitude distributions, they exhibit substantial biases in estimating causal intervention effects, potentially leading to flawed policy evaluations. For the first time, this work systematically uncovers the disconnect between LLMs’ descriptive fidelity and their causal inference capabilities, identifying key factors that exacerbate this gap. Through natural language prompt–driven simulations, cross-national empirical validation, and attitude–behavior coupling analyses across three country-level datasets, the research demonstrates that LLMs significantly misestimate causal effects for interventions relying on internal subjective experiences and their behavioral outcomes, thereby revealing fundamental limitations in their causal fidelity.

This work addresses the limited generalization of existing linear probes based on internal model activations under distribution shifts, which hinders reliable monitoring of harmful behaviors in language models. To overcome this, the authors propose incorporating personality coordinates as an inductive bias: they construct a deception–ingratiation personality axis using contrastive personality prompts and apply unsupervised PCA to extract dominant directions from activation spaces. Linear probes are then constrained to this low-dimensional subspace during training. This approach substantially enhances probe robustness and transferability across datasets and behavioral contexts. Experiments on ten evaluation benchmarks demonstrate that probes trained on personality-based principal component projections consistently outperform baselines trained directly on raw activations, with a unified personality axis further improving generalization performance.

This work proposes CausalSE, a novel framework that systematically integrates structural causal models (SCMs) with propensity score matching to rigorously identify the true causal effects of interventions—such as prompt engineering—on large language model code generation performance. Addressing a critical limitation in traditional software engineering empirical studies, which often rely on statistical associations vulnerable to confounding bias, this study introduces Pearl’s causal inference paradigm into the field. Empirical evaluation on the Galeras dataset reveals that while conventional association-based analyses suggest complex prompts improve performance, causal analysis under CausalSE finds no significant treatment effect, thereby exposing false-positive conclusions arising from unaccounted confounders. The paper further provides a reproducible methodology for causal inference in software engineering contexts.