PLanet: Formalizing Experimental Design

2025-05-14

AI Summary

Experimental designs in scientific papers often lack clarity, communicability, and comparability, undermining conclusion reliability and generalizability. To address this, we propose the first composable, formal syntax framework for experimental design—realized as a domain-specific language (DSL)—that supports three-stage modeling: experimental unit definition, trial sequence generation, and mapping. The DSL explicitly encodes implicit design decisions (e.g., Latin square allocation), enabling precise specification and reasoning about experimental structure. This framework fills a critical formalization gap in human-computer interaction and related empirical disciplines. We empirically evaluated it on 12 studies from CHI and UIST, successfully formalizing 11. Our analysis uncovered previously unstated design ambiguities and viable alternatives, thereby enhancing experimental transparency, reproducibility, and cross-study comparability.

Technology Category

• Application Domains: Humanities & Computational Social Science
• Humans and AI: User Experience and Usability
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Carefully constructed experimental designs are essential for drawing valid, generalizable conclusions from scientific studies. Unfortunately, experimental design plans can be difficult to specify, communicate clearly, and relate to alternatives. In response, we introduce a grammar of experimental design that provides composable operators for constructing assignment procedures (e.g., Latin square). We implement this grammar in PLanet, a domain-specific language (DSL) that constructs assignment plans in three stages: experimental unit specification, trial-order construction, and order-to-unit mapping. We evaluate PLanet's expressivity by taking a purposive sample of recent CHI and UIST publications, representing their experiments as programs in PLanet, and identifying ambiguities and alternatives. In our evaluation, PLanet could express 11 out of 12 experiments found in sampled papers. Additionally, we found that PLanet constructs helped make complex design choices explicit when the researchers omit technical language describing their study designs.

Problem

• Difficulties in specifying experimental design plans clearly
• Challenges in communicating and comparing design alternatives
• Lack of formal language for constructing experimental assignment procedures

Innovation

• Grammar of experimental design with composable operators
• Domain-specific language PLanet for assignment plans
• Three-stage construction: unit, trial-order, order-to-unit