Compositional Structures as Substrates for Human-AI Co-creation Environment: A Design Approach and A Case Study

2025-03-06

AI Summary

This paper addresses core challenges in human-AI co-creation—namely, weak AI controllability, fragmented creative processes, and users’ loss of directional awareness—by proposing a collaborative authoring framework grounded in compositional structures as its foundational abstraction. The method systematically embeds AI capabilities both within individual compositional units and across inter-structural interactions, enabling integrated support for exploration, planning, iterative refinement, and real-time governance. Empirical validation is conducted via visualization-based modeling, interactive video editing, and behavior-oriented evaluation. This work establishes compositional structures as the first unified abstraction layer for human-AI collaboration, facilitating seamless transitions among multimodal workflows (human-led, hybrid, and AI-assisted). Experimental results demonstrate significant improvements in users’ perceptibility of AI generation processes, sense of control, and overall creative coherence.

Technology Category

• Humans and AI: Human-AI Collaboration / Human-AI Teaming
• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Computer Vision: Computational Photography, Image & Video Synthesis

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

It has been increasingly recognized that effective human-AI co-creation requires more than prompts and results, but an environment with empowering structures that facilitate exploration, planning, iteration, as well as control and inspection of AI generation. Yet, a concrete design approach to such an environment has not been established. Our literature analysis highlights that compositional structures-which organize and visualize individual elements into meaningful wholes-are highly effective in granting creators control over the essential aspects of their content. However, efficiently aggregating and connecting these structures to support the full creation process remains challenging. Therefore, we propose a design approach of leveraging compositional structures as the substrates and infusing AI within and across these structures to enable a controlled and fluid creation process. We evaluate this approach through a case study of developing a video co-creation environment using this approach. User evaluation shows that such an environment allowed users to stay oriented in their creation activity, remain aware and in control of AI's generation, and enable flexible human-AI collaborative workflows.

Problem

• Designing environments for effective human-AI co-creation.
• Using compositional structures to enhance control and flexibility.
• Evaluating a video co-creation environment with user feedback.

Innovation

• Leverages compositional structures for human-AI co-creation
• Infuses AI within and across structured elements
• Enables controlled, fluid, and flexible collaborative workflows