Sketch2Colab: Sketch-Conditioned Multi-Human Animation via Controllable Flow Distillation

2026-03-02

AI Summary

This work addresses the challenge of generating coherent and physically plausible multi-human 3D animations from 2D storyboard sketches while satisfying complex interaction constraints—such as joint control, temporal consistency, and contact dynamics. To this end, the authors propose a controllable flow distillation framework: they first construct a sketch-driven diffusion prior, then distill it into an efficient rectified flow model in latent space, and further integrate a differentiable energy function with a continuous-time Markov chain (CTMC) event planner to enable fine-grained control over collaborative human motions. Evaluated on the CORE4D and InterHuman datasets, the method achieves state-of-the-art performance in constraint adherence and perceptual quality, while offering significantly faster inference than pure diffusion-based baselines.

Technology Category

• Humans and AI: Game Design — Procedural Content Generation & Storytelling
• Computer Vision: Diffusion Models for Vision
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

We present Sketch2Colab, which turns storyboard-style 2D sketches into coherent, object-aware 3D multi-human motion with fine-grained control over agents, joints, timing, and contacts. Conventional diffusion-based motion generators have advanced realism; however, achieving precise adherence to rich interaction constraints typically demands extensive training and/or costly posterior guidance, and performance can degrade under strong multi-entity conditioning. Sketch2Colab instead first learns a sketch-driven diffusion prior and then distills it into an efficient rectified-flow student operating in latent space for fast, stable sampling. Differentiable energies over keyframes, trajectories, and physics-based constraints directly shape the student's transport field, steering samples toward motions that faithfully satisfy the storyboard while remaining physically plausible. To capture coordinated interaction, we augment the continuous flow with a continuous-time Markov chain (CTMC) planner that schedules discrete events such as touches, grasps, and handoffs, modulating the dynamics to produce crisp, well-phased human-object-human collaborations. Experiments on CORE4D and InterHuman show that Sketch2Colab achieves state-of-the-art constraint adherence and perceptual quality while offering significantly faster inference than diffusion-only baselines.

Problem

• multi-human animation
• sketch-conditioned generation
• interaction constraints
• 3D motion synthesis
• human-object interaction

Innovation

• controllable flow distillation
• sketch-conditioned animation
• rectified flow
• differentiable physics constraints
• continuous-time Markov chain (CTMC)