Programmable Motion Generation for Open-set Motion Control Tasks


CVPR24 Highlight

Hanchao Liu1,2*, Xiaohang Zhan2†, Shaoli Huang2, Tai-Jiang Mu1†, Ying Shan2
1Tsinghua University, 2Tencent AI Lab
* Work done during an internship
†Joint corresponding authors
Paper arXiv Code Code(Jittor) Video

ProgMoGen is a unified framework for customizable motion control tasks.

Abstract

Character animation in real-world scenarios necessitates a variety of constraints, such as trajectories, key frames, interactions, etc. Existing methodologies typically treat single or a finite set of these constraint(s) as separate control tasks. These methods are often specialized, and the tasks they address are rarely extendable or customizable. We categorize these as solutions to the close-set motion control problem. In response to the complexity of practical motion control, we propose and attempt to solve the open-set motion control problem. This problem is characterized by an open and fully customizable set of motion control tasks.

To address this, we introduce a new paradigm, programmable motion generation. In this paradigm, any given motion control task is broken down into a combination of atomic constraints. These constraints are then programmed into an error function that quantifies the degree to which a motion sequence adheres to them. We utilize a pre-trained motion generation model and optimize its latent code to minimize the error function of the generated motion. Consequently, the generated motion not only inherits the prior of the generative model but also satisfies the requirements of the compounded constraints.

Our experiments demonstrate that our approach can generate high-quality motions when addressing a wide range of unseen tasks. These tasks encompass motion control by motion dynamics, geometric constraints, physical laws, interactions with scenes, objects or the character’s own body parts, etc. All of these are achieved in a unified approach, without the need for ad-hoc paired training data collection or specialized network designs. During the programming of novel tasks, we observed the emergence of new skills beyond those of the prior model. With the assistance of large language models, we also achieved automatic programming. We hope that this work will pave the way for the motion control of general AI agents.


Overview


Video result

More results


Atomic constraints

Directional constraint

Task:
walking + upper left arm pointing at direction d=(1,0,1)

Decomposed Constraints:
1. The bone between left_shoulder and left_elbow points at d.

Motion Programming:

total_err = 0; d = (1.0,0.0,1.0)
for frame in motions:
     dir_err = |getDir(frame[‘l_elbow’]) – d|
     total_err += dir_err

Logical operators

OR operation

Task:
walking on an inclined plane (slope=0.2)

Decomposed Constraints:
1. Either left foot or right foot is on the inclined plane.
2. Both left foot and right foot should not be under the plane.

Motion Programming:

# P is an inclined plane
total_err = 0
for frame in motions:
     total_err += min(distToPlane(frame[‘r_foot’], P),
                                   distToPlane(frame[‘l_foot’], P))
     total_err += underPlaneError(frame[‘r_foot’], P) +
                           underPlaneError(frame[‘l_foot’], P)

BibTeX

@article{liu2024programmable,
  author    = {Liu, Hanchao and Zhan, Xiaohang and Huang, Shaoli and Mu, Tai-Jiang and Shan, Ying},
  title     = {Programmable Motion Generation for Open-set Motion Control Tasks},
  journal   = {CVPR},
  year      = {2024},
}