Four real-world MoViC demos: a humanoid robot imitating a dancer, following a performer through the Dangerous dance, moving a box with both hands, and carrying a bag with one hand.

MoViC

Real-Time Humanoid Whole-Body Control
from Monocular Video Streams

One RGB camera · No wearable trackers · No per-video training

MoViC: Real-Time Humanoid Whole-Body Control from Monocular Video Streams

MoViC drives a full-size humanoid from a single RGB camera in real time, with no wearable trackers and no per-video training.

Abstract

Real-time whole-body imitation from monocular video requires both a robust universal tracker and high-quality real-time reconstruction. We present MoViC, a framework that uses live RGB video as a control signal for humanoid robots without wearable tracking devices. The framework combines monocular human motion reconstruction, robot retargeting, and a transformer-based behavior foundation model. The policy receives proprioceptive history, a single explicit next-step reference, and a compact frequency-domain motion representation that supplies temporal context without explicitly stacking future reference frames, which reduces the input dimension and network size. During deployment, our extrapolation method bridges the 10-Hz visual estimation with 50-Hz controller, enabling asynchronous visual processing and policy execution. Hardware experiments further demonstrate highly dynamic motion tracking and device-free teleoperation of bimanual and single-arm manipulation tasks from live monocular RGB input.

System Overview

1 RGB camera
No motion-capture suit, VR headset, or wearable trackers.
10 Hz → 50 Hz
Latent-dynamics extrapolation bridges vision and control rates.
One policy
No per-video or per-task training: the same tracker dances and teleoperates.

MoViC drives humanoid whole-body control from live monocular RGB video. An online video-to-motion module, SEG, reconstructs the performer with SAM 3D Body at approximately 10 Hz, retargets the motion to the robot, and extrapolates it into 50 Hz reference motions. A transformer-based behavior foundation model then tracks these references by combining proprioceptive history with an explicit next-step reference and a frequency-domain θ goal.

MoViC overview: imitation examples, the transformer tracker, tracking errors against baselines, and the end-to-end deployment pipeline from RGB video to the humanoid.

Overview. Bottom: monocular RGB video is reconstructed, retargeted and extrapolated, producing 50-Hz control references. Top: imitation examples (left), the transformer tracker with frequency-domain conditioning (center), and tracking errors on an unseen dance sequence (right; lower is better).

Algorithm Framework

Framework diagram: (a) frozen frequency-domain motion encoder, (b) transformer actor, (c) SEG deployment module, (d) PPO training with a privileged critic.

Overview of the whole-body motion-tracking framework.

(a)Frequency-domain motion encoder. A frozen encoder, built on latent dynamics, maps reference-motion windows to compact, phase-augmented frequency-domain codes θ.

(b)Transformer actor. The actor integrates ten frames of proprioceptive history, a single next-control-step reference target, and a θ token through self-attention. The output at a learnable query position is mapped to 24 joint position targets for PD control.

(c)Deployment with SEG. At deployment, reference motions are produced by the SEG module, which combines SAM 3D Body reconstruction, latent-dynamics extrapolation, and GMR motion retargeting. Extrapolation generates intermediate references for 50-Hz policy execution from an approximately 10-Hz motion stream.

(d)Training. The policy is optimized with PPO and a privileged critic that receives privileged state history and the next-step reference target, but no θ codes. The dashed branch is used only during training.

Real-World Experiments: Highly Dynamic Motion Tracking

Bencaogangmu dance from live RGB

Whole-body imitation of a live performer

Dangerous dance from live RGB

Real-World Experiments: Vision-Based Teleoperation

(a) Bi-manual pick, reorient, and place

(b) Single-arm pick, carry, and place