ViDiHand: The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction
๐ Paper | ๐ Project Page | ๐ป GitHub | ๐ญ Follow-up: ACE-Ego-Hand
Reconstructing 4D hand motion from egocentric video is a persistent challenge, largely because of the pervasive occlusion in real-world human activity. The demand for solving it has surged with the rise of embodied AI, where egocentric video provides a natural and scalable source for robot learning: recent efforts such as EgoScale, EgoMimic, EgoDex, and EgoVLA scale dexterous manipulation by pretraining on large amounts of egocentric human video, with hand and wrist trajectories serving as key supervision for imitation and policy learning. The quality of reconstructed hand motion therefore directly shapes what a policy can learn from egocentric video at scale, where dense labels are scarce.
Existing methods, however, struggle with exactly this kind of interaction-rich, occlusion-heavy footage. Image-based methods reconstruct each frame from a detector crop, so a missed detection is a missed reconstruction. Temporal-based methods add temporal modules on top, but learn them only from scarce hand-pose annotations.
We introduce ViDiHand, the first method to leverage a pretrained video diffusion model for 4D two-hand motion reconstruction from egocentric video. Rather than treating the generative model as a frozen feature extractor, we adapt it through a hand-overlay rendering task, and a dual-branch decoder then reads articulated 3D pose and per-joint 2D localization out of the adapted features, anchoring both hands at metric scale. The pipeline operates directly on full video frames, with no upstream hand detector, motion infiller, or test-time optimization, and sets a new state of the art on ARCTIC, HOT3D, and HOI4D.
In short, ViDiHand shows that video diffusion models are a powerful new foundation for hand motion reconstruction.
Three challenges that break the detector-crop-regress recipe
Existing hand recovery methods share the same recipe: detect the hand, crop it, and regress a mesh from the crop. Interaction-rich, occlusion-heavy egocentric video breaks this recipe in three places.
Occlusion. In egocentric scenes, hands are often occluded by the objects they manipulate or by each other. The detectors these pipelines depend on, such as YOLO or ViTPose, cannot fire without visual evidence of the hand, and under heavy occlusion a failed detection directly leads to a failed reconstruction.
Tracking. Image-based methods reconstruct each frame independently and cannot keep tracking a hand once it is occluded. Temporal-based methods use the visible frames as anchors and interpolate between them with a motion prior learned from 3D hand trajectories alone, decoupled from the surrounding scene and the ongoing interaction. The visual clues in the rest of the video go unused, and tracking through occlusion remains unreliable.
Smoothness. Per-frame methods carry no temporal information and produce visible jitter. Temporal-based methods suppress it with a learned motion prior or test-time optimization, but both need a track to smooth. Under occlusion the track is unreliable, and the jitter returns.
These limitations call for representations beyond image-only and hand-only priors: representations that capture the geometry, motion, and interaction of the visual world in which the hands operate.
The surprise: video diffusion models already encode these priors
Such representations are, in fact, increasingly available in large-scale video generative models. Trained to synthesize temporally and geometrically coherent video at internet scale, a video diffusion model must implicitly address the same structural challenges that 4D hand reconstruction faces: spatiotemporal consistency, 3D geometry from 2D observations, and reasoning about occluded content. In terms of the three challenges above, to generate a plausible clip it has to synthesize content through occlusion, keep stable identity and spatial placement across frames, and produce smooth motion. These are exactly the priors that hand recovery has long delegated to external modules.
The internal features of generative models have already been shown to support point tracking, dense prediction, and 3D scene awareness, yet no prior work had leveraged such priors for 4D hand reconstruction. The pretrained model already carries much of the signal. The adaptation we describe next sharpens it toward hands, in particular toward joints that are hidden.
ViDiHand: Adapt, Read, Decode
Adapt. We adapt Wan2.1-VACE by training it to render a semi-transparent hand overlay onto the input video: joint skeletons first, on EgoDex, then full MANO meshes, on ARCTIC and HOT3D. Only the VACE branch is trained, while the base DiT stays frozen so that its video prior is preserved. Because the overlay must also be rendered on frames where the hand is fully occluded, the backbone has to maintain a per-hand 3D state rather than merely inpaint the visible pixels.
Read. We extract features at a single point in the model: DiT layer 15 of 30, at denoising step ฯ โ 0.7. Layer 15 keeps enough spatial resolution to localize joints and enough abstraction to commit to articulated pose, and at ฯ โ 0.7 the latent has not yet committed to the rendered overlay appearance. Both were chosen by sweeping layers and denoising steps.
Decode. A dual-branch decoder reads MANO parameters from that feature. A hand-token branch, with one learned query per hand, captures articulated pose, a holistic property of the hand. A joint-heatmap branch localizes each joint on the image, a local one. Mutual cross-attention couples the two, and a mixed-projection head regresses depth and solves the in-plane translation in closed form. The decoder is trained on cached features, without backpropagating through the diffusion model.
Results
We evaluate on three egocentric benchmarks, ARCTIC, HOT3D, and HOI4D, against eight baselines. ViDiHand achieves superior performance across the board, ranking first on 26 of 27 metrics. On ARCTIC, frame accuracy rises from 0.919 to 0.997, cutting the frame-error rate 27ร. 2D end-point error falls from 50.5 px to 12.4 px, and jitter drops from 12.8 to 3.2, 4ร below the smoothest prior method. On HOT3D, MPJPE-p improves from 31.0 mm to 21.5 mm, and on held-out HOI4D ViDiHand ranks first on eight of nine metrics.
Qualitative results
4D comparison under occlusion. We visualize the recovered hands in 4D and pause at the hard frame to view the full hands from orbiting viewpoints, with each method's mesh overlay inset at the top right. Blue is the left hand and red is the right.
On ARCTIC, the left hand goes fully under the box lid. WiLoR and OmniHands recover only the visible right hand. ViDiHand recovers both.
On ARCTIC, the mixer hides most of both hands, and both sit at the frame edge with only small parts visible. The baselines detect a single hand, and ViDiHand recovers both.
On HOT3D, one hand is fully hidden by the bowl. WiLoR misses it, OmniHands misplaces it, and ViDiHand keeps it on the bowl.
Comparison with all baselines. Each example shows the input, all baselines, and ViDiHand as mesh overlays on the same clip.
On ARCTIC, one hand slides fully behind the box. Once it disappears, most baselines fail to detect it or swap its handedness.
On HOT3D, one hand is severely occluded by the bowl. Most baselines fail to detect it, and ViDiHand tracks it throughout.
In the wild, a towel severely occludes both hands. Several baselines fail to detect one or both of them.
Beyond ViDiHand: from renderer to encoder
ViDiHand establishes the prior, but it pays for it. Reading a mid-denoising activation needs a sampling pass, which is why ViDiHand runs at 5.5 fps across four A100 GPUs. And a hand that leaves the frame entirely is still out of reach.
ACE-Ego-Hand, a follow-up from several of the same authors, keeps the prior and drops the sampling. It treats the video diffusion model as an encoder rather than a renderer:
- One deterministic pass. A single noise-free forward pass over the clean latent replaces multi-step sampling. The backbone is adapted end to end with LoRA inside the training loop, so the features are shaped by the reconstruction objective instead of staying generic.
- Whole-clip decoding. A bidirectional spatiotemporal decoder attends over the entire clip. A hand that has left the frame keeps a coherent articulation and global orientation through the gap, inferred from past and future context. Where it sits in depth while unseen is regressed rather than observed, and stays the looser part of the estimate.
- Optional intrinsics. A ray-based camera solver supports a second, separately trained configuration that needs no camera intrinsics at test time.
Across five egocentric benchmarks it leads on 27 of 29 metrics. Measured against ViDiHand, it cuts MPJPE-p by 30% on ARCTIC (21.7 โ 15.3 mm), 40% on HOT3D (21.5 โ 12.9 mm) and 23% on zero-shot HOI4D. Once out-of-sight hands enter the evaluation, the reduction reaches 46โ61%.
Speed follows from the single pass. On one A100 it reaches 63.1 fps of end-to-end clip throughput, against 1.91 fps for ViDiHand in its accuracy configuration โ a 33ร gap on the same GPU. Both remain offline tools: ACE-Ego-Hand reads a whole 81-frame clip in one pass, so its latency is bounded below by clip length, not by that throughput.
Its most telling result is about the prior itself. The best discriminative video encoder comes close while the hands are visible, but degrades far more once they leave view. That suggests generative pretraining contributes a temporal-geometric prior that discriminative pretraining does not.
Outlook
For embodied AI, hand motion reconstruction is used mostly offline, to turn recorded human video into training data for policy learning. Accuracy therefore matters more than latency, and both methods sit squarely in that regime. Both also still depend on pose-labeled video for training โ though full MANO labels are not essential, since joint-only annotations already carry the first adaptation stage.
We read these results as early evidence of a shift in 4D hand reconstruction: away from hand-centric pipelines patched with specialized modules, and toward representations inherited from the video generative models the community is already advancing at scale.
Citation
If ViDiHand or ACE-Ego-Hand supports your research, please consider citing:
@misc{wang2026surprisingeffectivenessvideodiffusion,
title={The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction},
author={Yuxi Wang and Chengkai Jin and Yufei Liu and Wenqi Ouyang and Tianyi Wei and Zhiwei Zeng and Siyuan Huang and Zhiqi Shen and Xingang Pan},
year={2026},
eprint={2606.30308},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2606.30308},
}
@misc{liu2026aceegohandrepurposingvideodiffusion,
title={ACE-Ego-Hand: Repurposing Video Diffusion Models for Occlusion-Robust Egocentric 3D Hand Motion Recovery},
author={Yufei Liu and Xixi Wang and Hao Li and Ganlong Zhao and Kaitong Cai and Chengkai Jin and Chunxiao Liu and Jianbo Liu and Siyuan Huang and Xingang Pan and Hongsheng Li},
year={2026},
eprint={2608.20308},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2608.20308},
}


