Human-Centric Vision & Motion

MaskControl introduces controllability to generative masked motion models through a Logits Regularizer for training-time alignment and Logit Optimization for precise inference-time control. Differentiable Expectation Sampling enables optimization through discrete motion-token sampling. The method improves both motion quality and control precision while supporting any-joint-any-frame control, body-part timeline control, and zero-shot objective control.

Human mesh recovery (HMR) is crucial in many computer vision applications; from health to arts and entertainment. HMR from monocular images has predominantly been addressed by deterministic methods that output a single prediction for a given 2D image. However, HMR from a single image is an ill-posed problem due to depth ambiguity and occlusions. Probabilistic methods have attempted to address this by generating and fusing multiple plausible 3D reconstructions, but their performance has often lagged behind deterministic approaches. In this paper, we introduce GenHMR, a novel generative framework that reformulates monocular HMR as an image-conditioned generative task, explicitly modeling and mitigating uncertainties in the 2D-to-3D mapping process. GenHMR comprises two key components: (1) a pose tokenizer to convert 3D human poses into a sequence of discrete tokens in a latent space, and (2) an image-conditional masked transformer to learn the probabilistic distributions of the pose tokens, conditioned on the input image prompt along with randomly masked token sequence. During inference, the model samples from the learned conditional distribution to iteratively decode high-confidence pose tokens, thereby reducing 3D reconstruction uncertainties. To further refine the reconstruction, a 2D pose-guided refinement technique is proposed to directly fine-tune the decoded pose tokens in the latent space, which forces the projected 3D body mesh to align with the 2D pose clues. Experiments on benchmark datasets demonstrate that GenHMR significantly outperforms state-of-the-art methods.

Zero-shot skeleton-based action recognition aims to develop models capable of identifying actions beyond the categories encountered during training. Previous approaches have primarily focused on aligning visual and semantic representations but often overlooked the importance of fine-grained action patterns in the semantic space. To address these limitations, we propose a Frequency-Semantic Enhanced Variational Autoencoder (FS-VAE) to explore the skeleton semantic representation learning with frequency decomposition. FS-VAE consists of: (1) a frequency-based enhancement module with high- and low-frequency adjustments to enrich skeletal semantics and improve robustness; (2) a semantic-based action description with multilevel alignment to capture local details and global correspondence; (3) a calibrated cross-alignment loss that enables valid skeleton-text pairs to counterbalance ambiguous ones, mitigating discrepancies and ambiguities and ensuring robust alignment. Evaluations on benchmarks demonstrate that frequency-enhanced semantic features enable robust differentiation of visually and semantically similar action clusters, improving zero-shot action recognition.

3D human pose data collected in controlled laboratory settings present challenges for pose estimators that generalize across diverse scenarios. To address this, domain generalization is employed. Current methodologies typically utilize adversarial training to generate synthetic poses for training but face limitations: lack of target-domain prior complicates suitable augmentation via a single augmentor; discriminator objectives can impede exploration of out-of-source distributions; and optimization is not exposed to domain shifts. We propose a framework featuring two pose augmentors (weak and strong) with differential strategies for generation and discrimination to preserve source knowledge while exploring out-of-source distributions without target priors. We further leverage meta-optimization to simulate domain shifts during estimator optimization, improving generalization. The proposed approach significantly outperforms existing methods on benchmark evaluations.

The dominant paradigm in 3D human pose estimation that lifts a 2D pose sequence to 3D heavily relies on long-term temporal clues, which incurs performance saturation, intractable computation and the non-causal problem. This can be attributed to the inability to perceive spatial context, as plain 2D joint coordinates carry no visual cues. We propose leveraging intermediate visual representations produced by off-the-shelf (pre-trained) 2D pose detectors—no finetuning on the 3D task is needed. These representations implicitly encode joint-centric spatial context. We design a simple baseline, Context-Aware PoseFormer, and show that without temporal information it significantly outperforms context-agnostic counterparts and other state-of-the-art methods using up to hundreds of frames in both speed and precision.

Current interacting hand (IH) datasets are simplistic in background/texture, may contain inaccurate machine annotations, and have limited pose diversity—factors that hinder generalization. We present RenderIH, a large-scale synthetic dataset with accurate and diverse pose annotations, containing 1M photo-realistic images with varied backgrounds, perspectives, and hand textures. To generate natural and diverse interacting poses, we propose a new pose optimization algorithm. We also introduce a transformer-based pose estimation network, TransHand, to leverage correlations between interacting hands and demonstrate the effectiveness of RenderIH in improving results. Pretraining on this synthetic data significantly reduces error, and TransHand surpasses contemporary methods.