Generative & Spatial Intelligence

Recent generative models can synthesize high-quality images, but they often fail to generate humans interacting with objects using their hands. This arises mostly from the model’s misunderstanding of such interactions and the hardships of synthesizing intricate regions of the body. In this paper, we propose GraspDiffusion, a novel generative method that creates realistic scenes of human-object interaction. Given a 3D object, GraspDiffusion constructs whole-body poses with control over the object’s location relative to the human body, achieved by separately leveraging generative priors for body and hand poses and optimizing them into a joint grasping pose. This pose guides image synthesis to correctly reflect the intended interaction, creating realistic and diverse human-object interaction scenes. We demonstrate that GraspDiffusion can successfully tackle the relatively uninvestigated problem of generating full-bodied human-object interactions while outperforming previous methods.

To enhance controllability in text-to-image generation, ControlNet introduces image-based control signals, while ControlNet++ improves pixel-level cycle consistency between generated images and the input control signal. To avoid the prohibitive cost of back-propagating through the sampling process, ControlNet++ optimizes only low-noise timesteps using a single-step approximation, which ignores high-noise timesteps and introduces approximation errors. A straightforward alternative is Direct Preference Optimization (DPO), which prefers more controllable images over less controllable ones, but it is difficult to ensure win–lose image pairs differ only in controllability. We instead perform preference learning over control conditions rather than generated images. Specifically, we construct winning and losing control signals and train the model to prefer the winning condition. This Condition Preference Optimization (CPO) eliminates confounding factors and yields a low-variance objective, with theoretical variance reduction over DPO and superior empirical results. Extensive experiments show that CPO significantly improves controllability over ControlNet++ across multiple control types.

Existing instruction-based image editing datasets often contain noisy supervision due to mismatches between editing instructions and original–edited image pairs. Prior efforts improve editing models by generating higher-quality edits, pre-training on recognition tasks, or introducing VLMs but do not resolve this fundamental issue. We propose a solution by constructing more effective editing instructions for given image pairs: rectifying instructions to better align with the image pairs and using contrastive editing instructions to further enhance supervision. We define a unified guide for VLMs to rectify instructions based on observed step-wise generation attributes, and introduce contrastive supervision signals with positive/negative instructions via triplet loss to handle challenging scenarios. Our method avoids VLM modules and pre-training tasks used in previous work, providing a direct and efficient way to improve supervision. Results show significant improvements over existing approaches, including large gains on Real-Edit with much less training data and a smaller model.

Text-to-3D generation has grown rapidly but remains difficult to control precisely, especially with multiple objects. We present LucidDreaming, a pipeline enabling spatial and numerical control over 3D generation from textual commands or 3D bounding boxes. We show that LLMs possess 3D spatial awareness and can translate textual 3D information into precise 3D bounding boxes. Using bounding boxes, we propose clipped ray sampling and object-centric density blob bias to generate objects aligned with the boxes. The method adapts across score-distillation-based 3D generation frameworks and can insert objects into an existing NeRF scene. We also provide a dataset of prompts with 3D bounding boxes to benchmark spatial controllability. Experiments demonstrate improved object placement precision and fidelity while maintaining usability for non-experts.

ControlNet improves controllability in text-to-image diffusion models via image-based conditional controls, but aligning generated images with controls remains challenging. We propose ControlNet++, which explicitly optimizes pixel-level cycle consistency between generated images and conditional controls. For an input control, a pretrained reward model extracts the corresponding condition from generated images, and we optimize consistency between the input control and extracted condition. To avoid the time/memory costs of multi-step sampling gradients, we introduce an efficient reward strategy that perturbs input images by adding noise and uses single-step denoised images for reward fine-tuning. Extensive experiments show significant improvements in controllability across conditions such as segmentation masks, edges, and depth maps.

3D Gaussian Splatting enables real-time rendering with high appearance fidelity but can suffer from multi-view inconsistencies that harm geometric accuracy. 2D Gaussian Splatting enforces multi-view consistency but may compromise texture details. We propose Exchangeable Gaussian Splatting (EGGS), a hybrid representation integrating 2D and 3D Gaussians to balance appearance and geometry. EGGS introduces Hybrid Gaussian Rasterization for unified rendering, Adaptive Type Exchange for dynamic adaptation between 2D and 3D Gaussians, and Frequency-Decoupled Optimization to exploit strengths of each representation. A CUDA-accelerated implementation supports efficient training and inference. Experiments demonstrate improved rendering quality, geometric accuracy, and efficiency.

Multi-modal 3D scene understanding is vital for robotics, autonomous driving, and AR/VR, but existing approaches often naively embed semantic representations into reconstruction, leading to poor semantic rasterization for challenging materials and over-fitting to color. We propose a 3D vision-language Gaussian splatting model that better balances visual and semantic modalities with emphasis on language representation learning. A novel cross-modal rasterizer uses modality fusion with a smoothed semantic indicator for improved semantic rasterization, and camera-view blending improves semantic consistency between observed and synthesized views. Experiments show state-of-the-art open-vocabulary semantic segmentation, surpassing existing methods by a significant margin.

We introduce 3DEgo to synthesize photorealistic 3D scenes from monocular videos guided by text prompts. Conventional methods require a multi-stage pipeline using SfM (e.g., COLMAP), initializing a 3D model with unedited images, and iteratively updating the dataset with edited images. 3DEgo streamlines this into a single-stage workflow by removing COLMAP reliance and eliminating initialization cost. We edit video frames with a diffusion model prior to 3D scene creation using a noise blender module to improve multi-view editing consistency without additional training. Then 3D Gaussian Splatting builds the scene from consistent edited frames. Evaluations show strong editing precision, speed, and adaptability across multiple datasets.

Editing neural fields is difficult due to implicit geometry/texture encoding from multi-view inputs. We introduce LatentEditor for precise, locally controlled editing of neural fields using text prompts. We embed real-world scenes into a diffusion latent space, enabling a faster and more adaptable NeRF backbone for editing. A delta score computes a 2D mask in latent space to guide local modifications while preserving irrelevant regions. Our pixel-level scoring leverages InstructPix2Pix by measuring conditional vs. unconditional noise prediction disparity in latent space. Edited latents are iteratively updated to achieve 3D local editing. The method achieves faster editing and superior quality on multiple benchmark 3D datasets.