TeCH: Text-guided Reconstruction of Lifelike Clothed Humans
"TeCH: Text-guided Reconstruction of Lifelike Clothed Humans" by Huang et al. introduces a new method for reconstructing 3D humans from a single image. The paper addresses the challenge of accurately restoring the "unseen regions" of a clothed human with high-level details, which has been an unsolved problem in the field.
The authors propose a novel approach called TeCH, which leverages the power of foundation models to reconstruct 3D humans. The method uses descriptive text prompts, such as garments, colors, and hairstyles, which are automatically generated via a garment parsing model and Visual Question Answering (VQA). Additionally, TeCH employs a personalized fine-tuned Text-to-Image diffusion model (T2I) to learn the "indescribable" appearance of the individual.
To represent high-resolution 3D clothed humans at an affordable cost, the authors propose a hybrid 3D representation based on DMTet. This representation consists of an explicit body shape grid and an implicit distance field. The geometry and texture of the 3D humans are optimized through multi-view Score Distillation Sampling (SDS) and reconstruction losses based on the original observation.
The results of the paper show that TeCH produces high-fidelity 3D clothed humans with consistent and delicate texture, as well as detailed full-body geometry. Quantitative and qualitative experiments demonstrate that TeCH outperforms state-of-the-art methods in terms of reconstruction accuracy and rendering quality.
In summary, "TeCH: Text-guided Reconstruction of Lifelike Clothed Humans" by Huang et al. presents a novel approach for reconstructing 3D humans from a single image using descriptive text prompts and a personalized fine-tuned Text-to-Image diffusion model. The method produces high-quality results and outperforms existing methods. Check out the full paper here: https://arxiv.org/abs/2308.08545
Dual-Stream Diffusion Net for Text-to-Video Generation
"Dual-Stream Diffusion Net for Text-to-Video Generation" by Liu et al. introduces a new method for generating videos from text descriptions. The paper addresses the challenge of producing consistent and smooth videos with fewer flickers and artifacts, which has been a bottleneck in the field of text-to-video generation.
The authors propose a novel approach called Dual-Stream Diffusion Net (DSDN), which improves the consistency of content variations in generating videos. The method uses two diffusion streams, video content and motion branches, which run separately in their private spaces to produce personalized video variations and content. These two streams are aligned between the content and motion domains through a cross-transformer interaction module, which benefits the smoothness of generated videos. Additionally, the authors introduce a motion decomposer and combiner to facilitate the operation on video motion.
The results of the paper show that DSDN produces amazing continuous videos with fewer flickers. Qualitative and quantitative experiments demonstrate that the method outperforms existing methods in terms of consistency and smoothness of generated videos.
In summary, "Dual-Stream Diffusion Net for Text-to-Video Generation" by Liu et al. presents a novel approach for generating videos from text descriptions using two diffusion streams and a cross-transformer interaction module. The method produces high-quality results and outperforms existing methods. Check out the full paper here: https://arxiv.org/pdf/2308.08316.pdf
DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory
"DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory" by Yin et al. introduces a new method for generating videos with fine-grained control. The paper addresses the challenge of achieving fine-grained control in videos, which has been a limitation in existing works that focus on either text, image, or trajectory-based control.
The authors propose a novel approach called DragNUWA, an open-domain diffusion-based video generation model. To tackle the issue of insufficient control granularity in existing works, the authors simultaneously introduce text, image, and trajectory information to provide fine-grained control over video content from semantic, spatial, and temporal perspectives. To resolve the problem of limited open-domain trajectory control in current research, the authors propose trajectory modeling with three aspects: a Trajectory Sampler (TS) to enable open-domain control of arbitrary trajectories, a Multiscale Fusion (MF) to control trajectories in different granularities, and an Adaptive Training (AT) strategy to generate consistent videos following trajectories.
The results of the paper show that DragNUWA produces high-quality videos with fine-grained control. Experiments demonstrate the effectiveness of DragNUWA and its superior performance in fine-grained control in video generation.
In summary, "DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory" by Yin et al. presents a novel approach for generating videos with fine-grained control using text, image, and trajectory information. The method produces high-quality results and outperforms existing methods. Check out the full paper here: https://arxiv.org/pdf/2308.08089.pdf