Neural-ABC: Neural Parametric Models for Articulated Body with Clothes

Honghu Chen     Yuxin Yao          Juyong Zhang
University of Science and Technology of China    

Neural-ABC is a neural implicit parametric model with latent spaces of human identity, clothing, shape and pose. It can generate various human identities and different clothes. The clothed human body can deform into different body shapes and poses.

Abstract

In this paper, we introduce Neural-ABC, a novel parametric model based on neural implicit functions that can represent clothed human bodies with disentangled latent spaces for identity, clothing, shape, and pose. Traditional mesh-based representations struggle to represent articulated bodies with clothes due to the diversity of human body shapes and clothing styles, as well as the complexity of poses. Our proposed model provides a unified framework for parametric modeling, which can represent the identity, clothing, shape and pose of the clothed human body. Our proposed approach utilizes the power of neural implicit functions as the underlying representation and integrates well-designed structures to meet the necessary requirements. Specifically, we represent the underlying body as a signed distance function and clothing as an unsigned distance function, and they can be uniformly represented as unsigned distance fields. Different types of clothing do not require predefined topological structures or classifications, and can follow changes in the underlying body to fit the body. Additionally, we construct poses using a controllable articulated structure. The model is trained on both open and newly constructed datasets, and our decoupling strategy is carefully designed to ensure optimal performance. Our model excels at disentangling clothing and identity in different shape and poses while preserving the style of the clothing. We demonstrate that Neural-ABC fits new observations of different types of clothing. Compared to other state-of-the-art parametric models, Neural-ABC demonstrates powerful advantages in the reconstruction of clothed human bodies, as evidenced by fitting raw scans, depth maps and images.

Method

Overview of our Neural-ABC pipeline. Given query points and latent codes, our model can generate the corresponding implicit representation of the human bodies by three modules.

DressUp Dataset

Synthesis process of DressUp. Firstly, we prepare sewing patterns and virtual models. Then align the sewing pattern to the model. Afterwards, wear clothes through physical simulation. Finally, morph the model wearing clothes to the specified poses.

We prepare a large number of sewing patterns and wear them on models to build the dataset.

Disentangled Generation

Our method can disentangle generation for identity, clothing, shape and pose.

By modifying the four latent codes of identity, shape, clothing, and pose, corresponding clothing body geometry can be generated. Our four attributes are completely decoupled.s

Latent Space Interpolation

Our latent spaces of identity, clothing, shape and pose can be interpolated to obtain continuous generation results. This suggests the continuity of our model.

For designated identities, clothing can be changed smoothly, by only modifying the clothing code.

For designated clothed identity, shape can be changed smoothly, by modifying the identity code.

By modifying the identity code, the geometry of the underlying human body can be edited.

Fitting and editing

We demonstrate the fitting results of Neural-ABC and the editing results after fitting.

Fitting Monocular Depth Sequences

We show the results of fitting monocular depth Sequences.

Deformation Refinement

The dressed human body geometry obtained from our model can be enhanced with finer clothing details by incorporating additional deformation modules. For the reconstruction results, we can employ non-rigid registration to obtain finer clothing wrinkles.

Physics-based Animation

The body and clothing fitted using Neural-ABC can be employed with professional animation techniques to generate motion sequences that include intricate wrinkles.

BibTeX

If you find our paper useful for your work please cite:


      @article{Chen2024NeuralABC,
         author = {Honghu Chen, Yuxin Yao, and Juyong Zhang}, 
         title = {Neural-ABC: Neural Parametric Models for Articulated Body with Clothes}, 
         journal={IEEE Transactions on Visualization and Computer Graphics},
         year={2024},
         publisher={IEEE}}
    

Acknowledgements

This research was partially supported by the National Natural Science Foundation of China (No.62122071, No.62272433), and the Fundamental Research Funds for the Central Universities (No. WK3470000021).