Abstract:

Objective In the era of 5G Internet communication, virtual try-on technology not only enriches the online fashion consumption experience but also drives the advancement of digital fashion. However, existing generative 3-D virtual try-on models often lack adequate depth features, leading to issues such as stereo distortion and reduced accuracy. In this study, a high precision 3-D virtual try-on model called MV-3DVTON was introduced, which is based on a generative diffusion architecture.

Method A cross-attention mechanism was integrated into the generative diffusion architecture to facilitate the processing of semantic relationships between two information sequences: the features of the fitting image from the frontal view and the features of the target human body's position from the back view. The integration aimed to generate multi-view fitting images. A multi-view depth encoder-decoder network was employed to extract high-precision depth information from various perspectives. Subsequently, the generated multi-view information was consolidated into a 3-D colored point cloud, which, through Poisson reconstruction, enabled the realization of a 360° viewable, lifelike 3-D virtual try-on system.

Results In comparison to CASD and ADGAN, MV-3DVTON exhibits superiority across all metrics. Notably, MV-3DVTON achieved a 22.96% reduction in LPIPS (perceptual image patch similarity) and a 12.08% decrease in FID (fréchet inception distance) compared to the CASD model. This decrease signifies that the generated images better align with human perception, resulting in a more realistic image effect with higher distribution similarity. In order to visually demonstrate the model's generalization across diverse application scenarios, comparison results of multi-view image generation was presented for different genders across four clothing virtual fitting scenarios which are solid color, striped, checkered, and patterned. The generated images by MV-3DVTON exhibited increased similarity with real images in various test scenarios, indicating enhanced generalization capability and the ability to produce more realistic multi-view fitting images. Contrasted with M3D-VTON, MV-3DVTON demonstrated superior performance across all error indices. Particularly noteworthy is the significant decrease in the AbsRel (absolute relative error) index by approximately 12.21% and RMSE (the root mean square error) index by about 5%. These decreases highlight the new model's capability to reduce errors between the estimated and real values of depth information, thereby enhancing the accuracy of depth estimation. Moreover, the novel approach leveraged the frontal and back images of the human body as inputs to the model. With the ability to predict and integrate depth information from multiple viewpoints, the new model was successful in capturing finer depth details compared to existing methods that rely solely on the monocular frontal view's mirror image. By examining the depth maps generated by MV-3DVTON and M3D-VTON from the frontal and back views, it is evident that M3D-VTON exhibits inaccuracies in delineating human body and garment boundaries. Conversely, the depth maps generated by the new model offer clearer boundaries of various human body structures, with enhanced realism in details. Furthermore, the comparison results of selected front, side, and back effects demonstrate MV-3DVTON's ability to generate multi-view virtual fitting effects effectively. This underscores that the predicted multi-view human body depth features are more accurate, providing a comprehensive, multi-view encompassing, and high-precision 3-D virtual fitting experience.

Conclusion A thorough qualitative and quantitative evaluation of the MV-3DVTON method was proposed. The statistical metrics from the test dataset demonstrate that the new model, along with the system of multi-view generation diffusion incorporating a cross-attention mechanism proposed by MV-3DVTON, achieves high-precision generation, accurate prediction estimation, and effective fusion of multi-view depth feature information within a point cloud framework. The new model effectively addresses the challenges associated with inadequate depth information and low accuracy in 3-D feature extraction encountered in conventional monocular mirroring methods. It offers a superior 3-D virtual fitting experience with multiple viewpoints, accommodating various human body postures and clothing textures. This innovative model boasts user-friendly interaction and holds promising prospects for applications within the digital fashion domain.

Key words: 3-D virtual try-on, generation diffusion, cross-attention, multi-view generation, point cloud reconstruction