纺织学报 ›› 2023, Vol. 44 ›› Issue (03): 176-186.doi: 10.13475/j.fzxb.20211106611

• 服装工程 • 上一篇    下一篇

基于多模态融合的个性化服装搭配推荐

刘军平1,2,3, 张伏红1, 胡新荣1,2,3(), 彭涛1,2,3, 李丽1,2,3, 朱强1,2,3, 张俊杰1,2,3   

  1. 1.武汉纺织大学 计算机与人工智能学院, 湖北 武汉 430200
    2.纺织服装智能化湖北省工程研究中心, 湖北 武汉 430200
    3.湖北省服装信息化工程技术研究中心, 湖北 武汉 430200
  • 收稿日期:2021-11-12 修回日期:2022-10-23 出版日期:2023-03-15 发布日期:2023-04-14
  • 通讯作者: 胡新荣(1973—),女,教授,博士。主要研究方向为图形图像处理、计算机视觉。E-mail:hxr@wtu.edu.cn。
  • 作者简介:刘军平(1979—),男,副教授,博士。主要研究方向为工业大数据、人工智能。
  • 基金资助:
    湖北省高等学校优秀中青年科技创新团队计划项目(T201807);湖北省教育厅科学研究计划重点项目(D20191708)

Personalized clothing matching recommendation based on multi-modal fusion

LIU Junping1,2,3, ZHANG Fuhong1, HU Xinrong1,2,3(), PENG Tao1,2,3, LI Li1,2,3, ZHU Qiang1,2,3, ZHANG Junjie1,2,3   

  1. 1. College of Computer Science and Artificial Intelligence, Wuhan Textile University, Wuhan, Hubei 430200, China
    2. Hubei Provincial Engineering Research Center for Intelligent Textile and Fashion, Wuhan Textile University, Wuhan, Hubei 430200, China
    3. Engineering Research Center of Hubei Province for Clothing Information, Wuhan Textile University, Wuhan, Hubei 430200, China
  • Received:2021-11-12 Revised:2022-10-23 Published:2023-03-15 Online:2023-04-14

RichHTML

24

PDF (PC)

133

摘要：

为提高服装的匹配度且实现高精度推荐,从而满足消费者对个性化服装搭配推荐的巨大需求,研究了从服装颜色到类别的高度非线性复杂属性交互,并以服装搭配的匹配度量化标准为基础,构建了单品潜在特征表示空间的嵌入模型,通过构建融合多模态信息的矩阵分解框架模型,进一步分析了现有多模态特征融合算法的不足,刻画了不同用户的服装风格偏好,通过特征提取、多模态特征融合、匹配度计算等手段建立个性化服装搭配方案。实验结果表明:该模型计算出的服装匹配度达到了0.81,相较于传统方法提高了1.25%,实现了更高准确度和推荐精度的个性化服装推荐。

关键词: 服装搭配, 个性化推荐, 多模态, 特征提取, 特征融合, 匹配度

Abstract:

Objective In the context of fast fashion, most consumers do not have the keen insight of professional designers on fashion clothing matching, which leads to their capability of quickly selecting a set of appropriate, harmonious and suitable clothing from a large number of clothing. In order to better improve users' online shopping experience and help them accurately express their unique personality characteristics, professional identity, status and other image positioning to the outside world, this paper aims to achieve high-precision recommendation by improving the clothing matching degree, so as to meet the huge demand of consumers for personalized clothing matching recommendation.

Method By studying the highly nonlinear complex attribute interaction from clothing color to category, and based on the quantitative standard of matching degree of clothing matching, an embedded model of the potential feature representation space of an item was built. By building a matrix decomposition framework model that integrates multimodal information, the shortcomings of existing multimodal feature fusion algorithms were further analyzed, and the clothing style preferences of different users were depicted. Through feature extraction, multimodal feature fusion match degree was calculated and other operations were carried out to establish personalized clothing matching scheme.

Results PCMF (personalized clothing matching recommendation based on multi-modal fusion) with some conventional clothing matching methods were qualitatively compared. Compared with all baselines, the clothing matching degree calculated by this model reached 0.81, which is 1.25% higher than the AUC (area under curve) value of conventional methods (Tab.2). It is confirmed that the transposition fusion method of text features and visual features used in PCMF improves the correlation between features, making the presentation of individual style more accurate. In order to compare the difference of contribution of different modal information to the matching degree of PCMF modeled clothing, experiments under three different modal combinations were conducted, i.e. PCMF-T (only exploring the text information of items), PCMF-V (only exploring the visual information of items), and PCMF-TV (exploring the visual and text information of items) (Tab.3). The AUC value of PCMF-T reached 0.775, higher than that of PCMF-V (0.763), indicating that the text information of the piece can more succinctly summarize the key features of the piece, such as patterns, materials and brands. PCMF-TV shows better performance than PCMF-T and PCMF-V, which indicates the necessity of combining multi-modal information of items, and verifies the effectiveness of adding user factors to the general clothing matching modeling to make personalized clothing matching recommendation. In order to effectively evaluate the practical application of the PCMF model, the PCMF model was deployed in the complementary item retrieval task (Fig.5). It is demonstrated that the PCMF model can complete the personalized clothing matching recommendation task according to the user's preferences. In addition, the MRR (mean reciprocal rank) measurement method was used as the evaluation index to further evaluate the model. PCMF performs better than other models regardless of the number of clothing candidates (Fig.6).

Conclusion Through the combination of IGCM (item-item general clothing match modeling) and UPCM (user-item personalized clothing match modeling), a personalized clothing matching recommendation model based on multi-modal fusion is constructed, which facilitates high-precision personalized clothing matching recommendation. Specifically, the purpose is to match a lower garment that not only has a good match with a given user's top, but also meets the user's taste. In general, the research results show the necessity and effectiveness of combining visual and text modal information and introducing user factors in personalized clothing matching recommendation and the practical application value of PCMF in real scenes is verified. In the future, the clothing matching recommendation problem of two examples will be transformed into a multi-instance learning problem to provide users with personalized package recommendation including shoes and accessories.

Key words: complementary clothing matching, personalized recommendation, multi-modal, feature extraction, feature fusion, matching degree

中图分类号: 

  • TS941.26

图1

基于多模态融合的个性化服装搭配推荐模型PCMF结构"

表1

数据集IQON 3000的数据统计"

单品种类 数量/件
外套 35 765
上衣 119 895
下衣 77 813
连衣裙 25 816
鞋子 106 598
配饰 306 448

图2

单品的视觉特征抽取过程"

图3

单品的文本特征抽取过程"

表2

不同方法下AUC的性能比较"

方法 AUC值
POP 0. 421
RAND 0. 501
Bi-LSTM 0. 661
BPR-DAE 0. 699
VTBPR 0. 801
GP-BPR 0. 802
PCMF 0. 814

图4

PCMF与 GP-BPR 的AUC性能比较"

表3

不同模态下AUC的性能比较"

方法 AUC值
PCMF-V 0. 763
PCMF-T 0. 775
PCMF-TV 0. 814

图5

下衣推荐排行榜"

图6

6种基准方法在T范围下的MRR性能"

[1] AMED I, BALCHANDANI A, BERG A, et al. The state of fashion 2022: global gains mask recovery pains[EB/OL]. [2022-10-01]. https://www.mckinsey.com/industries/retail/our-insights/the-state-of-fashion.
[2] 朱伟明, 卫杨红. 不同情景下服装个性化定制体验价值差异研究[J]. 纺织学报, 2018, 39(10): 115-119.
ZHU Weming, WEI Yanghong. Experiential value differences of clothing personalized customization under different situations[J]. Journal of Textile Research, 2018, 39(10): 115-119.
[3] CHEN Huiyuan, LIN Yusan, WANG Fei, et al. Tops, bottoms, and shoes: building capsule wardrobes via cross-attention tensor network[C]// Fifteenth ACM Conference on Recommender Systems. New York: Association for Computing Machinery, 2021: 453-462.
[4] DONG Xue, SONG Xuemeng, FENG Fuli, et al. Personalized capsule wardrobe creation with garment and user modeling[C]// In Proceedings of the 27th ACM International Conference on Multimedia (MM'19). New York:Association for Computing Machinery, 2019: 302-310.
[5] CHEN Wen, HUANG Pipei, XU Jiaming, et al. POG: Personalized outfit generation for fashion recommendation at alibaba ifashion[C]// Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD'19). New York, USA: ACM Press, 2019: 2662-2670.
[6] LIN Yusan, MOOSAEI M, YANG Hao. Outfitnet: fashion outfit recommendation with attention-based multiple instance learning[C]// Proceedings of The Web Conference 2020. New York, USA: ACM Press, 2015: 77-87.
[7] VASILEVA M I, PLUMMER B A, DUSAD K, et al. Learning type-aware embeddings for fashion compatibility[C]// Proceedings of the European Conference on Computer Vision. Berli, Germany: Springer, 2018: 390-405.
[8] CHEN Xu, ZHANG Yongfeng, XU Hongteng, et al. Visually explainable recommendation[J]. Computer Science, 2018. DOI: 10.48550/arXiv.1801.10288.
doi: 10.48550/arXiv.1801.10288
[9] HIDAYATI S C, HSU C C, CHANG Y T, et al. What dress fits me best?: fashion recommendation on the clothing style for personal body shape[C]// Proceedings of the 26th ACM International Conference on Multimedia. New York, USA: ACM Press, 2018: 438-446.
[10] 杨争妍, 薛文良, 张传雄, 等. 基于生成式对抗网络的用户下衣搭配推荐[J]. 纺织学报, 2021, 42(7): 164-168.
YANG Zhengyan, XUE Wenliang, ZHANG Chuanxiong, et al. Recommendations for user's bottoms matching based on generative adversarial networks[J]. Journal of Textile Research, 2021, 42(7): 164-168.
[11] MCAULEY J, TARGETT C, SHI Q, et al. Image-based recommendations on styles and substitutes[C]// Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York, USA: ACM Press, 2015: 43-52.
[12] VEIT A, KOVACS B, BELl S, et al. Learning visual clothing style with heterogeneous dyadic co-occurrences[C]// Proceedings of the IEEE International Conference on Computer Vision. Berli, Germany: Springer, 2015: 4642-4650.
[13] CHEN Long, HE Yuhang. Dress fashionably: learn fashion collocation with deep mixed-category metric learning[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2018, 32(1): 2103-2110.
[14] HE R, PACKER C, MCAULEY J. Learning compatibility across categories for heterogeneous item recommendation[C]// 2016 IEEE 16th International Conference on Data Mining. Barcelona, Spain:IEEE, 2017: 937-942.
[15] SHIH Y S, CHANG K Y, LIN H T, et al. Compatibility family learning for item recommendation and generation[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2018, 32(1): 2403-2410.
[16] CHEN L C, PAPANDREOU G, KOKKINOS I, et al. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs[C]// IEEE Transactions on Pattern Analysis and Machine Intelligence. Piscataway, NJ: IEEE, 2018: 834-848.
[17] WANG Zheng, BAI Xiang, YE Mang, et al. Incremental deep hidden attribute learning[C]// Proceedings of the 26th ACM International Conference on Multimedia. New York, USA: ACM Press, 2018: 72-80.
[18] 刘华玲, 马俊, 张国祥. 基于深度学习的内容推荐算法研究综述[J]. 计算机工程, 2021, 47(7): 1-12.
LIU Hualing, MA Jun, ZHANG Guoxiang. Review of studies on deep learning-based content recommendation algorithms[J]. Computer Engineering, 2021, 47(7): 1-12.
doi: 10.1063/1.31437
[19] HE R, LIN C, WANG J, et al. Sherlock: sparse hierarchical embeddings for visually-aware one-class collaborative filtering[C]// In International Joint Conference on Artificial Intelligence. New York, USA: AAAI Press, 2016: 3740-3746.
[20] 张艺豪, 李梁, 赵清华, 等. 基于全局相似度的社交网络个性化推荐算法[J]. 计算机工程, 2018, 44(10): 204-208.
ZHANG Yihao, LI Liang, ZHAO Qinghua, et al. Personalized recommendation algorithm of social network based on global similarity[J]. Computer Engineering, 2018, 44(10): 204-208.
[21] BRILL M, ELKIND E, ENDRISS U, et al. Pairwise diffusion of preference rankings in social networks[C]// Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence. New York, USA: AAAI Press, 2016: 130-136.
[22] 张卓, 丛洪莲, 蒋高明, 等. 基于交互式遗传算法的Polo衫快速款式推荐系统[J]. 纺织学报, 2021, 42(1): 138-144.
ZHANG Zhuo, CONG Honglian, JIANG Gaoming, et al. Polo shirt rapid style recommendation system based on interactive genetic algorithm[J]. Journal of Textile Research, 2021, 42(1): 138-144.
[23] 甘美辰, 李敏. 女装搭配推荐系统的设计与实现[J]. 纺织学报, 2020, 41(10): 122-131.
GAN Meichen, LI Min. Design and realization of a collocation recommendation system for women's clothing[J]. Journal of Textile Research, 2020, 41(10): 122-131.
[24] HE Xiangnan, ZHANG Hanwang, KAN Min-Yen, et al. Fast matrix factorization for online recommendation with implicit feedback[C]// Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval. New York, USA: ACM Press, 2016: 549-558.
[25] KOREN Y, BELL R, VOLINSKY C. Matrix factorization techniques for recommender systems[J]. Computer, 2009, 42(8): 30-37.
[26] WANG Xiang, HE Xiangnan, WANG Meng, et al. Neural graph collaborative filtering[C]// Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. New York, USA: ACM Press, 2019: 165-174.
[27] RENDLE S, FREUDENTHALER C, GANTNER Z, et al. BPR: bayesian personalized ranking from implicit feedback[C]// Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence. Virginia, USA: AUAI Press, 2009: 452-461.
[28] HE R, MCAULEY J. VBPR: visual bayesian personalized ranking from implicit feedback[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2016, 30(1): 144-150.
[29] YU Wenhui, ZHANG Huidi, HE Xiangnan, et al. Aesthetic-based clothing recommendation[C]// Proceedings of the 2018 World Wide Web Conference. Lyon, France:IW3C2, 2018: 649-658.
[30] SONG Xuemeng, FENG Fuli, LIU Jinhuan, et al. Neurostylist: neural compatibility modeling for clothing matching[C]// Proceedings of the 25th ACM International Conference on Multimedia. New York, USA: ACM Press, 2017: 753-761.
[31] 贾俊杰, 刘鹏涛, 陈旺虎. 融合社交信息的矩阵分解改进推荐算法[J]. 计算机工程, 2021, 47(9): 97-105.
JIA Junjie, LIU Pengtao, CHEN Wanghu. Improved matrix factorization algorithm using social information for recommendation[J]. Computer Engineering, 2021, 47(9): 97-105.
[32] HU Yifan, KOREN Yehuda, VOLINSKY Chris. Collaborative filtering for implicit feedback data-sets[C]// Proceedings of the 2008 Eighth IEEE International Conference on Data Mining. New York, USA: IEEE Computer Society, 2008: 263-272.
[33] PAN Rong, ZHOU Yunhong, CAO Bin, et al. One-class collaborative filtering[C]// Proceedings of the 2008 Eighth IEEE International Conference on Data Mining. New York, USA: IEEE Computer Society, 2008: 502-511.
[34] REN Shaoqing, HE Kaiming, GIRSHICK Ross. Faster r-cnn: towards real-time object detection with region proposal networks[J]. IEEE Trans Pattern Anal Mach Intell, 2017, 39(6): 1137-1149.
doi: 10.1109/TPAMI.2016.2577031
[35] 孙洁, 丁笑君, 杜磊, 等. 基于卷积神经网络的织物图像特征提取与检索研究进展[J]. 纺织学报, 2019, 40(12): 146-151.
SUN Jie, DING Xiaojun, DU Lei, et al. Research progress of fabric image feature extraction and retrieval based on convolutional neural network[J]. Journal of Textile Research, 2019, 40(12): 146-151.
[36] 王柳程, 欧阳城添, 梁文. 基于改进特征金字塔网络的人体姿态估计[J]. 计算机工程, 2021, 47(8): 251-259,270.
WANG Liucheng, OUYANG Chengtian, LIANG Wen. Human pose estimation based on improved pyramid feature network[J]. Computer Engineering, 2021, 47(8): 251-259,270.
[37] 音松, 陈雪云, 贝学宇. 改进Mask RCNN算法及其在行人实例分割中的应用[J]. 计算机工程, 2021, 47(6): 271-276,283.
YIN Song, CHEN Xueyun, BEI Xueyu. Improved mask rcnn algorithm and its application in pedestrian instance segmentation[J]. Computer Engineering, 2021, 47(6): 271-276,283.
doi: 10.1063/1.31441
[38] LIN T, DOLLÁR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York, USA: IEEE, 2017: 936-944.
[39] KIM Y. Convolutional neural networks for sentence classification[C]// Proceedings of the Conference on Empirical Methods in Natural Language Processing. Stroudsburg: ACL, 2014. DOI: 10.48550/arXiv.1408.5882.
doi: 10.48550/arXiv.1408.5882
[40] 喻靖民, 向凌云, 曾道建. 基于Word2vec的自然语言隐写分析方法[J]. 计算机工程, 2019, 45(3): 309-314.
YU Jingmin, XIANG Lingyun, ZENG Daojian. Natural language steganalysis method based on Word2vec[J]. Computer Engineering, 2019, 45(3): 309-314.
[41] HU Zhiting, MA Xuezhe, LIU Zhengzhong. Harnessing deep neural networks with logic rules[C]// Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Stroudsburg: ACL, 2016: 2410-2420.
[42] SEVERYN A, MOSCHITTI A. Twitter sentiment analysis with deep convolutional neural networks[C]// Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York, USA: ACM Press, 2015: 959-962.
[43] ASAHARA M. NWJC2Vec: word embedding dataset from 'ninjal web japanese corpus'[J]. Terminology, 2018, 24(1): 7-22.
[44] KINGMA D P, BA J. Adam: a method for stochastic optimization[J]. Computer Science, 2014. DOI: 10.48550/arxiv.1412.6980.
doi: 10.48550/arxiv.1412.6980
[45] BRADLEY P. The use of the area under the ROC curve in the evaluation of machine learning algorithms[J]. Pattern Recognition, 1997, 30(7): 1145-1159.
doi: 10.1016/S0031-3203(96)00142-2
[46] SONG Xuemeng, HAN Xianjing, LI Yunkai, et al. GP-BPR: personalized compatibility modeling for clothing matching[C]// Proceedings of the 27th ACM International Conference on Multimedia. New York, USA: ACM Press, 2019: 320-328.
[47] HAN Xintong, WU Zuxuan, JIANG Yugang, et al. Learning fashion compatibility with bidirectional LSTMs[C]// Proceedings of the 25th ACM International Conference on Multimedia. New York, USA: ACM Press, 2017: 1078-1086.
[48] HE Xiangnan, LIAO Lizi, ZHANG Hanwang, et al. Neural collaborative filtering[C]// Proceedings of the 26th International Conference on World Wide Web. Republic and Canton of Geneva, CHE:IW3C2, 2017: 173-182.
[49] JIANG Lu, YU S I, MENG Deyu, et al. Fast and accurate content-based semantic search in 100m internet videos[C]// Proceedings of the 23rd ACM International Conference on Multimedia. New York, USA: ACM Press, 2015: 49-58.
[50] YIN Hongzhi, CHEN Hongxu, SUN Xiaoshuai, et al. SPTF: a scalable probabilistic tensor factorization model for semantic-aware behavior prediction[J]. IEEE Transactions on Knowledge and Data Engineering, 2014, 26(7): 1763-1777.
doi: 10.1109/TKDE.2013.168
[1] 马传旭, 张宁, 潘如如. 基于支持向量机的筒子纱纱管品种检测[J]. 纺织学报, 2023, 44(01): 194-200.
[2] 马玲, 吕叶馨, 刘驰, 陈佳珍, 邹奉元. 基于服装属性的女装搭配规则构建[J]. 纺织学报, 2022, 43(08): 161-166.
[3] 杨争妍, 薛文良, 张传雄, 丁亦, 马颜雪. 基于生成式对抗网络的用户下装搭配推荐[J]. 纺织学报, 2021, 42(07): 164-168.
[4] 徐增波, 张玲, 张艳红, 陈桂清. 基于复杂网络提取和支持向量机模型分类的服装领型研究[J]. 纺织学报, 2021, 42(06): 146-152.
[5] 郑晓萍, 刘成霞. 基于激光测量的机织服装起拱性客观评价[J]. 纺织学报, 2021, 42(05): 143-149.
[6] 甘美辰, 李敏. 女装搭配推荐系统的设计与实现[J]. 纺织学报, 2020, 41(10): 122-131.
[7] 夏海浜, 黄鸿云, 丁佐华. 基于迁移学习与支持向量机的服装舒适度评估[J]. 纺织学报, 2020, 41(06): 125-131.
[8] 汪会, 孙洁, 丁笑君, 龙颖, 邹奉元. 织物纹样特征提取与匹配方法比较[J]. 纺织学报, 2020, 41(04): 45-50.
[9] 孙洁, 丁笑君, 杜磊, 李秦曼, 邹奉元. 基于卷积神经网络的织物图像特征提取与检索研究进展[J]. 纺织学报, 2019, 40(12): 146-151.
[10] 朱浩, 丁辉, 尚媛园, 邵珠宏. 多纹理分级融合的织物缺陷检测算法[J]. 纺织学报, 2019, 40(06): 117-124.
[11] 邢文宇, 邓娜, 辛斌杰, 于晨. 基于多特征融合图像分析技术的羊毛与羊绒鉴别[J]. 纺织学报, 2019, 40(03): 146-152.
[12] 赵树志 狄岚 何锐波. 基于改进判别性完整局部二值模式与格分割的织物瑕疵检测方法[J]. 纺织学报, 2018, 39(09): 57-64.
[13] 董蓉 李勃 徐晨. 应用积分图的织物瑕疵检测快速算法[J]. 纺织学报, 2016, 37(11): 141-147.
[14] 曹霞 李岳阳 罗海驰 蒋高明 丛洪莲. 蕾丝花边的改进型纹理特征检索方法[J]. 纺织学报, 2016, 37(06): 142-154.
[15] 黎聪 闫学娜 曾祥忠 梁猛 张莹. 应用一维傅里叶变换的剖幅区自动识别与定位[J]. 纺织学报, 2016, 37(01): 147-151.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
京ICP备14006648号　 京公网安备11010502044800号
版权所有 © 2021 《纺织学报》编辑部
地址： 北京市朝阳区延静里中街3号主楼6层《纺织学报》杂志社　邮政编码：100025　
电话：010-65017711,65917740　传真：010-65016539　Email:fangzhixuebao@vip.126.com
本系统由北京玛格泰克科技发展有限公司设计开发