| Title |
Design and Application of Live Strip Merchandise Recognition System Based on Multimodal Learning |
| DOI |
https://doi.org/10.5573/IEIESPC.2025.14.5.657 |
| Keywords |
Multimodal learning; Live streaming with goods; Goods with goods; Image recognition; ; Recognition system |
| Abstract |
Given the existing problems such as weak interaction ability, low information recognition rate, and poor accuracy of goods identification, a new system of goods identification based on a multi-modal learning method was designed. By integrating commodity visual, voice, text, and other multiple information, a multi-modal information recognition model is built, commodity data is mapped to the model analysis module using modal retrieval, ITC contrast training, ITM interactive training, and ITG weighted training is carried out, and multi-modal commodity information is initially integrated. SDI+HDMI dual-interface encoder was used to encode the commodity graphic frequency information, and TRIOPC-MCAT-2 controller and IC identifier were selected to optimize the hardware equipment, effectively improve the commodity identification computing power and information response rate, and enhance the stability of the system. The multimodal learning model is used to transform the commodity information into 10-dimensional mapping vectors, multimodal coding is carried out according to the model input hierarchy, key features are extracted using SVM classifier, and multimodal feature information fusion vectors of commodities are obtained through interactive guided weighting operation, and commodity recognition is carried out according to the training results. According to the experimental results, given the massive commodity information of multiple categories, the success rate of the multi-modal learning-based live broadcast cargoes identification system designed in this paper for the commodity multi-modal information fusion recognition has reached more than 88%, the recognition time is less than 90ms, and the recognition accuracy rate is higher than 95%, indicating that the system studied in this paper has good recognition performance. The practical application effect is better than the traditional method, which can meet the current needs of live broadcast goods identification. |