3.1. System Implementation of Spatial Sensing by Integrating Smart Helmet Devices and BCIs

With the continuous development of science and technology, people have become more demanding in their interaction with the environment ^[17]. IE spatial perception technology is a kind of cutting-edge technology that has emerged in this context ^[18]. It enables users to interact with computers in a more natural and intuitive way by interacting with virtual environments, thus achieving a more realistic experience feeling ^[19]. Fig. 1 shows common SWDs.

Fig. 1. Common intelligent wearable devices.

Whether it is a smart watch, smart bracelet, smart glasses or smart helmet, they can be connected to smartphones and other devices through various built-in sensors. These can collect the user’s physiological data, behavioral data, etc. and transmit them to smartphones and other devices for processing. This can help users better understand their physical condition, location and other information for a smarter, more convenient life. Wavelet time-frequency analysis is usually used in signal pre-processing to identify and process the signal, in which the low-pass wave filter output equation of the signal is shown in Eq. (1).

In Eq. (1), $g$ is the low-pass filter. $n$ is the signal length. $x[n]$ is the sound signal. Eq. (1) is employed to describe the output of a low-pass filter, with the objective of eliminating high-frequency noise and preserving the low-frequency component of the signal. In the study, the equation can be utilized to distinctly identify and process physiological signals from SWDs, thereby ensuring that subsequent data analysis is based on a purer signal. The output equation of the corresponding high pass filter is shown in Eq. (2).

In Eq. (2), $h$ is the high pass filter. Moreover, $a$ is the order. Eq. (2) is employed for the purpose of removing low-frequency noise and retaining only the high-frequency components of the signal. This can assist in the examination and examination of rapidly transforming signal characteristics when processing data, such as instantaneous alterations in EMG signals. Then the output raw signal is weighted and summed and the equation for weighted sum is shown in Eq. (3).

In Eq. (3), $N$ is the number of filters. Moreover, $b_i$ is the coefficient of the $N$th filter at the $i$th moment. Eq. (3) is employed to assign weights to the filter output, thereby integrating data from multiple sensors. In the process of MSIF, the weighted sum of different signals serves to enhance the reliability and accuracy of the overall data. Where the expression of $b_i$ is shown in Eq. (4).

$k$ in Eq. (4) is the signal filtering. Eq. (4) describes the expression form of the signal after filtering to ensure that the necessary feature information can be retained in the signal processing. Among these wearable smart devices, smart helmets can track the user’s head position and line of sight, as well as recognize the user’s voice commands to achieve a more natural and intuitive interaction. Moreover, BCI technology can realize the control of external devices by decoding the brain activity signals, and control the SWD by intention, thus realizing a more natural and intuitive interaction with the surrounding environment. The reason for focusing on the integration of BCIs and SWDs is to enhance the naturalness of human-computer interaction, strengthen personalized and adaptive capabilities, and solve the problem of action limitations. The functionality of SWDs can be controlled by the user’s brain activity, with electrical signals being collected in real time via electrodes or other sensors attached to the scalp via a BCI. Following the appropriate pre-processing, the collected signal is then analyzed and decoded by a machine learning or signal processing algorithm. The decoded instructions are then transmitted to a SWD. Therefore, in order to improve the IE spatial perception technology, the study integrates the BCI technology and the smart helmet device, so that the user can feel the IE in the virtual environment more realistically. The pre-processing of the EEG signal in the BCI technology is one of the important steps, so the study adopts the spatial filtering algorithm to collect the characteristics of electrode signals, and its algorithm expression is shown in Eq. (5).

In Eq. (5), $V_i^{ORIG}$ is the original peak of the channel. and $m$ is the number of all channels selected. Eq. (5) is used to extract characteristic peaks from the electrode signals, and the key is to effectively identify the instantaneous peaks of brain activity from the processed EEG data. Then wavelet transform has been obtained for its signal feature recognition, the expression of wavelet transform as shown in Eq. (6).

$x(t)$ in Eq. (6) is the signal presented by brain activity or external events. $\psi(t)$ is the wavelet substrate. $a$ and $b$ are the scale and translation variables, respectively. Eq. (6) is employed for the analysis of the instantaneous characteristics and frequency components of the signal, facilitating the effective capture of changes in transient signals. In the research, it enables an in-depth analysis of EMG, including dynamic movement features, and enhances the understanding of the user’s movement and psychological state. The final IE spatial perception system based on BCI and smart helmet is shown in Fig. 2.

Fig. 2. Immersive experience space perception system.

Fig. 2 shows a framework for studying immersive spatial perception systems based on BCIs and smart helmets, in which BCIs play a key role in interpreting electrical signals from the brain directly into commands that can drive interactions in virtual environments. For example, by interpreting the brain’s response to specific visual or auditory stimuli, the BCI can “touch” or “move” virtual objects. A smart helmet is a hardware device that integrates an array of sensors that capture information about the movement and position of the head and transmit it to a processing unit. This allows the user’s movements and observations in the virtual environment to be mapped to the virtual world in real time. The system’s processing unit receives data from the BCI and the smart helmet and converts this data into interactive commands to the virtual environment through advanced signal processing algorithms. To provide a more realistic IE, the system provides immediate feedback, including visual feedback (e.g., 3D images), auditory feedback (e.g., sound effects or music), and haptic feedback (e.g., vibration or temperature changes). Moreover, the user interface enables users to interact with the system, including BCI’s hardware devices, smart helmets, and interfaces for entering commands and receiving feedback. The roadmap for the realization of one of these immersive spatial perception technologies is shown in Fig. 3.

Fig. 3. Immersive experience space perception technology.

In spatial perception technology, BCI can capture electrical signals from the brain, which include but are not limited to visual, auditory, tactile, motor and cognitive. By interpreting these signals, BCI can determine the user’s intention and thus directly translate the user’s thinking into interactive commands to the virtual environment. The smart helmet, on the other hand, captures head movement and position information through built-in sensors and transmits this information to the processing unit in real time. The processing unit determines the user’s perspective and position based on this information, thus mapping the user’s observations in reality to the virtual environment. By doing so, BCI and the smart helmet together create a new, immersive spatial perception experience.

3.2. IMSIF Technology Improvement

With the continuous development of science and technology, people’s reliance on and demand for intelligent devices are increasing ^[20]. Especially in the information age, the acquisition, processing and utilization of information have become the key to various applications ^[21]. SWD and BCI, in addition to their application potential in spatial perception systems, can also satisfy people’s demand for more efficient, convenient and accurate information. In IE, compared with spatial perception, MSIF technology emphasizes more on the accuracy and comprehensiveness of situational perception. Therefore, the study integrates BCI and smart helmet to improve MSIF technology. In which the instantaneous sequence expression of MSIF technology to obtain information energy is shown in Eq. (7).

In Eq. (7), $S_i(t)$ is the original signal. Eq. (7) is employed to delineate the transformation of signal energy, offering a pivotal contribution to the integration of multi-source data, which can elucidate the intensity of the user’s physiological condition within a particular context. Then the average energy of the signal is acquired by the sensor as shown in Eq. (8).

In Eq. (8), $W$ is the window length. $t$ is the time. In the modified MSIF technique the input and processing of joint motion signals is performed through sensors worn on various parts of the body, and the joint angle is calculated as shown in Eq. (9).

In Eq. (9), $a$ and $b$ are the data recorded by different sensors. Eqs. (8) and (9) assist the system in comprehending the user’s body posture and movement patterns. This is achieved by calculating data recorded by multiple sensors in order to obtain the average energy and joint angles. Then it is binarized and the result is shown in Eq. (10).

In Eq. (10), $p$ is the information real-time value. Moreover, $G$ is the judgment threshold. The binary operation of Eq. (10) is capable of extracting key information from complex data during the process of information processing. This is of particular importance for real-time monitoring and decision support. The technology roadmap for the MSI technology improvement process is shown in Fig. 4.

Fig. 4. Emg processing by multi-source information technology.

In the improvement of MSI technology firstly, various physiological parameters of human body, such as heart rate, blood pressure, blood glucose, body temperature, EEG, etc., as well as environmental parameters, such as temperature, humidity, ultraviolet rays, air quality, etc., are acquired by SWD and BCI technology, and the information is acquired in real time by the smart sensors and BCI technology and transmitted to the processing center. In the information processing stage, the acquired data are cleaned, analyzed and processed by algorithms and models, mainly for noise reduction and optimization of Electro-Myographic Signals (EMS). After that, information from different sources and media are fused to produce more comprehensive and accurate information, and finally feedback information in the form of visual, auditory and tactile senses. Where the equation for processing EMS is shown in Eq. (11).

In Eq. (11), $x(t)$ is the signal. $\sigma$ is the scale parameter. $\tau$ is the time parameter. $\psi$ is the basis function. $*$ is the complex conjugate. Eq. (11) processes the EMG signal through the selected basis function, thereby enhancing the recognition rate of signal features. This is a pivotal step in the processes of movement recognition and rehabilitation training. The frequency optimization function is taken in optimizing the features of EMS and its expression is shown in Eq. (12).

In Eq. (12), $p[k]$ is the power spectrum. $k$ is the spectral coefficient. $N$ is the signal length. The frequency optimization of Eq. (12) can improve the effectiveness of target recognition in signal analysis, especially in processing rapidly changing physiological signals, and ensure accuracy. The frequency optimization of Eq. (12) can improve the effectiveness of target recognition in signal analysis, especially in the processing of rapidly changing physiological signals, and ensure accuracy. Based on the above analysis, this study explores the use of SWDs and BCI for data collection. The collected original signal is denoised and feature extracted, and the data is cleaned by filtering and transforming. High-frequency noise is removed using wavelet transforms and filtering of existing signals to improve signal quality, e.g., low-pass and high-pass filtering using Eqs. (1) and (2) to ensure that subsequent analysis are based on good signals. By analyzing the EEG and EMG signals, the features related to the user’s emotional state and action are extracted. In the feature extraction stage, bandpass filters and wavelet transform are used to ensure that important dynamic features in the signal can be captured. The information obtained from different sensors is fused to form a comprehensive information input. In this process, the data must be weighted to improve the accuracy and credibility of the data, and Eq. (11) is used to extract the EMG features, and Eq. (12) is used to optimize the frequency. By optimizing the frequency coefficient, the noise component can be effectively removed, thereby ensuring that the recognition rate can be maintained in the context of rapidly changing physiological signal processing and improving the performance of action recognition.

Then, the extracted data of all parties are entered into the data set and classified. In which the EEG signal acquisition for BCI is performed, the distribution of brain electrodes is shown in Fig. 5.

Fig. 5. Distribution of brain electrodes.

In MSIF technology, brain electrodes are mainly used for signal acquisition, which captures electrical signals from the brain directly. These signals include, but are not limited to, visual, auditory, tactile, motor, and cognitive. Specifically, brain electrodes record the activity of neurons in the brain to obtain information about cognitive, emotional, motor, and other neural activities. In MSIF, these EEG signals can be fused with data from other types of sensors, such as health monitoring data from sensors worn on various parts of the body, various environmental parameters from smart homes, and so on. Through this fusion, a more comprehensive and accurate judgment and understanding of the user’s state and needs can be made from multiple perspectives and levels. The primary objective of the IMSIF technology is to enhance the precision and dependability of data, fortify the comprehensive comprehension of knowledge, and facilitate real-time responsiveness and feedback. By integrating data from disparate sensors, the IMSIF technology can eliminate noise and errors that may be introduced by a single data source, thereby improving the accuracy and reliability of data. The system is capable of forming a more comprehensive understanding of the user’s state through the fusion of multiple data sources. This multidimensional data analysis method facilitates a more comprehensive understanding of the user’s situational perception and psychological state. The IMSIF technology is capable of processing data from disparate sensors, expeditiously generating response commands, monitoring the user’s physiological signals in real time, and adjusting the training plan based on the data analysis results to align with the individual needs of the user.

4.1. Effectiveness Analysis of IMSIF System

In the experimental analysis, experiment used OpenBCI as an open source hardware and software toolbox for EEG acquisition, which is capable of connecting multi-channel EEG sensors to record the electrical activity of the brain in real time and visualize the signals through a graphical interface. The MATLAB Data Acquisition Toolbox makes it easy to read and monitor sensor data in real time. The visualization tool uses Matplotlib to clearly display the perceptual results of different dimensions, the accuracy of action recognition, and the effect of emotion classification. After the data are obtained, the signal is divided according to the time window, and the data in different time windows are processed, and the time-frequency features are extracted by using wavelet transform or Fourier transform. The data from different sources are normalized, the basic features of the signal are calculated, and the features are standardized to facilitate model training and testing.

Before the experiment, the BCI competition data are selected, and five experimenters are selected for the acquisition of EEG data ^[22]. The acquisition location is the forehead of the brain, the waveform is 0-100 Hz, the emission frequency of the signal is 2 s, the data precision is set to 5 bits, and the hardware response time is less than 2 ms. The subjects’ emotional data is monitored and collected through physiological signals such as heart rate variability and skin electrical response. The emotional data is about 2,440 pieces of data in total. The head movement data set is collected by motion sensors, including the angle changes of movements such as head up, head down, and head turning, and then used to analyze the user’s perspective and focus shift. The basic information of the five subjects is as follows: the average age is 37.2 years old, the oldest age is 45 years old, and the youngest age is 32 years old. 3 males and 2 females. All subjects has no major illnesses or physical impairments, and no major surgeries is recorded in the last six months. 3 subjects exercises regularly and 2 subjects exercises occasionally. In the experiment, the study selects specific subjects for assessment tests, including mental state assessment, movement recognition, physiological data monitoring, experience in a virtual environment, and MSIF. The fundamental principles of mental state assessment are primarily based on psychological and neuro-scientific theories. The use of physiological monitoring technology, such as EEG, enables the observation of the electrical activity of the brain. This observation can provide insight into the emotional and cognitive status of individuals. Based on the principles of human-computer interaction and sports science, movement recognition helps users interact more naturally with the virtual environment and provides important feedback information for sports training and rehabilitation. Physiological data monitoring is based on biomedical and psychophysiological principles, and physiological data is often closely related to the mental state and behavior of an individual. The experience in the virtual environment is to study the integration effect of SWDs and BCI technology. The choice of MSIF is to obtain more comprehensive user status information and improve decision support by integrating information from different sources. Then, the system is tested, and the four psychological states of the experimenters are selected for the analysis of the band components of the EEG waves, and the results of the analysis are shown in Table 1. The analysis results are shown in Table 1.

Table 1 shows the characteristics of each band of the EEG of the test subjects in different mental states, when which the frequency of Alpha in the occipital and parietal parts is 8-17 Hz when the subjects are in the awake, quiet, and closed-eye state, the frequency of Beta in the frontal lobe of the anterior central round is 13-31 Hz when the subjects are in the excitatory state of the cortex, and the frequency of Parietal and Temporal lobe of the parietal lobe and temporal lobe of the temporal lobe is 4-37 Hz when the subjects are in the sleep state. Theta’s frequency is 4-37 Hz, and Delta’s frequency in the frontal and temporal lobes is 0.5-200 Hz when the subject is in a deep sleep state. Its test results match the actual situation, which shows that the system of this research can effectively analyze the psychological state of the subjects. Then the subjects are immersed in the virtual world experience using the BIC competition data and smart helmets, after which the six dimensions of the sense of presence are utilized to score the immersive perception results of the subjects. The results are shown in Fig. 6.

Fig. 6. Immersion perception scores of subjects in different dimensions.

In Fig. 6, different dimensions correspond to circles of different colors, and the size of the circle indicates the score size of the subject’s corresponding dimension. The larger the circle, the larger the score of the corresponding dimension. Subject 1 has the highest ratings for perception and attention, and subject 2 has higher ratings for self-awareness, social interaction, and cognition in the experiment. Subject 3 has a higher level of overall cognition and perception of the scenario. Subject 4 scores higher in perception and emotional response while the last subject scores higher in cognitive level. The results show that the proposed method can understand the user’s emotional state in the IE by analyzing different dimensions. It can be customized according to the user experience to improve the adaptability and flexibility of the system. Next, the accuracy of response movement recognition of subjects wearing smart helmets is tested to verify the sensitivity of the system to human perception. A total of three actions, head down and head up and head turn, are designed in the virtual scenario in the experiment, and the final test results are shown in Fig. 7.

Fig. 7. Accuracy analysis of subject’s head movement recognition.

Fig. 7 illustrates the subject’s head movement recognition accuracy. The box figure represents the average recognition accuracy of the movement, while the upper and lower lines represent the standard deviation of the average accuracy. Among them, the average accuracy rate of head-up movement recognition is 95.6%, the average accuracy rate of head-down movement recognition is 91.3%, and the average accuracy rate of head-down movement recognition is 82.2%. Moreover, the difference between the recognition accuracy of each subject is within 5%. The findings demonstrate that the proposed method is capable of adjusting the feedback in a timely manner within a dynamic environment. It is also able to swiftly and accurately comprehend the user’s intention and effectively translate the user’s natural actions into a prompt system response, thereby enhancing the sense of immersion. Then the brainwave characterization of BCI is used to analyze the user’s emotional and mental state to verify the immersive perception of the research user, and the verification method is analyzed by confusion matrix, and the analysis results are shown in Fig. 8.

Fig. 8. Confusion matrix analysis of emotion recognition model.

In the emotion recognition of the subjects in the scenario experience, in which the highest accuracy of emotion recognition is happy, with an accuracy of 97.3%, followed by the emotions of fear and anger, with accuracy of 96.8% and 95.4%, respectively, and the predictive accuracy of the emotion perception of sadness and peace are all around 90%. It can be concluded that this scenario test, the research system is able to sensitively recognize the emotional changes of each subject, and the accuracy of the recognition is high. The results show that the method can capture the user’s emotional state in real time, provide personalized feedback to the user, and achieve emotional interaction.

4.2. Performance Analysis of IMSIF Technology

In order to test the performance of MSIF technology that incorporates SWD and BCI, sensory information is collected through sensors and data pre-processing is performed. Three feature sets of odor, smell and vision are selected for the study and the samples are tested using multivariate statistical analysis for analyzing the sensitivity of the IMSIF technology to the sensory samples, and the results of the experiments are shown in Fig. 9.

Multivariate statistical analysis results: perceptual sample feature classification.

Fig. 9(a) the feature classification results show that the perceptual samples are clustered within the group and the combing between the groups is obvious, and it is verified by the replacement test in Fig. 9(b), the variable intercepts are less than 0. The results show that the system does not have the overfitting problem, and the classification distances between the variables are small, which indicates that the system is more sensitive to the features of the perceptual information. The score of R2 of the system is 0.91 and the score of Q2 is 0.89. The results show that high intra-group clustering and clear inter-group separation indicate that the selected features have efficient ability to discriminate situational perception. Meanwhile, the multivariate statistical analysis method enhances the model’s ability to process multidimensional data, so that the system can maintain good performance in different scenarios and provide guarantee for subsequent complex real-time data processing. Then the experiment is conducted to collect action data and perception data from the subjects. Based on the classifier, the different data collected are filtered and noise reduction as well as pre-processing, and the data features are classified. The classification results are shown in Fig. 10.

Fig. 9. 3D visualization of feature recognition: classification of motion data and perception data.

Fig. 10 shows that in the researched technique can effectively analyze the physiological and behavioral signals of the user and shows the changes in the user’s movements in three dimensions and process and classify the features of different perceptual and movement data, compared to the human eye, the researched MSIF technique can objectively evaluate and classify the user’s movements. The results demonstrate that the proposed method is capable of providing an intuitive and comprehensive analysis, as well as enabling dynamic monitoring and evaluation of users.

Furthermore, it provides the requisite visual basis for the improvement of training or treatment strategies. Then the predicted real values of the improved MSIF technology and the MSIF technology before the improvement are compared, and the advantages of the Improved Multi-Source Information Fusion (IMSIF) technology have been verified, and the comparison results are shown in Fig. 11.

Fig. 11(b) displays the analysis of predicted values for the improved MSIF technique, indicating a prediction accuracy of 0.98. This accuracy value is 27% higher compared to the pre-improved MSIF technique. The findings demonstrate that the enhanced method is more efficacious in addressing data processing scenarios that necessitate high accuracy and reliability. Furthermore, it plays a pivotal role in guiding the practical application of such scenarios.

To further verify the advanced nature of the proposed method, Transformer model, Long Short-Term Memory network model (LSTM), and Self-Attention network model (SA) are compared in this study. Moreover, the model prediction accuracy, recall rate, F1 score and processing time are evaluated. Specific results are shown in Table 2.

Table 2 results illustrates that in terms of accuracy, IMSIF is 0.935, Transformer is 0.883, LSTM is 0.905, and SA is 0.913. The results display that although the comparison model has certain performance in multi-source data fusion tasks, it lacks adaptability in processing complex user interaction data. In terms of recall rate, IMSIF recall rate reaches 0.913, which has obvious advantages compared with the comparison algorithm, indicating that the technology can capture user status changes while reducing the leakage of processing. In terms of F1 scores, IMSIF reaches 0.922, indicating that the method can effectively deal with the challenge of data imbalance. In terms of processing time, compared with Transformer, LSTM and SA, IMSIF method shortens 70ms, 110ms and 80ms respectively. It indicates that this method can meet the response requirements of real-time interaction.