1. Introduction

Information retrieval, a paramount technology addressing the issue of information overload, has seen the rise of prominent search engines worldwide, such as Google, Yahoo, Bing, and Baidu. Despite their widespread usage, these engines may fall short in adapting to specific real-world contexts due to formal processes, limited personalization capabilities, and underutilization of user historical data ^{[1, 2]}. For instance, during occasions like the Chinese New Year, traditional e-commerce platforms equipped solely with search engine technologies struggle to anticipate user needs or offer personalized product suggestions based on historical data. Recommendation technology emerges as a focal point in both academic and industrial circles, particularly as big data ushers in a plethora of resources for Internet users and precise user-project descriptions for businesses ^[3]. Nevertheless, challenges arise in recommendation systems due to sparse data leading to accuracy issues and performance limitations on single machines caused by the sheer volume of data ^[4]. A recommendation system acts as an active tool in assisting users by sifting through extraneous or uninteresting information ^{[5, 6]}. It operates by extracting user and project attributes from descriptions or utilizing historical behavioral data to gauge the compatibility between users and projects ^[7]. While search engines wait for user queries passively, recommendation systems take the proactive approach of providing tailored project suggestions. By leveraging user data, project descriptions, and historical interactions, recommendation systems excel in furnishing personalized recommendations–a facet often lacking in conventional search engine results ^[8]. In its developmental journey, recommendation systems drew inspirations from search engine technologies, extracting item features from descriptive content and user interests from historical behavior data. Initially akin to search queries and web pages ^{[9, 10]}, users were seen as equivalents to queries, moving towards a content-based approach to rank the relevance between users and items as a recommendation metric. Noteworthy examples like Tapestry, functioning as an email filter ^[11], and successful early developments in news recommendation systems utilizing collaborative filtering stand as milestones in the field’s progress. Collaborative filtering methods resonated with everyday interactions, where individuals tend to endorse products, they find appealing to others. These advancements underscore the evolution of recommendation systems, distinguishing them from search engines through a proactive, personalized approach facilitated by user data and historical behaviors ^[12].

The inception of big data technology found its roots in the search engine industry, notably through Google’s three pivotal articles on GFS, MapReduce, and Bigtable. Subsequently, the Apache Foundation developed the open-source Hadoop platform based on these technologies, revolutionizing the landscape of big data processing ^{[13, 14]}. Empowering structured data processing, Hive, ensuring distributed application consistency with Zookeeper, and facilitating data transfers with traditional databases via Sqoop have solidified Hadoop’s indispensable position in the industry ^[15]. However, as big data complexities surged, the efficacy gap in simple MapReduce processes began to surface, notably in handling “intermediate data landing.” Spark, an open-source big data processing platform that excels in data reuse across tasks, emerged as a solution. Particularly valuable in machine learning and interactive applications, Spark has enticed numerous domestic and international enterprises to migrate their data processing operations to its robust environment ^{[16, 17]}. Notably, amidst a landscape of diverse recommendation algorithms, the author observed the limitations of content-based, collaborative filtering, and SVD matrix decomposition approaches in handling non-textual files, data sparsity, and storage challenges in big data environments. Consequently, an emphasis on refining the English education platform model recommendation algorithm ensued, recognized for its adaptability in such robust data environments. The author further delved into distributed data processing platforms, ultimately selecting Spark for its efficiency and scalability in handling data-intensive operations ^{[18, 19]}.

2. Recommendation Algorithm

3. Recommendation Algorithm for English Education Platform Model and Improvement in Big Data Environment

3.1. Function Gradient Descent

Recommend based on speculative ratings, use the methods introduced in the first two steps to obtain the user’s speculative rating for recommended items, sort the recommended items according to the size of the speculative rating, and finally extract the top N items with the highest estimated rating to generate a recommendation list ^{[20, 21]}. The UBCF algorithm was widely used in the early days due to its simple principle, easy implementation, and good personalized recommendation effect. However, as the number of users and items in recommendation scenarios significantly increased, the problems of the algorithm were exposed; These problems are concentrated in the problems of data sparsity and scalability. Due to the increase in the scale of recommendation systems, user feedback data is very sparse ^{[22, 23]}. At this time, the credibility of similarity calculated based on the shared historical data of users is getting worse and worse; When a user does not have the same historical data as other users, recommendations cannot even be generated for the user. As the scale of recommendation systems increases, the number of users and items can reach millions ^{[24, 25]}. When generating recommendations, UBCF needs to calculate both the user’s nearest neighbor user set and the user’s rating estimate for the items. The similarity between projects can be calculated in advance, and IBCF does not require real-time calculation of project similarity when generating recommendations, which greatly reduces computational complexity and facilitates system expansion ^{[26, 27]}. A hybrid recommendation system overcomes the shortcomings of a single recommendation algorithm and achieves better performance by combining multiple recommendation algorithms. There are various recommendation algorithms, each with its own adaptive environment and shortcomings. Hybrid recommendation utilizes different hybrid ideas to combine multiple recommendation algorithms, striving to provide users with more suitable recommendations. Common mixing strategies include weighted mixing. Weighted mixing first uses multiple recommendation algorithms to generate their own recommendations, combines the obtained results with certain weights, and adjusts the final results to recommend to users ^{[28, 29]}. The conversion combination strategy is to select one of the alternative recommendation algorithms based on the specific situation of the recommendation, such as the problem background, specific requirements, etc., and generate the corresponding recommendation ^[30]. In this combination method, Fig. 1 shows the evaluation process of collaborative filtering algorithm recommendation effectiveness. Other hybrid strategies include hybrid combination, feature combination, feature expansion, etc., which will not be elaborated here.

Fig. 1. Collaborative filtering algorithm recommendation performance evaluation flowchart.

Rating estimation indicators measure the effectiveness of recommendation algorithms by verifying the difference between the estimated and actual ratings of the algorithm. The rating estimation index evaluates recommendation algorithms by measuring the difference between the algorithm’s inferred rating and the actual rating, while the recommendation performance index evaluates recommendation algorithms based on whether the algorithm can correctly predict the user’s preference and attitude towards the project. According to the predicted user attitudes towards projects based on recommendation algorithms and their true attitudes towards projects, projects can be divided into four types: projects recommended by recommendation algorithms that users like, projects that users like but recommendation algorithms do not recommend, projects that users do not like but recommendation algorithms recommend, and projects that users do not like but recommendation algorithms do not recommend. The Hadoop ecosystem is an early representative of big data processing technology, and the Map Reduce programming framework and HDFS together constitute the two major technological supports of Hadoop. Map Reduce and HDFS have both effectively encapsulated the specific underlying details, reducing the entry barrier for users and ensuring the reliability of the development program, so they have quickly been accepted by everyone. However, with the popularization of big data processing, increasingly complex tasks have emerged, and a simple Map Reduce programming framework cannot meet the processing requirements of these complex tasks. When encountering a complex task that cannot be processed by a Map Reduce process, it is necessary to manually divide the complex task into small tasks. After each small task is completed, the intermediate results obtained will be stored in the hard drive. The next “small task” will read the intermediate results of the previous step from the hard drive as input data until all “small tasks” are completed and the final result is obtained. Although this processing method ensures the program’s fault tolerance, the IO overhead is very high. Many common machine learning algorithms have this requirement, and these algorithms often take the form of: repeatedly using a simple algorithm to process a dataset to obtain the optimal solution of the model; If using the Hadoop platform, for each iteration, it is necessary to read the dataset and model parameters from the hard drive, and then store the updated model parameters again. Fig. 2 shows a content-based recommendation algorithm diagram. In the era of big data, rich data is conducive to solving the overfitting problem of models in machine learning, but training programs based on the Hadoop platform are very time-consuming.

Fig. 2. Content based recommendation algorithm diagram.

In a distributed environment, data is first partitioned for storage, and the random gradient descent method is not convenient for training models in partitions. For example, data is stored in partitions A and B. To effectively train a model using random gradient descent, the data needs to be traversed on partition A and the parameters need to be updated. After receiving the updated parameters, partition B can only train the model on the partition. It is not possible to train the model on both partitions in parallel, because when training the model on both partitions simultaneously, two sets of model parameters will be generated, and the two sets of model parameters cannot be effectively fused. The least squares method does not have this problem, so the model training module uses alternating least squares method to train the model. The core idea of the alternating least squares method is: after controlling the hidden attributes of the user set, the loss function becomes a convex function about the hidden attributes of the item set. After obtaining the optimal solution of this function, the hidden attributes of the item set are updated; Re control the hidden attributes of the project set and update the hidden attributes of the user set.

3.2. Recommendation Algorithm for English Education Platform Model Based on Function Space Gradient Enhancement

The first step in training a model using the least squares method is to determine the information required to construct the problem. Updating user hidden attributes requires item attributes and ratings, while updating item attributes requires user attributes and ratings. To facilitate efficient computation, four RDDs are designed: User In Blocks, User Out Blocks, Item In Blocks, and Item Out Blocks. These structures reduce network traffic when accessing hidden attributes. Since data partitions may differ, repartitioning is needed to group records by user or item. During this process, hidden attribute data may be repeatedly transmitted. The In Blocks and Out Blocks design reduces redundant transmission by enabling data reuse. Spark’s RDD abstraction supports efficient parallel computation and fault-tolerant distributed processing.

The most important concept in Spark Core is distributed elastic datasets. With these operators, users can perform processing operations on RDD like they would on a local dataset, which greatly reduces the programming burden on developers. For example, to filter the content of rows containing “error” in the log file, when the operator acts on RDD, a new RDD will be generated. The newly generated RDD is an abstraction of a dataset, and real data may not be immediately calculated, but these RDDs will record all the information on how it was formed; There is a dependency relationship between RDDs to ensure their fault tolerance: when data from a certain RDD needs to be used during the calculation process, and this RDD is replaced out of memory, Spark will reconstruct the RDD based on its Lineage; Compared to the method of backing up data, the “coarse-grained conversion” method adopted by RDD is much more efficient. Spark SQL can easily and efficiently load and query structured data. Data Frame and Data Set are two data types used in Spark SQL to store structured data. They are conceptually equivalent to tables in relational data, but have been optimized at the underlying level and can be constructed from external data sources, RDDs, or SQL query structures. Compared to RDD, Data Frame has added new operators, such as the ability to perform SQL queries and optimize execution plans. In addition, users can use SQL to query and analyze data in Spark programs or external tools connected to Spark SQL through standard data connectors. In collaborative filtering algorithms, it is necessary to measure the similarity between users or items. The data used to measure the similarity between items and users is either related item data shared by two users or related user data shared by two items. In the big data environment, the amount of historical interaction data between users and projects is very large, but the potential situation is even larger in magnitude; Taking an e-commerce platform as an example, assuming that there are 1 million users and 10 million products provided by the platform, and assuming that each user purchases an average of 50 products on the platform, there are a total of 50 million historical data. However, there are potential scenarios of 1 million * 10 million, which is as much as 10 trillion yuan. This is the data sparsity problem in the field of recommendation systems in the big data environment. In the case of sparse data, it becomes difficult to measure the similarity between users and projects, mainly because the amount of data used to calculate similarity is too small, which leads to a decrease in the reliability of the calculated similarity indicators. When calculating neighbors, Fig. 4 shows the distribution evaluation map of user learning behavior. The user based collaborative filtering algorithm needs to scan the user set to calculate similarity information, while the item based collaborative filtering recommendation algorithm needs to scan the item set to calculate similarity information of items; In the big data environment, the running time of collaborative filtering algorithms is a major issue for projects and user numbers that can reach millions. In real-life scenarios, projects with the same nature may be named differently, and they are essentially similar. However, collaborative filtering algorithms cannot discover or utilize this similarity.

Fig. 3. Solution diagram for cold start problem of recommendation algorithm.

Fig. 4. Distribution evaluation of user learning behavior.

4. Research on the Application of Course Recommendation Algorithms in English Education Platforms Based on Big Data Analysis

The matrix factorization in the algorithm uses singular value decomposition. To apply SVD in the field of recommendation systems, the first step is to address the problem of data sparsity. Because the data in the recommendation scenario is very sparse, in order to obtain meaningful decomposition results, it is necessary to fill the matrix; The rating matrix can be filled using either the average project rating or the average user rating; It is also possible to standardize the matrix. Finally, perform singular value decomposition on the preprocessed matrix, considering that the essence of the matrix decomposition model is to learn how to describe user preferences and item attributes in a low dimensional space, and then infer user ratings by multiplying user and item attribute vectors. The most important difference between the English education platform model and the singular matrix decomposition model is that the English education platform model does not require mean filling of historical rating data. That is to say, the English education platform model no longer uses pure mathematical methods for matrix decomposition, and then obtains the attribute matrix of users and projects; The English education platform model, from the perspective of machine learning technology, understands the low dimensional representation of users and projects as parameters of the model. Historical rating data is used as the dataset, and the quadratic difference between the estimated and actual ratings is used as the cost function of the model. After training the model, the optimal solution for the hidden attributes of users and projects is obtained, which is the model parameter with the smallest difference between the estimated and actual ratings; These hidden attributes represent users and items in a low dimensional space. Table 1 shows the comparison of experimental results for identifying English article errors. By using this function, the property information of the house can be mapped to the price dimension; Furthermore, predicting the housing price problem can be transformed into finding some suitable functions $F(X;P)$, where $X$ represents the variables of the function, $P$ represents the property information of the house in the housing price prediction problem, and $P$ represents the parameters of the function. We hope that the houses in the training set can be mapped with this function to minimize the difference with the real housing price, that is, to find the optimal model parameter $P$.

The most excellent recommendation algorithm, the English education platform model, can still achieve good recommendation quality even in sparse data, which has attracted attention from academia and industry. Designs an improved English education platform model recommendation algorithm based on function space gradient enhancement. Compared with traditional English education platform model algorithms, this improved FGBLFM algorithm can achieve better results. The English education platform model recommendation algorithm was born in the recommendation algorithm competition organized by Netfix company and achieved excellent performance in the competition. It inherits the idea of using singular matrix factorization-based recommendation algorithms to extract hidden attributes of users and items, without the need for data filling, reducing the storage and calculation costs of the model, ensuring the credibility of the final results of the model, and the recommendation effect is better than that of collaborative filtering recommendation algorithms. However, the English education platform model itself also has these issues. This algorithm combines function space gradient enhancement with the English education platform model, using boosting strategy to gradually increase the complexity of the model, while using contraction factors to control the effect of later added dimensions, thus obtaining better model parameters compared to traditional English education platform models, and improving the effectiveness of recommendations. The recommendation algorithm for English education platform models based on function space gradient enhancement can be divided into five steps. The first step is to use traditional English education platform models to decompose the rating matrices of users and projects, and obtain the implicit attributes of the initial users and projects. The second step is to use the user and project hidden attributes obtained in the first step to infer the user’s rating on the project. The third step is to use the residual rating matrix as the direction of improvement for the model in the function space, and decompose the residual rating matrix again to form the dimension of improvement for users and projects. Step four, combine the dimension of improvement with the existing hidden dimensions of users and projects in a certain shrinkage ratio to form new hidden dimensions of users and projects, and repeat steps two to four until the training is completed. Step 5, use the hidden attributes of the end user and project to make recommendations. In GBDT, the mean of all samples is used as the initial predicted value for all objects; A similar method can also be used for the English education platform model with function space gradient enhancement, and even further, using the average score of each user or the average score of each item as the initial model. However, the author does not adopt the above method here. Considering that the effectiveness of the English education platform model is already good, Fig. 5 shows the evaluation chart of course completion rate and user activity, which can better estimate the user’s rating on the project. Here, the author directly uses the implicit attribute parameters obtained from the English education platform model that does not require very precise training as the initial values of the entire model. One effect of doing so is to accelerate the training speed of the entire model on the basis of a reasonable assumption.

Fig. 5. Course completion rate and user activity evaluation chart.

Training a model using the alternating least squares method involves fixing a parameter matrix ($P$) to ensure convexity of the overall loss function, then deriving another parameter matrix ($Q$). This iterative process optimizes matrices $P$ and $Q$ by sequentially fixing and updating them, gradually minimizing the loss function until convergence. Alternate optimization methods like random gradient descent are known for simplicity and speed in single-machine scenarios. In contrast, the alternating least squares method optimizes hidden attribute matrices efficiently. For instance, once the user's hidden attributes ($P$ matrix) are fixed, it only requires related rating data to compute each item's attributes ($q_i$) without needing other items' attributes. After calculating score discrepancies, residuals are utilized to enhance gradient direction in the function space, forming a residual score dataset for further model refinement. In scenarios with modest data volumes, collaborative filtering and English education platform models can be trained and deployed on a single server. However, the big data landscape characterizes significant data volumes posing challenges to recommendation algorithms. Handling vast data volumes necessitates distributed recommendation algorithms. By comparing results from different fusion models in big data processing platforms, a move towards implementing the FGBLFM algorithm on Spark, known for its iterative model training capabilities, was decided. The chapter will delve into the nuanced implementation details of this algorithm on Spark, recognizing the necessity for handling multiple iterations while training model parameters effectively. By embracing distributed recommendation algorithms on robust big data processing platforms like Spark, models such as FGBLFM can address the challenges posed by massive data volumes in the realm of recommendation systems effectively.

Firstly, before describing the specific functional modules, let’s introduce the situation of the data source. Considering that the general form of recommendation systems is to recommend projects to users and recommend users to projects, and the data source is mainly easily obtainable user rating data or user invisible rating data, the author has not considered incorporating some social network information, category attribute information, rating time information, etc. into the algorithm. The recommendation scenario is too simple, so this recommendation system does not involve specific recommendation scenarios, but is designed for general scenarios where the data source is historical data of users and projects. The specific data used includes three parts: user ID, project ID, and rating. This design not only retains the essential characteristics of recommendation system scenarios, but also improves the universality of implementing candidate recommendation algorithms. For example, if it is an e-commerce platform, after collecting user feedback data and organizing it into the format described above, the implemented FGBLFM recommendation algorithm can be used to provide recommendations to users; If it is an online video service platform, after collecting user feedback data, algorithms can be directly used for model training; That is to say, the algorithm is not limited to specific application scenarios and has good universality. Data preprocessing, filtering out incomplete or numerical data with unreasonable sizes, and rearranging the data format. Model training, implementing the training process of FGBLFM in a distributed environment, mainly includes constructing In Blocks and Out Block information. Fig. 6 shows the evaluation graph of user preference course types for different age groups, constructing the least squares method problem, solving the least squares method, updating In Blocks information, and other parts. The main basis for rating inference is based on the hidden attributes of users and items obtained after training. Based on the inferred rating, the Top N is selected for recommendation, including recommending items that users may like and recommending users who may be suitable for them. Store and load the user and project hidden attributes of the model to avoid duplicate online training of the model and ensure fast provision of recommendation services.

Fig. 6. Evaluation chart of course types preferred by users of different age groups.

5. Experimental Analysis

Find the residual of the current model on the training set. Firstly, use the user hidden attribute set, project hidden attribute set, and the initial Ratings dataset for join operations. Further map the newly formed RDD to obtain the inferred score, and then obtain the residual dataset of the true score and the true score, which is in the same form as the initial score dataset. Fig. 7 shows the user’s geographical location and course selection evaluation chart, filtering out all project sets that the user had previously intersected with; Then perform a join operation between the filtered RDD and the project hidden attribute matrix to obtain the candidate project hidden attribute set RDD; Similarly, user hidden attributes can be obtained from the user hidden attribute dataset; Afterwards, dot product the user’s hidden attributes with the hidden attributes of the selected item to obtain the model’s inference score RDD; Finally, perform a top operation on the model to obtain the n items with the highest inference score, which will be used as recommendations for users.

Fig. 7. User geographical location and course selection evaluation chart.

There are two feasible solutions for this situation. Call the interface of situation A to generate recommendations for each user. The other method uses all items as candidate items without filtering, and performs Cartesian product operation with the user set to generate all possible users and item pairs. Then, perform inner linking operation with the user and item hidden attribute matrix in order of the user and item pairs to obtain the user and item hidden attribute pairs for all users and item pairs. Fig. 8 shows the course rating and click through rate evaluation graph, and then obtain the speculated evaluation set. Then, perform top By Key operation on the speculated evaluation set to obtain the Top-N recommendation for each user. The first approach has accurate recommendation results but low efficiency; In the big data environment, the projects that users have encountered often do not make up 1% of the total number of projects, and the probability of recommended projects overlapping with the projects that users have used is low, which basically does not affect the recommendation effect. Therefore, the author adopts the second method to recommend the user set. A recommendation system can generally be divided into two parts.

Fig. 8. Course rating and click through rate evaluation chart.

Fig. 9. Evaluation of the correlation between user retention rate and recommended courses.

Common operations such as data collection, data cleaning, and data filtering are also related to specific algorithms, such as project-based collaborative filtering algorithms. Due to the stable nature of project attributes, the similarity between projects can be calculated offline in advance; Alternatively, models based on matrix factorization classes require offline model training. Another part is the online processing part, mainly aimed at presenting the recommendation results to users as soon as possible. After the offline training of the model is completed, it is necessary to store the model, mainly including the model’s hidden attribute dimension, user’s hidden attribute matrix, and item’s hidden attribute matrix. Spark provides an interface to save RDD to a specified path; At the same time, the hidden attribute dimension information of the model can also be saved in the form of a regular file under this path. Fig. 9 shows the correlation evaluation between user retention rate and recommended courses. As data recording, collection, and transfer can all lead to data loss, it is necessary to check the data before starting model training.

During the data inspection process, remove damaged data records to ensure data quality. The data preprocessing module not only performs data cleaning, but also performs certain transformations on data formats. Firstly, let me introduce some basic knowledge. Spark is written in the Scala language at the grassroots level, and the underlying language of Scala is the Java virtual machine. Fig. 10 shows the evaluation chart of course difficulty level and user satisfaction. At this time, the data format stored in RDD is the class of rating records, which consists of user ID, project ID, and rating data. RDD uses the default partitioning method to partition the rating records. To facilitate future iterative training of the model, it is necessary to re plan the partitioning method of the scoring records.

Fig. 10. Course difficulty level and user satisfaction evaluation chart.

Fig. 11. Assessment of course update frequency and user engagement.

To achieve the re grouping of scoring data, it is necessary to reset the Key and Value values of scoring records. The Key determines the partition of the records, and the Value is the scoring record. The new Key value consists of a user ID hash value and an item ID hash value. The way of storing review records as comment objects is very crude. A review record represents the storage cost, mainly including necessary information and object costs. When the dataset contains a large enough amount of data, the storage cost of review objects cannot be ignored, especially when data needs to be transmitted in the network. Therefore, it is necessary to readjust the data representation in the partition. The Value of the new RDD is a class composed of three parts of data, namely the set of user IDs, the set of project IDs, and the set of ratings. In this case, the data with the same index in the three arrays represents this rating record. Fig. 11 shows the frequency of course updates and the evaluation of user engagement. Compared to using scoring objects as values, the cost of additional storage and network transmission of data is greatly reduced. In addition, this design also provides convenience for obtaining the information required for algorithm iteration in the future.

6. Conclusion

A systematic analysis and research have been conducted on various algorithms, and English education platform model recommendation algorithms. Based on careful analysis and research of various recommendation algorithm principles, the English education platform model recommendation algorithm was ultimately chosen as the main research object of this article. Aiming at the problem of difficulty in training English education platform model recommendation algorithms to obtain optimal solutions due to excessive parameters in big data environments, an improved English education platform model recommendation algorithm FGBLFM based on function space gradient enhancement is proposed. The FGBLFM algorithm is used for training, which can obtain better hidden attributes of users and items than traditional English education platform models, and improve the accuracy of English education platform model inference ratings. We conducted in-depth analysis and research on the current mainstream distributed processing platform Spark, and implemented the improved English education platform model recommendation algorithm FGBLFM on the Spark distributed processing platform, verifying the feasibility of the algorithm.

The current intelligent systems related to English translation teaching either have the shortcomings of single functionality and do not meet the requirements of the new teaching mode, mainly focusing on grammar modification, essay correction, English translation, etc; Either there is a problem with the generalization of educational platform functions and the lack of customized educational platforms for language translation teaching. Missing translation error detection based on information content and synonym set, and spelling error detection based on Aspell improvement. In terms of effectiveness, the F1 value of the AEDSNN algorithm has increased by 10.4% compared to using semantic features alone, and by 30.9% compared to using the neural network model alone (BiGRU+BERT), indicating significant algorithm performance. The F1 value of the missed translation error detection algorithm is 0.634. Due to the limited number of recognition algorithms for this type of translation error, this work has certain significance.