Software Method for Distributed Asynchronous Big Data Processing of Fundraising
摘要
The article describes proposed enhanced software method for predicting volunteer fundraising success with asynchronous big data processing using machine learning algorithms (logistic regression, random forest and gradient boosting). Results are compared in terms of model accuracy and speed. To scale, parallel processing with serverless AWS Lambda and Simple Queue Service is implemented. Features for asynchronously sending messages for processing by the web application component to the message broker are implemented. The broker redirects messages for processing to the existing Lambda environment containing the code for predicting fundraising success. For the investigation, a set of programming technologies integrating with cloud services is used. The proposed software architecture ensures high throughput for updates on fundraising with the ability for parallel, independent processing by the implemented code. Taking into account the processing time of all messages in the fundraising updates dataset as the main criterion for evaluating the speed of the implemented software method, an analysis and comparison were conducted between architectures using the basic and proposed methods. Despite similar processing time indicators for both architectures with a message set not exceeding 10, a reduction in the required processing time for larger datasets was achieved. The proposed architecture ensures high throughput and reduces processing time by 55–81% for datasets exceeding 30 messages compared to the basic architecture. The difference in results, depending on the number of messages, is explained by the cold start of message processors, with the final initialization of the last ones coinciding in time with the completion of processing the message set by the architecture with sequential processing. According to the research, increasing the number of messages and using pre-initialized environments minimize the part of the total processing time allocated to launching the event handler. The proposed software method is optimized for processing data in services that require high throughput.