Pipeline Processing During Importing Large Files into Web-Based Systems
摘要
This article explores the challenges of importing large datasets into web-based systems and presents effective solutions for handling extensive data streams. The study focuses on formats such as XLSX, CSV, and XML, often containing tens or hundreds of thousands of records, which can lead to termination or failure of web applications due to server response time constraints. To address these issues, the research identifies two key strategies: segmenting large files for manageable processing and implementing batch processing to prevent prolonged execution times. A system was experimentally developed using a PHP backend and MySQL, demonstrating the feasibility of segment-based data reading and mitigation of memory-related errors. The study proposes pipeline-based architectures utilizing modern techniques, including file imports via web forms, direct links, and APIs. These architectures were validated as robust and scalable, effectively managing data volume challenges. A notable outcome is the separation of the data processor from the main storage into two independent systems, enabling the creation of a Software-as-a-Service (SaaS) for large file processing. This approach facilitates compatibility with distributed cloud platforms, ensuring flexibility and scalability. Results show significant improvements in efficiency and reliability compared to traditional methods, providing a practical framework for developers to handle large-scale data imports across various web technologies. The findings highlight the potential for future enhancements, such as data matching protocols, to further extend the architecture’s applicability.