NanoCSV: Enabling Efficient Parallel CSV Extraction with Hierarchical Finite-State Transducer

The growing use of comma-separated values (CSV) formatted files for various applications has highlighted the need for efficient parsing and analysis. However, traditional methods struggle with parallel transformation due to complex parsing rules, resulting in poor scalability and high overhead. Recent CSV processing techniques face mainly three key challenges: (1) inevitable performance degradation resulting from ineffective parallelization caused by character-by-character scanning, (2) limited parallelism on modern processors, and (3) redundant indexing costs for specific queries. This paper proposes a novel high-performance CSV processing framework, NanoCSV (The source code can be found at: https://github.com/nanocsv/nanocsv .). First, NanoCSV designs a new hierarchy finite-state transducer strategy to facilitate accurate parallel parsing with minimal states and transitions, increasing scalability. Second, NanoCSV accelerates parallel CSV data extraction and conversion through a SIMD string vectorization and extraction pipeline that incorporates fast SIMD-aware CSV string parsing. Furthermore, to handle queries with varying workloads, NanoCSV uses a dynamic decision-making indexing mechanism that combines hybrid bitmap and comprehensive tabular indexing. Extensive evaluations demonstrate the substantial performance improvement achieved by NanoCSV compared to state-of-the-art frameworks such as SIMDCSV, Spark, MongoDB, PostgresRaw, and GIO.