How fast can you parse a CSV file in C#?(lemire.me) |
How fast can you parse a CSV file in C#?(lemire.me) |
I really wish the author would show some proof for claims like this. io is a complicated beast and it's not certain that the subsystem could go faster with the patterns the code is doing or not. Plus there's no indication of what else is going on simultaneously on the system, so perhaps a background job spin up and the disk is io bound. We don't know and it appears that neither does the author.
That said, I apologize if I came off more critical than I intended.
There’s a lot of overhead as soon as you involve a filesystem rather than a block device, even on a dedicated disk, particularly with btrfs. I don’t know if the same is true with MacOS and APFS; this isn’t the area I usually work in. However copy-on-write file systems (which I believe apfs is) are somewhat predisposed to fragment files as part of the dedup process; I don’t know if apfs runs it online in some way so it could have affected the article’s author’s results.
The standard library implementation details can also have a huge impact, eg I observed with Rust for a prior project when I started fiddling with the read buffer size:
https://github.com/rust-lang/rust/issues/49921
The other issue that I see is that their I/O is implicitly synchronous and requires a memory copy. They might see better performance if they can memmap the file, which can probably solve both issues. Then if C# allows it, they can just parse the CSV in-place; with a language like Rust, you can even trivially do this in a zero-copy manner, though I suspect it’s more involved with C# since this requires setting up strings / parsing that point at the memmaped file.
At that point, the OS should be theoretically able to serve up the cached file for the application to do some logic with, without ever needing to copy the full contents again into separate strings.
However, the fastest-performing implementations in 1BRC challenge that were written in C# ended up with inconclusive results whether using memory-mapping over RandomAccess.Read API (which is basically a thin wrapper over read/pread calls) is faster or not: https://github.com/noahfalk/1brc/?tab=readme-ov-file#file-re...
You can relatively easily do 2 GiB/s reads with RandomAccess/FileStream as long as sufficiently large buffer size is used. FileStream default settings already provide a quite good performance, and make it use adaptive buffer size under the hood. Memory-mapping is convenient but it's not a silver bullet (in this context) and page-faulting then mapping the page and filling it with data by performing the read within kernel space is not necessarily cheaper than passing a pointer to a buffer to read into.
The challenges in Rust and C# are going to be very similar in this type of task since C# can just pin the GC-allocated arrays to read into, call into malloc or 'stackalloc' the temporary buffer inline, and the rest of implementation will be subject to more or less identical constraints. C# is probably the closest* "high-level" language in feature set to Rust, even if this sounds strange. There's a sibling submission that covers an another angle to this: https://news.ycombinator.com/item?id=41963259
* have not looked through Swift 6 changes in detail yet
The link to source code is there. It uses BenchmarkRunner class, which handles warmup and multiple runs. I am assuming author ensured that stddev was small enough that the raw numbers is valid. And with 11MB file size, it will certainly be cached between runs - even if something else evicts the cache, it will show up as high stddev and then presumably author would rr-run it again on quiter systems.
Also the fastest library in TFAs benchmark is Sep [2], which makes use of SIMD.
* for all intents and purposes that's what C# compiles to under NAOT. Even with JIT there is no interpreter stage, CIL is always compiled to native code.