Show HN: A wc rewrite in Rust that makes it over 100 times faster

Show HN: A wc rewrite in Rust that makes it over 100 times faster(github.com)

2 points by CallMeAlphabet 5 hours ago | 3 comments

ventana 5 hours ago |

At the first glance, the 100x difference is surprising. My first thought was that the Rust code [1] uses AVX instructions, which would've explained the difference if the regular wc was a naive C code counting characters in a loop; but from what I see, at least the regular coreutils wc [2] uses AVX too, so the reasons of the 100x difference are not immediately obvious to me. Have you got any good explanation why? I don't believe it's Rust vs C, it might be some optimization that does not exist in the coreutils C code?

[1]: https://github.com/CallMeAlphabet/fastwc/blob/main/src/main....

[2]: https://github.com/coreutils/coreutils/blob/master/src/wc.c

rurban 1 hour ago | |

It's doing a tenth of the work, ignoring unicode. Locale Latin1, not much more. No more ws delims than ASCII, no word boundaries.

collinfunk 4 hours ago | |

I co-maintain GNU coreutils and wrote the Neon 'wc -l' code, for reference.

Based on my quick look at the source code, it looks like this program can handle ASCII whitespace. GNU coreutils uses iswspace, which is locale-aware. That limits how much we can optimize. We could probably add more optimization if the current locale is ASCII, e.g., LC_ALL=C on GNU/Linux.