How fast can you validate UTF-8 strings in JavaScript?

How fast can you validate UTF-8 strings in JavaScript?(lemire.me)

19 points by greghn 2 years ago | 17 comments

pi-e-sigma 2 years ago |

The article claims that accepting 'invalid' UTF-8 can be a security risk. Can someone explain how?

bhaney 2 years ago | |

For necessary performance reasons, a lot of code does not ensure that its input is valid UTF-8, and simply leaves that burden to the programmer. Strings are often passed through many, many layers of various types of processing, and it would slow code down significantly if each step in this process needed to validate the entire string all over again, so it's typical to expect that the string is validated once at the beginning, and then passed through various routines that each blindly assume their input is valid UTF-8 and guarantee to return valid UTF-8 as well if that assumption is true.

If you skip this initial validation, you open yourself to security problems caused by differences in the undefined behavior of these various routines that are assuming valid input. As an example, imagine if your code for escaping HTML saw an invalid byte sequence immediately before a "<" and failed to escape it because of that, but then the HTML parser you pass the result to treats the invalid byte sequence differently and successfully sees and parses the "<" that should have been escaped.

chubot 2 years ago | |

I would like to see some real examples too, but I think a pretty common thing is bypassing URL deny-lists with different representations of the same string, or any kind of ACL

e.g. the developer might either deny or allow /admin/, or /admin/.*

But if a given Unicode string [1] has two representations, then that check may not work as intended for one of them.

---

UTF-8 represents code points with 1-4 bytes, which are roughly similar to base 64 digits (because a continuation byte is 10xx_xxxx, it has 6 bits of freedom)

This opens up the possibility of "overlong encodings", which a UTF-8 validator must reject.

That is, you can do the equivalent of representing a number as "09" instead of "9" -- that's an overlong encoding. A decoder may understand "09" and "9" as both being the digit 9, but it's a different sequence of bytes.

So a naive decoding algorithm can accept a spelling of "admin" as 6, 7, 8, ... 20 bytes, not 5. There are many overlong encodings!

The only valid UTF-8 spelling has 5 bytes, but decoders that don't validate will "naturally" accept more (in fact I think I even wrote one of these :-/ )

In summary, if you don't do UTF-8 validation, then one layer doesn't reject the invalid representation, and another layer may decode it into a valid Unicode string.

Also, some regex engines work on UTF-8 encoded bytes, and some work on arrays of code points.

---

Again I'd be interested in more real examples.

[1] A sequence of "Unicode scalars" -- code points not in the surrogate range

superjan 2 years ago | |

Parser mismatch vulnerabilities.

https://www.brainonfire.net/blog/2022/04/11/what-is-parser-m...

LoganDark 2 years ago | |

Assuming that data is valid UTF-8 can result in you passing this data to other systems that make the same assumption. If nobody validates the data, it can cause unintended behavior or crashes. If someone downstream validates the data and returns you an error, it may be an error you didn't expect, and you may not handle it properly.

torstenvl 2 years ago | |

I'm not sure. However, there are some rendering engines that will completely choke on certain data. For a while, iOS would crash every time you opened iMessage if someone sent you an invalid sequence. Perhaps invalid UTF-8 could cause this? If so it would make an effective DoS attack vector.

eddd-ddde 2 years ago | |

There are functions that will throw on invalid UTF-8. Perhaps not a full security risk but unexpected errors are always a danger.

wolverine876 2 years ago | |

As on example, you could mislead users by replacing valid codepoints (characters) with visually identical ones, for example in URLs.

LoganDark 2 years ago | | |

That's not what invalid UTF-8 means. It doesn't mean replacing codepoints with other codepoints, it means that the encoding of the codepoints is invalid. Since JavaScript uses UTF-16 strings, it's possible to create JavaScript strings that don't have a valid UTF-8 representation (I believe unpaired surrogates can do this). Also, if you are working with raw bytes, there is no guarantee that it has any encoding at all (this seems to be what the article is about). There is WTF-8 [0] that proposes a possible solution to the UTF-16 problem, but purely as an internal representation and not as an interchange format. I think the Rust standard library uses WTF-8 to represent Windows wide-strings, which allows ASCII and UTF-8 algorithms to work on them, while still being able to represent any UTF-16 string.

[0]: https://simonsapin.github.io/wtf-8/

orangepanda 2 years ago | | |

Wouldn’t "visually identical” tokens still be valid utf8?

d3w4s9 2 years ago |

Looks like "...validate UTF-8 strings in Node.js" is a more accurate title, as valid8 runs on Node.js and the other uses a Node.js std library function. The TextDecoder method should work in the browser, but I don't see why you want to do this kind of validation in the browser.

gcau 2 years ago |

Not tested is https://www.npmjs.com/package/utf-8-validate (1,624,873 weekly downloads) - which I use.

striking 2 years ago | |

From https://github.com/websockets/utf-8-validate/tree/master/dep...

> This C++ library is part of the JavaScript package utf-8-validate. The utf-8-validate package is routinely downloaded more than a million times per week.

> If you are using Node JS (19.4.0 or better), you already have access to this function as buffer.isUtf8(input).

And at https://github.com/websockets/utf-8-validate/tree/master/dep...

> # Reference

> John Keiser, Daniel Lemire, [Validating UTF-8 In Less Than One Instruction Per Byte](https://arxiv.org/abs/2010.03090), Software: Practice & Experience 51 (5), 2021