ChaCha can be accelerated with generic SIMD instructions on any arch that supports them. Personally I think that's preferable to algorithm-specific instructions like AES-NI.

Other crypto is used for things like password hashing, which is something that's a common target for brute force attacks. You want that to be slow.

Suppose an attacker has to try 1,000,000 passwords until the correct one is found. If the password hashing algorithm is fast, so the attacker can try 1,000 guesses per second, it will take the attacker less than an hour. If the password hashing algorithm is slow, so the attacker takes a whole second to try each guess, it will take the attacker many days. With a complex enough password, and a slow enough password hashing algorithm, the attacker will die of old age before the correct password is found.

But that's not the case for other classes of crypto algorithms. For symmetric encryption algorithms like AES or ChaCha20, hashing algorithms like SHA-2 or Blake2, elliptic curve algorithms like secp256r1 or curve25519, and so on, being faster is a good thing.