I can assure you: If you're asking this question, you are far from that point.

Nowadays you can host projects on services like aws, heroku, etc. and they have pretty good performance right out of the box. You will rarely reach a point where performance is a huge issue.

This may happen if you hit the HN front page, get covered by the NYTimes, etc. but even then, you could just spin up some dynos to handle it temporarily and think about if this is just a spike or an actual growth. In 99% of the cases it's just a spike and you'll be back to normal the next day.

If you DO reach the point where performance becomes important, that probably means you have enough users. And in the worst case you could even ask your users for help. But if you're doing this well, you can really just tweet for help and many people will try to help you.

TLDR: Don't optimize until you get to a point where it becomes painfully obvious that you MUST optimize.

For my own projects: Straight away to an extent. I take pride in making things work fast, within reason.

For work: I just try to avoid making things slow out of stupidity by avoiding things likes n+1 queries. On modern hardware this get's you to the "good enough" point. In enterprise software no one paying me has every cared about performance, they don't have a problem if it takes 15 seconds to load a page showing 10 items for instance, they certainly won't pay me to get that 500ms request down to 30ms.

And that's why commercial software is so slow and bloated, no one will pay a cent to make it otherwise.

If it's a concern just use a language that's within an order of magnitude of C and don't write anything exceedingly stupid.

Java, C#, Golang, and Node are all fast enough that you could run 99.9% of the sites on the internet with a single machine.