oh yeah, I'm pretty comfortable to managing a debian distro. Definitely the first time was a bit of a pain, but I built a template for my subsequent projects and now its about 20 minutes of work for a totally new repo.
I also just don't like handing the keys to the kingdom over to Vercel as it were (just a personal preference really). I'm working on migrating over to a privately instanced Gitea so that I don't have to depend on Github either for my deployments but that's still in the works.
So I actually use serverless functions as ad-hoc "workers" if I don't feel like setting up a full fledged backend API - so sometimes they'll be working for upwards of a minute on a request and then lodge the final results in a notification-redis. It's not really what serverless functions are intended for, but... it's nice to sometimes not have to deal with building an entirely separate backend.
Since I control the VPS, I can easily spin up more resources to scale without worrying about Vercels payment options. It also makes it easier to migrate somewhere else if I want to since its dockerized - trivial to move to Google Cloud, Digital Ocean, AWS, etc.
A friend of mine whose project sort of blew up and went viral also said that while the autoscalibility of Vercel was very nice - it was a bit of a double edged sword, he mentioned that it was extraordinarily difficult to figure out which routes and files werer contributing to his high usage on his bill. Maybe things have changed, that was a few years back.