I’ve been running a VPS with the same IP (v4 and v6) for ~15 years, so I’d say its stable enough. What should I use as an upstream source of time in this case that would actually be useful?
You can use pretty much anything except hardcoding pool.ntp.org itself, including individual servers in the pool. You don't have to be a specific stratum or anything, and the monitors are comically generous with time accuracy (+-100ms)