I think it is generally fast enough just using Supabase js library, but if you want to manage your own pools and care about handling millions of concurrent requests, only then I think moving away from it might show some visible performance benefits.
it's also worth noting here that a big benefit of supabase-js is that you can use it from the browser (assuming you have sufficient RLS policies). PostgREST is hosted on the same server as Postgres, so essentially you remove an entire network hop.