The more moving parts you add to a system , the more points of failure and latency you will add to your system. Not to mention their impact on your marginal costs.

Some specialized services are not only essential but also replicating them in-house is beyond the expertise of a normal start-up. I would put Stripe, AWS/Heroku/Google, Github in this list.

Others are neither essential (mission critical ) nor rocket science to implement in-house. The start-up has to figure out if the impact of these non-essential services on the marginal costs is justified or not. And how long can you get away without spending money on nice-to-have services ?

- vendor lock-in: by integrating whoever's API in your code and making it part of your application you make it hard to switch to a similar service offered by someone else. Example: say you're writing a service that has to scale automatically. Cool, you can do this with EC2's APIs, so you integrate that in your application. When the threshold of request per second crosses a certain value, you spin up a new VM. If it's deeply integrated with Amazon's APIs (i.e you're using it to its full potential, tracking spot instances, whatever) it's very hard to switch to DigitalOcean, because they might have different concepts, and I'm not talking about just nomenclature.

- black box syndrome: you stop knowing how things work. This is convenient at first, but if you forget how to configure things "from the ground up", you stop being in control of your infrastructure/data, and ultimately, of your service.