Hey there varlock! It's a good thing you're asking this question here.
Now the answer to your question[s] will really depend on what it is you're trying to build (and more importantly, sell).
Take for instance a website like Stackoverflow, it will be quite hard for anyone to develop and get people to use an alternative for the same because the entire value the website delivers is derived from its user base.
However, if your idea is something like Mailchimp, it will be a lot easier for you to build and sell the same because the value delivered by the product depends mostly on whether it works for prospective users and your ability to convince them about the same.
> What would you do to differentiate the product? What steps would you follow?
I would pick a niche (subset) from the market the incumbent player is catering to and build something superior catered only towards them and then grow horizontal from there.
> Or would you drop the ball altogether (hopefully not!)?
Hello no!
Depends of whether networks effects are an important aspect of the incumbent's success and the resources that are at my disposal. So if I don't already have a bunch of money or risk appetite (even in terms of time), I wouldn't really invest myself in doing a marketplace sort of thing (where network effects matter i.e. product value is derived from user base).[1]
If network effects don't matter and I can build and sell this thing on my own or if I at least have access to the sort of people that can help me, I would go ahead. This might be a good read: https://invertedpassion.com/copying-ideas-is-highly-underrat...
> What dimensions would you (or did you) consider to differentiate your product?
Pick a niche from your competitor's market. Get into the skin of your user / customer from this niche. You'll start finding aspects which you can improve and build upon to make a prospective user possibly switch. Don't dismiss things like being a small bootstrapped company with an accessible team can be a differentiator too.
[1] - However, this will differ a lot from person to person because everyone's risk assessment in life is quite different based on their goals and current situation.