I did the first half of my career in the Bay Area (>10 years) and am finishing it in NYC. Been in startups in both places. Here are the biggest differences I've noticed since moving to NY:
1. Old money. In California, the VCs and Angels I met were mostly self-made; they had cashed-out of the dotcom boom, or recently been apart of an IPO. They were all 20 - 40 somethings. NY is very different. I had never been exposed to the "country club" / "summer house in Nantucket" / "trust fund" scene before. It definitely feels like a different planet sometimes. So if you're raising money, be prepared to have conversations with the type of people you might have thought didn't exist anymore.
2. No echo-chamber. For the best and brightest in NYC, tech is often NOT the obvious choice. In many circles, going to the tech scene, (let alone the startup scene), is seen as odd. ("Why didn't you go into finance?" "Why didn't you go into law?" "Why aren't you a doctor yet?"). The greatest minds of the generation are becoming MOTUs (Masters of the Universe) at hedge funds and big banks. They're making a ridiculous amount of money, and really have no interest in this little tech thing we're doing. Talking about your startup in NYC feels odd sometimes. You're not praised or catered-to like you are in SF. Nobody really cares what you're doing, and nobody takes you that seriously until your run-rate is the hundreds of millions per year. It's a nice change in context actually. The SV echo chamber is gone.
Besides that (and NYC's insane weather), things are pretty much the same. Lots of talent, but it's expensive to hire. Lots of businesses, some good, some bad. Lots of VCs and Angels. Terrible rent, good food, etc.
Is it worth the move? Yes. Simply because when you move to NYC you may be seated at a cramped restaurant shoulder-to-shoulder with your next investor; and in SF you might see @sama walking down Market street. That kind of "right place at the right time" moments are more rare outside of those cities. Which coast you choose is entirely up to you; either way you're making a good move.