1) The vast majority of working people do not have much (if any) vacation time, nor the job security/financial security to take it.

2) Even if they do, they may have young children, which would prevent them from taking trips of this sort. Not just practically (where to leave the kids?) but also, many parents are reluctant to go to far flung places and participate in relatively risky activities that would increase the likelihood of their child(ren) being left with a disabled parent or worse, without a parent.

3) Requires a big chunk of disposable income, also less common than one would surmise.

4) Many people are simply not interested in adventure travel, preferring, well, normal travel (sightseeing, local cuisine and culture, etc).

There is certainly a market for adventure travel companies (why "startup"?) and they do exist. However, to be mainstream, the product would need to be useful and accessible to the masses, which for the above reasons it is unlikely to be.

This graph also has something to do with it:

http://cepr.net/images/stories/report_images/no-vacation-nat...

Because there is no room to scale an adventure travel business. The market is niche.