Ask HN: Startup Disaster Recovery

5 points by bdonohue 13 years ago | 6 comments

Our company is in the process of raising a round, and one of our investors has asked us to put in place some disaster recovery procedures before investing. What DR procedures do you have in place for your company?

Some background: * I am sole coder * All code is on GitHub * App is deployed to Heroku * Domain names are on NetworkSolutions

eduardordm 13 years ago |

Very 'interesting' they asking you this. Specially when it looks you are just starting.

I run a credit card company on AWS, our website runs on a single, small aws instance, but the POS/Financial servers are setup this way:

- 8 multi-az oracle rds instances, 4 of them are slaves. It means we have 16 servers where 8 are manually standing by and 8 are on rds multi-az system. The 4 slaves are set to be used to recover from a data tampering event, not aws outages.

- Some dozens of app servers on the same region of each database. We use a custom tcp/ip server I built in java almost a decade ago. We are thinking about migrating to apache Mina.

My personal tips:

- Why are you paying the heroku premium? Build you own aws servers, put nginx and set cheap instances behind it. Let capistrano be your heroku.

- Use some dynamic dns service, I would recommend dynect. Avoid amazon Route 53.

- Use multi-az, set up the same database in another region (continent)

- Get out of us-east

Build the very same setting into another region, create scripts to load the slave country's database if possible.

If your database is too fancy or large:

Shard or distribute.

(Edit) Almost forgot: ALWAYS have fresh as possible, offline backups of everything. Oracle is god when it comes to making this easy.

bdonohue 13 years ago | |

Maybe I've worded this incorrectly, but they're basically saying, "What happens if you get hit by a bus?" My co-founder has access to one of our three projects, but no one else has access to Heroku, GitHub, etc.

Also, the Heroku vs EC2 is another can of worms. Heroku is the better decision for where we are now.

eduardordm 13 years ago | | |

Incorporation agreements must have outlines on how to deal with such events. Other than that, you need someone to replace you in case something happens to you. That person should have the same access and info than you.

chris_dcosta 13 years ago | |

Nice answer - but I don't think that's what he was asking. I think the suggestion was that he's got his code secure on external services - so what is there to lose?

Sharma 13 years ago | |

eduardordm, nice details you have put there.

Can u please elaborate more on :

"Why are you paying the heroku premium? Build you own aws servers, put nginx and set cheap instances behind it. Let capistrano be your heroku."

Like how does that setup works?

whichdan 13 years ago |

Few thoughts:

- Don't worry about code backups, assuming you have the following: 1. Local copy 2. Local backups 3. GitHub 4. GitHub's own backups 5. Copy on Heroku 6. Heroku's backups (?) 7. Any server backups

That's a hell of a lot of backups.

- Make sure you can time how fast you can provision a new server and restore backups. When you actually have to, it'll give you some peace of mind knowing that your backups can be restored, and that you can reliably guess how long it'll take.

- Make sure your domains are "Locked" in NetworkSolutions, which prevents transfer-out requests. I'm still mixed on whois privacy, but you're probably okay using it with NetworkSolutions.

- I've also struggled with figuring out what to do if I can't run the business suddenly. For your domain, you can register it for several years or set it up on auto-renew, so that's not really an issue. As far as GitHub/Heroku, ... that's a tough question.