Ask HN: How does your team decide who works oncall on Christmas?

5 points by TimeWeSp 2 years ago | 14 comments

If you're on a software team that maintains cloud services or other infrastructure that needs to keep running 24x7, how does your team decide who is going to work oncall for shifts like Thanksgiving, Christmas, New Year's, or other times when most people don't want to be oncall? Is it just a top-down manual manager decision? Or do you have some kind of process/algorithm to help make it fair? Do new people get worse oncall shifts than people who have been on the team for longer?

elmerfud 2 years ago |

Maybe the better question is why is this a problem? I've been in oncall rotations my entire multi decade career and in the beginning I was frustrated that I was getting called all the time. Now I understand that the reason for that was bad management/owners.

I'm still oncall but if there's a issue that triggers a review to make changes so it never happens again. N+1 on everything, and N+2 redundancy on critical systems. A single failure should be a non-issue. It should take multiple failures inside of a single system to engage the oncall. Human error should be eliminated by not allowing changes during big holidays. Many companies lock down from Thanksgiving to new years.

While things can and will still happen, properly implemented and managed systems should eliminate the most if the oncall issues. The ones that do come up will be an "all hands on deck" kind of thing. People will be less afraid of oncall then they know they rarely get called.

TimeWeSp 2 years ago | |

I think what you describe is the ambition of every oncall team. But even when this works really well, so the calls are few and far between, just being scheduled to work oncall means you're more limited in what you can do. You've got to be reachable by phone. You have to be within a few minutes of accessing a computer with good networking. There'll be no road trip drives or flights to see the in-laws while you're the one on oncall duty. Do you think people prefer to avoid just being on oncall duty over big holidays, even if there aren't many calls? Perhaps that preference only shows up in a small portion of oncall teams.

elmerfud 2 years ago | | |

My point is that often times we are active participants in bad infrastructure and bad oncall programs. If you're not calling for post outage remediation plans and actively implementing them, then you're enabling bad process. If management is not budgeting for and prioritizing this work and not paying over time for time worked when on call, and you haven't found another job, you're enabling. Good employees don't enable bad practices, good employees don't work at companies that allow for bad practices. Even in a bad job market there is always room for good employees to shift to better employers. Good employers welcome these kind of changes because they attach a financial cost to on call and then they look to reduce that as a cost center. Bad employers do not track or attach a budget line item to on call outages therefore they continue to abuse the employees that allow themselves to be put in that situation.

Once you've worked on call for a good employer, you'll absolutely understand what I'm saying. The idea of being alone person shackled to a pager biting your nails because it's a holiday becomes a thing of the past.

gustavus 2 years ago |

Well back in the day it wasn't a problem because we had an on call allowance and you got paid for that, nowadays they told everyone

"you're too good for hourly work, we're going to make you salaried, it's great now you can just focus on getting your work done and not have to worry about hours on the clock. But you still have a timesheet and we still expect you to be here 9-5 and you can't leave early unless it's approved by your manager. Also you'll have to work 60-80hrs a week until we are caught up which should be sometime around Q3 of next year. But you're empowered now and we are flexibile"

screwturner68 2 years ago | |

They used to give whoever was on call for the week $200, we had a guy saving up for his wedding so he offered to be on-call for anyone who didn't want to be on call. It was a good deal for him and everybody was happy. It's amazing how cheap companies are today, for $10K you could make a lot of people happy.

thiago_fm 2 years ago |

Just give people extra off-days, like 2 days per each 1 of oncall for those days, or just give them more Oncall money. If there is the incentive and it is good, people will step up to do it, and overall that shouldn't be costly for a company that needs to run their infra 24x7.

ruairidhwm 2 years ago |

As a manager, I try and take most of it on when possible. The team have worked hard all year and I view it as good leadership to let them have that space.

philomath_mn 2 years ago |

Whatever the scheme, being on call is better than having to work 12 hr shifts every other holiday like my wife does (ICU nurse).

rolandhughes 2 years ago |

Generally they will hire foriegn workers on H1b.

COOL_DUDE_2112 2 years ago |

When I had a team, we'd trade based on faith -- Jewish or Muslim folks worked Christmas, and on their holidays us "cultural Catholics"[1] covered so they could have "their" day off.

Even among those who were no longer religious, folks often had families who still hold these beliefs.

(I was educated in a Catholic school, was an altar boy, and will never return to either place, so I use the term "cultural Catholicism" to distinguish my agnosticism from the full blown atheism I see on HN which often has a bit too right libertarian branding for my left libertarian tastes.)

TimeWeSp 2 years ago | |

Didn't you still run into times when most people wanted the time off? For example, times when schools are off, so everybody who is a parent wants that time off, regardless of whether it lines up with their cultural or religious important dates?

toast0 2 years ago | | |

Not everyone has kids. Even if everyone has kids, most of the time employees will be in different school districts and have different holiday calendars.

COOL_DUDE_2112 2 years ago | | |

dashack 2 years ago |

throw a dice and swap some shifts when possible