Ask HN: How does your company set RPC timeouts for microservices?

2 points by hq6 6 years ago | 3 comments

There are two sub-questions here. 1. Is the same timeout T set for every instance of service X making request A to service Z, or is a timeout chosen at runtime? 2. How is the static (or dynamic) timeout value chosen?

As a clarification, this is more of a poll to understand what people are actually doing, rather than a request for recommendation on what I should do. If you are willing to state which company you work for, I would be extra appreciative!

rachelbythebay 6 years ago |

What does the business need?

What can the service deliver?

If you need it faster than it can deliver... then setting your timeouts/paging/whatever to that point will just bring you pain and suffering.

I think you'll find a lot of people just say "what's p99 right now", add a little to it so "it won't page me", and leave it there.

Actually driving down p99... is not nearly common enough.

hq6 6 years ago | |

Thank you for replying!

I completely agree that people in most companies do not push down P99 enough.

I think this is a direct response to the second question, and assumes that people are setting timeouts statically. Is my understanding correct?

rachelbythebay 6 years ago | | |

It's been my experience that people usually build systems that expect a certain timeout for a certain service or perhaps service+method.

That said, not all clients are created equal. The ads folks who used to hit my system way back when had far tighter deadline requirements than the calendar or spreadsheet people, for instance.