Rate limiter for LLMs outperforms exponential backoff

Rate limiter for LLMs outperforms exponential backoff(github.com)

2 points by sftombu 1 year ago | 1 comment

sftombu 1 year ago |

rateLLMiter is a Python rate limiter that smoothes out requests to LLM APIs to get faster, more consistent performance. It uses a ticket bucket algorithm rather than the usual exponential backoff.

sftombu 1 year ago |

rateLLMiter is a Python rate limiter that smoothes out requests to LLM APIs to get faster, more consistent performance. It uses a ticket bucket algorithm rather than the usual exponential backoff.