Ant Group – scaling to 1.37M QPS on Ray

Ant Group – scaling to 1.37M QPS on Ray(anyscale.com)

3 points by george_123 3 years ago | 1 comment

george_123 3 years ago |

This is a guest engineering blog post from Ray contributors at Ant Group, discussing how Ant Group implemented scalable Ray Serving architecture atop Ray, deploying 240,000 cores for model serving, scaling by 3.5x from previous year, and reaching 1.37 million TPS during peak times.

(resubmission, URL redirected to /Engineering last time)

george_123 3 years ago |

This is a guest engineering blog post from Ray contributors at Ant Group, discussing how Ant Group implemented scalable Ray Serving architecture atop Ray, deploying 240,000 cores for model serving, scaling by 3.5x from previous year, and reaching 1.37 million TPS during peak times.

(resubmission, URL redirected to /Engineering last time)