The benchmark - Codeforces programming contest.
GPT-4 Codeforces Rating is 392 points, improving GPT-3.5’s 260 points.
AlphaCode by DeepMind achieves 1,238 points!
Those who tried Bard or Codey will very likely agree that Google’s models and solutions are not better than OpenAI ones. So, what is going on here?
The secret sauce? AlphaCode is composed of two components, not one. A Code Generation component, and a Code Integrity component (that includes test generation, filtering and clustering according to tests runs).