The new Sonnet model definitely kills GPT-4o in the published benchmarks. I evaluated it for real-world use cases and compared against GPT-4o. It did better in all the cases.

Test Case 1: Python Code Generation Write a script to generate email address from name and domain

Test Case 2: Web Page creation Create an HTML file that displays a simple personal portfolio webpage. The webpage should include a header with your name, a profile picture, a brief introduction about yourself, and a list of your skills. Use basic HTML tags to structure the content and include some inline CSS to style the elements

Test Case 3: API Query Generation Write a cURL to call dall-e-3 API, and generate image of a Unicorn with a rainbow horn

Assessment: - Sonnet provides a more direct response to the coding requests. When we asked for a cURL command, Claude directly gave that, whereas GPT-4o created a bash script. - The web page created by Claude was much more aesthetically pleasing, and almost readily usable. Great for non-tech folks who want to create web pages. - Python code generation: This one is hard to say, both perform well. GPT-4o needs a bit more detailed instructions. - Pricing: Claude is cheaper than GPT-4o ($3 per million input tokens vs $5 per million tokens for GPT-4o) - Speed: Claude is faster at generating the first token.

Here's a detailed write up: https://blog.getbind.co/2024/06/21/claude-3-5-sonnet-does-it-outperform-gpt-4o/