Testing AI/LLMs to see how well they can build a website from scratch

2 points by swax 2 years ago | 2 comments

Wrote an open source command shell wrapper for LLMs to use. Just testing it with with writing a website from scratch. Comparing GPT-4 and Claude 3 Opus. The results: Pretty similar, they can write a PHP page, with components, styles, and active elements like a fun quiz on one of the pages.

Video: https://www.youtube.com/watch?v=4TnaJLq0E-o

Article: https://naisys.org/articles/3-dollar-website-gpt4-vs-claude-opus

GitHub repo: https://github.com/swax/NAISYS

matt3210 2 years ago |

A real test would be to invent a framework and see if it can do anything on the new framework. Otherwise it’s just a search engine that amalgamates pieces

swax 2 years ago | |

Amalgamating the pieces is the test. LLMs really can't do it that well.

Is it context size, training, number of iterations, number of agents? That's what's being tried to improve the results.