Ask HN: How fast could Google/Baidu create and deploy their ChatGPT equivalent? I’ve been invited to invest in OpenAI. The valuations are pretty high. And while it may be worth it, I have to wonder how difficult it will be for others (Google, Baidu, AWS, etc.) to create and deploy similar tech if they see ChatGPT gets enough traction and it makes business sense for them to do it. |
I’m not sure if the next one will be a lot smarter than ChatGPT, in particular the ‘accuracy/honesty’ problem is not going to be easy to address without some kind of structural change (pair up the neural network with an SMT solver the way AlphaGo pairs game tree search with a network.)
What will change, however, is that the next one will be more resource efficient for training and inference. People still don’t really understand how deep networks work but they are figuring it out and there are many little changes that can be made that will add up to big gains.
Another problem w/ those LLMs is they all have a fixed window size, I think it is 4096 subword tokens for ChatGPT, I have been playing around with RoBERTa 3 for which it is 512 subword tokens. Those models are good on what they are trained to do but there is no really great way to apply them to larger texts that doesn’t break the ‘magic’. I have plenty of documents I want to cluster and classify that are much longer than that.
People will certainly be training models with larger windows but it seems like something that is scalable (more text makes a larger vector) or that has some way of consolidating multiple windows into a larger structure (think of how different it is read a paragraph critically than to read a book critically.)
There will be scalability problems going in that direction but I think that’s where the mountain is.
I get the feeling the same applies for Google et. al. They can deploy a chatbot whenever they want, but engineering the product of a chatbot takes time. You need bonus features, syntax highlighting, systems integration and, of course, a prompt and accompanying filter.
Personally, I see OpenAI as extremely vulnerable. If you're still feeling impressed by their technology, you should reassess the barriers-to-entry for their competition, and who exactly those competitors become...
The lack of any real-time information will hurt them in the long run. I doubt they have the setup and infra to do that. Maybe this is where Microsoft comes in, but we know how these software mixers tend to go
1. The technology behind it is in the open, and relatively simple ( the models are well understood, and easy to define ) To replicate that ... (with proper engineering etc a good and small team would probably need 1-3 months ) (The model itself could be built in a day )
2. Training: Here comes the biggie, training the above model on lots and lots of data is what gives it its quality. This is the bulk of time and cost, and also why smaller companies have a hard time replicating this. We are talking trainings costs in the 9 figures.
But 2 does not really matter to those giants tech companies, just smaller competitiors.
I would estimate depending on the desired outcome 3-6 months to replicate ChatGPT for those companies.
To answer the question it's the business model, remember google likes the status quo and it would just take time to train, also here's a breakdown if you want to see the blueprint. https://huggingface.co/blog/rlhf