OpenAI API(beta.openai.com) |
OpenAI API(beta.openai.com) |
Bloomberg reports the API is based on GPT-3 and “other language models”.
If that’s true, this is a big deal, and it epitomizes OpenAI’s namesake. The largest NLP models require vast corporate resources to train, let alone put into production. Offering the largest model ever trained (with near-Turing results for some tasks) is a democratization of technology that would otherwise have been restricted to well-funded organizations.
Although the devil will be in the details of pricing and performance, this is a step worthy of respect. And it bodes well for the future.
We saw this OpenAI demo: https://player.vimeo.com/video/427943452
and were just blown away. Very cool!!
I guess a spreadsheet is never too old [1] to learn new tricks :)
[1] Founder of https://mintdata.com here, so a bit biased (& opinionated about) spreadsheets, take the above with a pound or 10 of salt.
[2] I've sent them this example how we'd invoke their APIs, hopefully they'll let us into the beta, fingers crossed :) https://mintdata.com/docs/learn/core-mechanics/work-with-dat...
Imagine MS saying they "democratised" operating systems because, hey, you can buy their binaries, so everyone can use their operating system. Compare that kind of "democratisation" with open source oSs.
No, the truth is that as more and more resources are necessary to wring the last few drops of performance out of the current generation of deep neural net models it is only large, well-funded companies that have the resources to innovate - and everyone else is forced to follow in their wake. Any expectations that progress would lead to "democratisation" of deep neural networks research has gone out the window.
> They saw ratings hover around 60% with their original, in-house tech — this improved by 7-8% with GPT-2 — and is now in the 80-90% range with the API.
> The F1 score of its crisis classifier went up from .76 to .86, and the accuracy went up to 96%.
> With OpenAI, Algolia was able to answer complex natural language questions accurately 4x as often as it was using BERT.
I think the most informative are the first two, but the most _important_ is the final comparison with BERT (a Google model). I am, uh, a little worried about how fast things will progress if language models go from a fun lil research problem to a killer app for your cloud platform. $10m per training run isn't much in the face of a $100bn gigatech R&D budget.
Recently, OpenAI set the GPT-3 GitHub repo to read-only: https://github.com/openai/gpt-3
Taken together, this seems to imply that GPT-3 was more intended for a SaaS such as this, and it's less likely that it will be open-sourced like GPT-2 was.
Easy access to the 175B model would indeed be valuable, but it's entirely possible they're using a smaller variant for this API.
This makes a lot of sense and it seems they are telegraphing to monetize what they have been doing. It also seems like this is why they don't release their models in a timely manner.
Some interesting papers and datasets:
NL2Bash: https://arxiv.org/abs/1802.08979
ls is 2 syllables list dir is also 2 syllables with more meaning.
Ultimately, with natural language, the effectiveness seems to be when it is coupled with speech-to-text
Generic machine learning APIs are a shitty business to get into unless you plan on hiring a huge sales team and selling to dinosaurs or doing a ton of custom consulting work, which doesn't scale the way VCs like it to. Anybody who will have enough know how to use their API properly can jus grab an open source model and tune it on their own data.
If they plan on commercializing things they should focus on building real products.
I could make a much better site coding my own website from scratch and setting up servers myself, but for some projects I wouldn't even think about it that way, because using Heroku or Squarespace I can save a LOT of time and get the results I need much quicker.
Is OpenAI just a submarine so the tech giants can do unethical research without taking blame??? Its textbook misdirection, nonprofit and "Open" in the name, hero-esque mission statement. How do you make the mental leap from "we're non-profit and we won't release things too dangerous" to "JK we're for-profit and now that GPT is good enough to use its for sale!!". You don't. This was the plan the whole time.
GPT and facial recognition used for shady shit? Blame OpenAI. Not the consortium of tech giants that directly own it. It may just be a conspiracy theory but something smells very rotten to me. Like OpenAI is a simple front so big names can dodge culpability for their research.
What alternatives do people like?
Is it a confidence problem? Are the OpenAI folks not confident on a single use case? Or did I miss the live demo somewhere?
The crucial question is : is this paradigm viable for OTHER types of data?
My hypothesis is YES. If you train a HUGE image model using vast quantities of raw images, you will then be able to REUSE that model to work for specific computer vision problems, either by fine-tuning or 0/1-shotting.
I'm especially optimistic that this paradigm will work for image streams from autonomous vehicles. Classic supervised learning has proved to be difficult if not impossible to get to work for AV vision, so the new paradigm could be a game-changer.
This has been demonstrated for many years, it's not news. Many of the SOTAs like BiT require pretraining on JFT-300M, or Instagram, or what have you.
There are a few good examples of educational help on the list but it's really only scratching the surface.
I'm really excited and hope Kognity and EdTech in general can use this for even more value-full (both for students and teachers) tasks soon.
1) (on cluelessness) If Sama/GDB were as smart as they claim to be, would they not have realized it is impossible to run a non profit research lab which is effectively trying "to compete" with DeepMind.
2) (on disingenuity) The original openAI charter made OpenAI an organization that was trying to save the world from nefarious actors and uses of AI. Who were such users? To me it seemed like, entities with vastly superior compute resources who were using the latest AI technologies for presumably profit oriented goals. There are few organizations in the world like that, namely FAANG, and their international counterparts. Originally OpenAI sounded incredibly appealing to me, and a lot of us here. But if their leadership had more forethought, they would perhaps not have made this promise. But given the press, and the money they accrued, it has now become impossible to go back on this charter. So the only way to get themselves out of the whole they dug into was by making it into a for profit research lab. And by commercializing perhaps a more superior version of the tools Microsoft, Google and the other large AI organizations are commercializing, is OpenAI any different from them?
How do we know OpenAI will not be the bad actor that is going to abuse AI given their self interest?
All we have is their charter to go by. But given how they are constantly "re-inventing" their organizational structure, what grounds do we have to trust them?
Do we perhaps need a new Open OpenAI? One that we can actually trust? One that is actually transparent with their research process? One that actually releases their code, and papers and has no interest in commercializing that? Oh, that's right, we already have that -- research labs at AI focused schools like MIT, Stanford, BAIR and CMU.
I am quite wary of this organization, and I would encourage other HN readers to think more careful about what they are doing here.
To clarify: MS windows is closed source, but you can't very well train a large GPT model unless you're someone with the resources of OpenAI. So you're stuck with whatever they choose to train and make available to you.
[1] Cf second paragraph of https://openai.com/blog/openai-api/
For instance, if the original model is trained on English text exclusively and you want to fine-tune it on Greek text you are S.O.L.
That's wrong in almost too many ways to list. Sam left YC over a year ago, nor would he do such a thing. Nor does YC have that kind of power over companies, nor would it use it that way if it did. That would be wrong and also dumb.
With the way the field is moving, GPT-3 will be old news in a month, when more advances are made and open sourced.
I know it's trendy (and partly justified) to look down on OpenAI, but can you actually give any basis for this claim?
What kind of research is OpenAI doing that all the other big AI players (Google/DeepMind, FB, Microsoft) aren't also invested in? And even if others are doing the same, what part of OpenAI's research do you consider unethical?
> It may just be a conspiracy theory
Yea, it very much looks like that to be honest.
I believe all of them are doing unethical research, especially facial recognition. Notice the public backpedaling this week from all the big tech companies on this too. By directing their cash through OpenAI they can avoid whatever fallout comes from unleashing things like GPT3 on the world.
The most straightforward use case for GPT3 is generating fake but believable text. AKA spam. That's what it was designed to do. If you think fake news is a problem now, wait till someone is generating a dozen fake but believable news articles per minute by seeding GPT3 with a few words and hitting a button.
Its a conspiracy theory with some circumstantial evidence. We will probably never know either way, because who would admit to it if it was true.
They're directly selling GPT 3 even though they originally said they wouldn't release it because of potential bad uses.
They paid MS a ton of money for hardware and got a huge equity investment from them.
And lets be honest here, the easiest and most straight-forward use of GPT3 is generating spam and low quality clickbait. Its the only use case that requires zero effort. The whole thing is built to generate fake but believable text. Its DeepFakes for text.
I'm not saying the whole thing is nefarious and evil, just suggesting that OpenAI may not be what it seems. There's a lot of odd things going on with it. They should have done what universities do, spin off the technology into a different for-profit company and sell it. Instead of redefining their entire org structure to make money.
> What specifically will OpenAI do about misuse of the API, given what you’ve previously said about GPT-2?
> We will terminate API access for use-cases that cause physical or mental harm to people, including but not limited to harassment, intentional deception, radicalization, astroturfing, or spam; as we gain more experience operating the API in practice we expect to expand and refine these categories.
Burn all social media to the ground, I say.
This OpenAI work is almost certainly a way for these bigger corps to collude. Proving that would be impossible, though.
I could get poems to generate well. Tweets were a bit harder but I don't think we are at the point where you could just use a generative model to fool people that would be cheaper than actually hiring someone to write fake news. (Also shameless plug below)
[1] 1400 - TALK.8 - "A way to make fake tweets using GPT2" - Joshua Jay Herman https://thotcon.org/schedule.html
And yes, there is often no need to call something open explicitly, if it really is. Is into OpenOS, or just Linux?
I can’t say I blame them, when they realize they are sitting on the technological equivalent of a mountain of gold. What would you do?
Greed is not justified. I get that people are weak, selfish, they can't stop themselves. Some feel sympathy because they've been weak too. "Maybe it's justified," they like to think. "Everybody lies." But seriously, those who care so much about money and power they can't do things in a civilized respectable way: they are not yet an adult and must be hard barred from the upper tiers of capitalism until they learn that life does not revolve around them.
I blame them for being shitty, and blame everyone around them for letting it happen.
hmm i dont love this. either OpenAI has implicitly promised to monitor all its users, or has adopted a "report TOS violations to us when they happen and we will judge" stance. neither are great roads to go down.
I am not sure if this particular service solves the problem for them in any way, but to my ear it sounds like there is a need for code and infrastructure reuse in the data scientists domain that is ripe for innovation.
Truth about the situation: - Transformers generalize well and don't need much fine tuning - OpenAI can probably fine tune for your use case better than you can - Getting new models into production takes 6 months to a year at companies of this size, if you did have Data Scientists in house, it might just be better to go with a solution like this for velocity - Not every company has the talent to make an in house ML program successful.
I'd say they're perfect candidates for the API as a service model.
The essence of open source is that the resources are made available to you (without warranty). That isn’t the case here.
Just wanted to add: It’s amazing all the negativity in this discussion. Whatever happened to the creative tech community who loves to push boundaries? Isn’t that still part of the hacker ethos, isn’t this still hacker news? Just because a tool has the potential to be used for bad doesn’t mean we shouldn’t be excited to find new ways to use it for good.
Greed is not rational. It's just rationalized.
Yes all of them are doing facial recognition research, except... OpenAI, so how exactly is OpenAI used as a scapegoat to be able to do that kind of research without public backlash?
> By directing their cash through OpenAI they can avoid whatever fallout comes from unleashing things like GPT3 on the world.
GPT-3 si not unethical research. It is what you decide to with it and how you decide to release it that can potientially be unethical.
Also, OpenAI is just ahead of other labs because they have an insane compute budget and really talented people, but if you have been following a little bit the NLP news, you will see that your theory of OpenAI being a front for unethical research just makes no sense. OpenAI release GPT-2, 1.5B billions parameters, then NVIDIA realeased Megatron, 8B parameters, Google released T5 at 11B and recently Microsoft did turing-nlg at 17B. So they are clearly working on this in their own names and very much publicizing their work.
Signal is a needle in a haystack. Its not worth trying to keep fixing and reshaping the haystack so we dont keep loosing the needle. Lets just admit this tool isnt working and move on to better alternatives.
Edit: Clarifications (Need for post-editing also supports my point btw. )
Basically everyone on their death bed says focus on the experience, not the material. And everyone who does it agrees.
I'm obviously biased since I work at an industry AI lab, but we both have important roles to play.
The paper evaluated Winograds: https://arxiv.org/pdf/2005.14165.pdf#page=16
But with the speed of the field, maybe we can figure it out in three years. It just seems like we're still missing some key components. Primarily, reasoning and learning causality.
'an' is only mean to proceed a vowel. Should say
"OpenAI technology, just a HTTPS call away"
I remember coming across it not too long ago and felt unwelcomed/disappointed.
I want to create a software that can generate new code given business case hints, by studying existing open source code and their documentation.
I know this is vague, but sounds like what we eventually want for ourselves right?
As a European I am much more used to diversity meaning national or religious diversity. Still getting used to this notion of it mainly been used in the context of skin color in US.
From what I understand, its not just that the GPT-3 has impressive performance but more what is signifies and that is the fact that massive scaling didn't produce diminishing return, and if this pattern persists, it can get them to the finish line.
many people have different definitions for AGI though. for me it clicked when i realized that text has this universality property of capturing any intent.
If you really probe at GPT, you'll see anything that goes beyond an initial sentence or two really starts to show how it's purely superficial in terms of understanding & intelligence; it's basically a really amazing version of Searle's Chinese room argument.
> Exactly what terms Microsoft and OpenAI have agreed on with this $1 billion investment isn’t clear.
https://www.theverge.com/2019/7/22/20703578/microsoft-openai...
> Researchers will be strongly encouraged to publish their work, whether as papers, blog posts, or code, and our patents (if any) will be shared with the world.
I'm not sure if it's ever been publicly elaborated on.
"I took the water bottle out of the backpack so that it would be [lighter/handy]"
What is lighter and what is handy? No amount of stochastic language manipulation gets you the answer, you need to understand some rudimentary physics to answer the question, and as a precondition, you need a grammar or ontology.
It sounds like you're saying "It doesn't work because it can't work", but you haven't actually shown that it doesn't work.
In the example above it guesses wrongly, but again this is not surprising because it can't possibly get the right answer (other than by chance). The solution here cannot be found by correlating syntax, you can only answer the question if you understand the meaning of the sentence. That's what these schemas are constructed for.
edit: Retracted a test where it seemed to know which to select, because further tries revealed it was random.
edit: I did some more tries, and it does seem to be somewhat random, but the way it continues the sentence does seem to indicate that it has some form of operational model. It's just hard to prompt it in a way that it is "forced" to reveal which of the two it's talking about. Also, it seems to me its coherence range is too short in GPT-2. I would love to try this with GPT-3.
I also would contend that there is reasoning happening and that zero-shot demonstrates this. Specifically, reasoning about the intent of the prompt. The fact that you get this simply by building a general-purpose text model is a surprise to me.
Something I haven't seen yet is a model simulate the mind of the questioner, the way humans do, over time (minutes, days, years).
In 3 years, I'll ping you :) Already made a calendar reminder
I look forward to this ping :)
