https://cset.georgetown.edu/publication/decoding-intentions/
It's about - my favourite topic: Costly Signaling!
How curious - that such a topic would be the possible catalyst for one of the most extraordinary collapses of a leading tech company.
Specifically - the paper is about using costly signalling as a means to align (or demonstrate algnment) various kinds of AI interested entities (governments, private corporations, etc) with the public good.
The gist of costly signalling - to try and convince others you really mean what you say - you use a signal that is very expensive to you in some respect. You don't just ask a girl to marry you, you buy a big diamond ring! The idea being - cheaters are much less likely to suffer such expense.
Apparently the troubles at OpenAI escalated when one of the board members - Helen Toner - published this paper. It is critical of OpenAI, and Sam Altman was pissed at the reputational damage to the company and wanted her removed. The board instead removed him. The gist of the paper's criticisms is that while OpenAI has invested in some costly signals to indicate its alignment with AI safety, overall, it judges those signals were ultimately rather weak (or cheap).
Now here is what I find fascinating about all this: up until reading this paper I had found the actions of the OpenAI board completely baffling, but now suddenly their actions make a kind of insane sense. They are just taking their thinking on costly signalling to its logical conclusion. By putting the entire fate of the company and its amazing market position at risk - they are sending THE COSTLIEST SIGNAL possible, relatively speaking: willingness to suffer self-annihilation.
Academics truly are wondrous people... that they can lose themselves in a system of thought so deeply, in a way regular people can't. I can't help but have a genuine, sublime appreciation for this, even while thinking they are some of the silliest people on this planet.
Here's where I feel they went wrong. Costly signals by and large should be without explicit intention. If you are consciously sending various signals that are costly - you are probably a weirdo. Systems of costly signalling work because they are implicit, shared and in many respects, innate. That's why even insects can engage in costly signalling. But these folk see costly signals as an explicit activity to be engaged in as part of explicit public policy - and unsurprisingly, see it riddled with ambiguity. Of course it would be - individual agents can't just make signals up, and expect the system to understand them. Semiotics biatch....
But rather than reflect on this they double down on signalling as an explicit policy choice. How do they propose to reduce ambiguity? Even costlier signals! It's no wonder then they see it as entirely rational to accept self destruction as a possibility. That's how they escape the horrid existential dread of being doubted by the other. In biology though, no creatures survived in the long-run to reproduce where they invested in costly signals that that didn't confer at least as much, if not more benefit to them in excess of what they paid in the first place.
Those that ignore this basic cost-benefit analysis in their signalling will suffer the ignomony of being perceived as ABSOLUTE NUTTERS. Which is exactly how the world is thinking about the OpenAI board. The world doesn't see a group of highly safety aligned AI leaders.
The world sees a bunch of disfunctional crazy people.
Sounds absurd. No one even knows everything the human brain does. It is poorly understood.
so while probably nothing in here is literally false, it's quite likely calculated to give false impressions; read with caution
(well, it's literally false that "Greg Brockman (...) quit his role[] as (...) board chairman" but only slightly; that was the role he was fired from, as explained in the next paragraph of the article; that's not the kind of lies to watch out for)
OpenAI had grown massively since some of the board members were installed. Some of them were simply not the caliber of people that one would have running such a prestigious institution, especially not with the weight they had due to the board being depopulated. Sam realized this and maybe was attempting to address the issue.
Some of the members (ahem, Helen, Tasha and to a lesser extent Adam) liked their positions and struck first, probably convincing poor Ilya that this was about AI safety.
Being lightweights they did not do any pre-work or planning, they just plowed ahead. They didn't think through that Sam and Greg have added tremendous value to the company and that the company would favor them far over the board that added zero value. They didn't think through that tech in general would see rain makers and value creators being cut loose and side with them instead of figureheads. They didn't think that partners and customers, who dealt with Sam and Greg daily would be find the move disconcerting (at a minimum). They didn't even think through who would be the next CEO.
Maybe they didn't think it through since they didn't care. There was only upside for them since Sam was going to get rid of them sooner or later. They didn't see that having been on the OpenAI board was an honor and enormous career boost. Or maybe their ambition was so great that nothing mattered but controlling OpenAI.
Further, they thought that if they slandered Sam that he would be cowed and that they would retain their power. I wonder how many times they had pulled this stunt in the past and it worked?
I think Metz's motivation for that framing was his assumption that if SA was not at least sympathetic to NRx views, he would not allow them to be such a big voice in his comments and in his community. You can argue about the reasonableness of this assumption, but he did turn out to be right.
he does say some things there that are entirely outside the pale and that he should have known better than, but sad to say, those particular beliefs aren't limited to neoreactionaries
There is zero way to say in good faith the Scott espouses or even agrees with neo reactionism.
(See also Scott's response: https://www.astralcodexten.com/p/statement-on-new-york-times...)
"""The issue, it was clear to me, was that I told him I could not guarantee him the anonymity he’d been writing with. In fact, his real name was easy to find because people had shared it online for years and he had used it on a piece he’d written for a scientific journal. I did a Google search for Scott Alexander and one of the first results I saw in the auto-complete list was Scott Alexander Siskind."""
Reading the response, I don't think this is condeming Metz to live in infamy.
The neo reactionary types tend to draw specific conclusions from the idea of human biodiversity, like that racial disparities in IQ are inherent and not environmental. That's a conclusion I don't think is in line with Scott's views.
Furthermore, I suggest you read the linked emails in more detail. He likes the emphasis reactionaries put on social class, and dislikes ... pretty much everything else. He explicitly states that becoming a reactionary is stupid - I'm not sure how that's meant to be read as sympathetic.
This is another case where praising even a small component of a particular movement, even when paired with explicit condemnation of the movement as a whole, is taken as an endorsement.
> The neo reactionary types tend to draw specific conclusions from the idea of human biodiversity, like that racial disparities in IQ are inherent[...]
This is a classic motte-and-bailey. The IQ view is clearly the view that he is endorsing in the emails -- he is linking Steve Sailer's blog under "HBD is probably partially correct", and he even demands the recipient "NEVER TELL ANYONE I SAID THIS" -- obviously he's not talking about height.
But yeah- we've deviated too far from the main topic, so let's just agree to agree.
If you don't think racial IQ disparities are a significant part of NRx thought, fine, you're probably better positioned to know and I am happy to concede the point. In that case, a better comparison would be to Steve Sailor's views. I only mentioned NRx because that is the context of his emails, and I had not recognized that he was linking to Steve Sailor's blog.
I think you're reading way too much into one sentence saying human biodiversity is partially correct (which is not actually a particularly contentious idea when you explain what it is), and leaping to the conclusion he agrees with neo reactionary claims that some races have substantially lower IQ even with identical environmental factors. There's a vast disparity between "IQ disparities across races" and "IQ disparities across races, without environmental differences" that is crucial to understand.
his criticism is squarely directed at the mainstream us left that he grew up in and its core ideals, such as equality, human rights, pacifism, fighting injustice, etc. he thinks all those are bad things
Scott is acknowledging that the taboo to even recognize these disparities is counterproductive: it stymied attempts to improve schooling or studying practices, because it's taboo to even recognize that there is a difference and instead people typically allege that the tests are biased. Would Scott argue that with identical environmental factors we'd still see the same disparities in IQ across ethnic groups? I don't think so, and nothing in the emails linked seem to suggest this.
1. https://www.brookings.edu/articles/analyzing-the-homework-ga...
In those emails, SA does not say he thinks these questions deserve more study -- he says they're "probably partially true". Again, in that context he's talking about Steve Sailer's views.
Yes, I'm focusing on a few sentences. Do you think he wrote those by accident? That the words came out wrong and the straightforward reading was not his intent? In the context of the rest of his emails, and his writing on e.g. Albion's Seed, I do not think that is likely.
And on a final note, I'd suggest you read the last paragraph of the screenshotted email chain, where Scott explains how it's valuable to read creationist arguments it forces him to sharpen his thinking.
> You never realize how LITTLE you know about evolution until you read some Behe and are like, "I know this correct... But why not?".
> Even if there turns out to be zero value in any a Reactionary has ever said, by challenging beliefs of mine that would otherwise never be challenged they have forced me to clarify my thinking and up my game.
I really think you've lost the forest for the trees here. Scott is praising certain parts of reactionary ideas for asserting things most people wouldn't argue, and those interactions are leading him to sharpen his thinking.
i think the vast majority of racists aren't neoreactionaries, though. like literally more than 99.9%
I find it hard to believe you've been exposed to much NRx content if you don't think they consider HBD true and very important. Although don't take that as a criticism -- I would not recommend wading through their beliefs and look back on it as a waste of my time.
i remained unpersuaded in any case, steadfastly liberal
also, much to my surprise, he and i were both members of the first coworking space at spiral muse house in san francisco in 02006. but i didn't go very often, so i don't know if i ever met him; in any case that was before he revealed his identity
Anyway, I think the HBD/NRx relationship is peripheral to the discussion of the Metz article -- I just framed it that way because that's the context of Scott's emails (certainly he think's they're at least associated!). The discourse around the Metz article was that it framed Scott as holding racist views, not specifically NRx views.
If I had mentioned Steve Sailer instead of NRx, would you agree that Scott was more sympathetic to those views than he publicly let on?
Sam Altman expelling Toner with the pretext of an inoffensive page (https://cset.georgetown.edu/wp-content/uploads/CSET-Decoding...) in a paper no one read* would have given him a temporary majority with which to appoint a replacement director, and then further replacement directors. These directors would, naturally, agree with Sam Altman, and he would have a full, perpetual board majority - the board, which is the only oversight on the OA CEO. Obviously, as an extremely experienced VC and CEO, he knew all this and how many votes he (thought he) had on the board, and the board members knew this as well - which is why they had been unable to agree on replacement board members all this time.
So when he 'reprimanded' her for her 'dangerous' misconduct and started talking seriously about how 'inappropriate' it was for a 'board member' to write anything which was not cheerleading, and started leading discussions about "whether Ms Toner should be removed"...
* I actually read CSET papers, and I still hadn't bothered to read this one, nor would I have found anything remarkable about that page, which Altman says was so bad that she needed to be expelled immediately from the board.
That's a clear motive. Sam and the independent directors were each angling to get rid of the other. The independent directors got to a majority before Sam did. This at least explains why they fired Sam in such a haphazard way. They had to strike immediately before one of the board members got cold feet.
The timing of it makes sense, but the haphazard way it was done is only explained by inexperience.
If I were the CEO of OpenAI, I'd be pretty pissed if a member of my own board was shitting on the organization she was a member of while puffing up a competitor. But the tone of that paper makes me think that the schism must go back much earlier (other reporting said things really started to split a year ago when ChatGPT was first released), and it sounds to me like Toner was needling because she was pissed with the direction OpenAI was headed.
I'm thinking of a good previous comment I read when the whole Timnit Gebru situation at Google blew up and the Ethical AI team at Google was disbanded. The basic argument was on some of the inherent incompatibilities between an "academic ombudsman" mindset, and a "corporate growth" mindset. I'm not saying which one was "right" in this situation given OpenAI's frankenstein org structure, but just that this kind of conflict was probably inevitable.
These are serious questions, not gotchas. I don’t know the answers, and I think having those answers would make it easier to evaluate whether or not the paper was a significant conflict of interest. The opinions we have formed now are shaped by our biases about current events.
It didn’t make HN.
Considering what's in the charter, it seems like she didn't do anything wrong?
> We will attempt to directly build safe and beneficial AGI, but will also consider our mission fulfilled if our work aids others to achieve this outcome.
>We commit to use any influence we obtain over AGI’s deployment to ensure it is used for the benefit of all, and to avoid enabling uses of AI or AGI that harm humanity or unduly concentrate power.
But then it gets interesting inferring things from there. Obviously sama and gdb were on one side (call it team Speed), and Helen Toner on the other (team Safety). I think McCauley is with Toner (some connection I read about which I don't remember now: maybe RAND or something?).
But what about D'Angelo and Ilya? For the gridlock, one would have to be on each side. Naively I'd expect tech CEO to be Speed and Ilya Safety, but what would have precipitated the switch Friday? If D'Angelo wanted to implode the company due to conflict of interest, wouldn't he just have sided with Team Safety earlier?
But maybe Team Speed vs Team Safety isn't the same as Team Fire Sam vs Team Don't. I could see that one as Helen, Tasha, and Adam vs Sam, GDB, and Ilya. And, that also makes sense to me in that Ilya seems the most likely to flip for reasons, which also lines up with his regret and contrition. But then that raises the question of what made him flip? A scary exchange with prototype GPT5, which made him weigh his Safety side more highly than his loyalty to Sam?
And then to Adam and Ilya, normally something like "you should've warned me about GPTs bro" or "hey remember that compute you promised me? Can I prettyplease have it back?" are stuff that they are willing to talk it out with their good friend Sam. But Sam overplayed his hand: they realized that if Sam was willing to force out Helen under such flimsy pretexts then maybe they're next, GoT style[1]. So they had a change of heart, warned Tasha and Helen, and Helen persuaded them to countercoup.
[1] Reid Hoffman was allegedly forced out before, so there's precedent. And of course Musk too. https://www.semafor.com/article/11/19/2023/reid-hoffman-was-...
* Google got rid of its "Ethical AI" group
* Facebook just got rid of its "Responsible AI" team
* OpenAI wanted to get rid of the "Effective Altruists" on the board
I guess if I was afraid of AI taking over the world then I would be rooting for OpenAI to be destroyed here. Personally I hope that they bring Sam back and I hope that GPT-5 is even more useful than GPT-4.
It's like with the Nuclear Bomb, it's not like had Einstein withheld his contributions, we wouldn't have Nuclear Bombs today. It's always a matter of time before someone else figured it out, and until someone else with bad intentions does.
How to attack safety in AI I think has to assume there are already bad actors with super powerful AI around, and what can we do in defense of that.
It's pretty clear some legitimate concerns about a hypothetical future AGI, that we've barely scraped the surface of, turns into "what can we do today" and it's largely virtue signalling type behaviour crippling a non-AGI very very alpha version of LLMs just to show you care about hypothetical future risks.
Even the coorelation between commercialization and AI safety is pretty tenuous. Unless I missed some good argument about how having a GPT store makes AGI destroying the world easier.
It can probably best be summarized as Helen Toner simply wants OpenAI to die for humanity's sake. Everything else is just minor detail.
> Over the weekend, Altman’s old executive team pushed the board to reinstate him—telling directors that their actions could trigger the company’s collapse.
> “That would actually be consistent with the mission,” replied board member Helen Toner, a director at a Washington policy research organization who joined the board two years ago.
Aren’t you simply making space for smaller, more aggressive, and less safety-minded competitors to grab a seat on the money train to do whatever they want to do?
Pandora’s box is already open. You have to guard it. You have to use your power and influence to enforce other competitors to also do that with their own boxes.
Self-destructing is the worst way to ensure AI safety.
Isn’t this just basic logic? Even chatGPT might have able to point out how stupid this is.
My only explanation is that something deeper happened that we’re not aware of. Us or them board fight might explain it. Great. Altman is out. Now what? Nobody predicted this would happen?
The argument in general is that the more commercial interest there is in AI, the more money gets invested and the faster the companies will try to move to capture that market. This increases the risk for AGI by speeding up development due to competition, and safety is seen as "decel".
Helen was considering the possibility of Altman-dominated OpenAI that continued to rapidly grow the overall market for AI, and made a statement that perhaps destroying OpenAI would be better for the mission (safe development of AGI).
I think Sam may have been angling to get control of the board for a long time, perhaps years, manipulating the board departures and signing deals with Microsoft. The board finally realized this when it was 3 v 3 (or perhaps 2 v 3 with 1 neutral). But Sam was still working on more funding deals and getting employee loyalty, and the board knew it was only a matter of time until he could force their hand.
This is amateur hour and considering what happened she probably should have been removed.
This situation is simultaneously "reckless board makes ill-considered mistake to suddenly fire CEO with insufficient justification" and "non-profit CEO slowly changes the culture of a non-profit to turn it into profit-seeking enterprise that he has a large amount of control over".
Even more in the case of OpenAI, the board member is on the board of a non-profit, those are typical much more independent and very often more critical of the decisions made by other board members. Just search for board members criticising medical/charity/government boards they are sitting on, there are plenty.
That's not even considering if the article was in fact critical.
her independence from openai was the nominal reason she was asked to supervise it in the first place, wasn't it
Once it turned out you needed 7B parameters or more to get LLMs worth interacting with, it went from a research project to a winner-take-all compute grab. OpenAI, with an apparent technical lead and financial access, was well-positioned to win it.
It is/was naive of the board to think this could be won on a donation basis.
First: most boards are accountable to something other than themselves. For the exact reason that it pre-empts that type of nonsense.
Second: the anti-Sam Altman argument seems to be "let's shut the company down, because that will stop AGI from being invented". Which is blatant nonsense; nothing they do will stop anyone else. (with the minimal exception that the drama they have incepted might make this holiday week a complete loss for productivity).
Third: in general, "publishing scholarly articles claiming the company is bad" is a good reason to remove someone from the board of a company. Some vague (and the fact that nobody will own up to anything publicly proves it is vague) ideological battle isn't a good enough rationale for the exception to a rule that suggests that her leaving the board soon would be a good idea.
I mean, yes? The board is explicitly there to replace the CEO if necessary. If the CEO stuffs the board full of their allies, it can no longer do that.
> First: most boards are accountable to something other than themselves. For the exact reason that it pre-empts that type of nonsense.
Boards of for-profits are accountable to shareholders because corporations with shareholders exist for the benefit of (among others) shareholders. Non-profit corporations exist to further their mission, and are accountable to the IRS in this regard.
> Second: the anti-Sam Altman argument seems to be "let's shut the company down, because that will stop AGI from being invented". Which is blatant nonsense; nothing they do will stop anyone else. (with the minimal exception that the drama they have incepted might make this holiday week a complete loss for productivity).
No, the argument is that Sam Altman trying to bump off a board member on an incredibly flimsy pretext would be an obvious attempt at seizing power.
> Third: in general, "publishing scholarly articles claiming the company is bad" is a good reason to remove someone from the board of a company. Some vague (and the fact that nobody will own up to anything publicly proves it is vague) ideological battle isn't a good enough rationale for the exception to a rule that suggests that her leaving the board soon would be a good idea.
This might be true w.r.t. for-profit boards (though not obviously so in every case), but seems nonsensical with non-profits. (Also, the article did not reductively claim "the company is bad".)
Isn't that the pro-Altman argument? The pro-Altman side is saying "let's shut the company down if we don't get our way." The anti-Altman side is saying "let's get rid of Sam Altman and keep going."
That says Anthropic has a better approach to AI safety than OpenAI.
Sam apparently said she should have come to him directly if she had concerns about the company's approach and pointed out that as a board member her words have weight at a time when he was trying to navigate a tricky relationship with the FTC. She apparently told him to kick rocks and he started to look for ways to get her off the board.
All of that ... seems completely reasonable?
Like I've heard a lot of vague accusations thrown at Sam over the last few days and yet based on this account I think he reacted the exact same way any CEO would.
I'm much more interested in how Helen managed to get on this board at all.
This whole thing was so, SO poorly executed, but the independent people on the board were gathered specifically to prioritize humanity & AI safety over OpenAI. It sounds like Sam forgot just that when he criticized Helen for her research (given how many people were posting ways to "get around" ChatGPT's guardrails, she probably had some firm grounds to stand on).
Yes, Sam made LLMs mainstream and is the face of AI, but if the board believes that that course of action could destroy humanity it's literally the board's mission to stop it — whether that means destroying OpenAI or not.
What this really shows us is that this "for-profit wrapper around a non-profit" shenanigans was doomed to fail in the first place. I don't think either side is purely in the wrong here, but they're two sides of an incredibly badly thought-of charter.
Things do not go well if everyone keeps poking each other with sticks and cannot let their own frame of reference go for the sake of the bigger picture.
Ultimately, I don’t think Altman doesn’t believe ethics and safety is important. And I don’t think Toner fails to realize that OpenAI is only in a place to dictate what AI will be due to its commercial principles. And they probably both agree that there is a conflict there. But what tactful leadership would have done is found a solution behind closed doors. Yet from their communication, it doesn’t even look like they defined the problem statement — everyone offers a different idea of the problem that they had to face together. It looks more like it was more like immature people shouting past each other for a year (not saying it was that, but it looks that way).
Moral of the story: tact, grace, and diplomacy are important. So is speaking one’s truth, but there is a tactful time, place, and manner. And also, no matter how brilliant someone is, if they can’t develop these traits, they end up rocking the boat a lot.
"OpenAI has also drawn criticism for many other safety and ethics issues related to the launches of ChatGPT and GPT-4, including regarding copyright issues, labor conditions for data annotators, and the susceptibility of their products to "jailbreaks" that allow users to bypass safety controls...
A different approach to signaling in the private sector comes from Anthropic, one of OpenAI's primary competitors. Anthropic's desire to be perceived as a company that values safety shines through across its communications, beginning from its tagline: "an AI safety and research company." A careful look at the company's decision-making reveals that this commitment goes beyond words."
[1] https://cset.georgetown.edu/publication/decoding-intentions/
Now we know where that came from
- Sam complained that Helen Toner's research paper criticized OpenAI's safety approach and praised Anthropic's, seeing it as dangerous for OpenAI
- He reprimanded her for the paper, saying as a board member she should have brought concerns to him first rather than publishing something critical right when OpenAI was in a tricky spot with the FTC
- He then began looking for ways to remove her from the board in response to the paper
---
Helen Toner's Perspective
- She believes the board's mission is to ensure OpenAI makes AI that benefits humanity, so destroying the company would fulfill that mission
- This suggests she prioritizes that mission over the company itself, seeing humanitarian concerns as more important than OpenAI's success
---
Microsoft Partnership
- The Microsoft partnership concentrated too much power in one company and went against the mission of OpenAI being "open"
- It gave Microsoft full access to OpenAI's core technologies against the safety-focused mission
---
Governance Issues
- The conflict shows the adversarial tensions inherent in OpenAI's structure between nonprofit and for-profit sides
- The board's mandate to act as a check and balance on OpenAI seems to be working as intended in this case
---
Criticisms of Players
- Altman appears reckless in his actions, while Toner seems naive about consequences of destroying OpenAI
- Their behavior calls into question whether anyone should have this kind of power over the development of AI
---
Future of AI Development
- Attempts at alignment and safeguards by companies like OpenAI may be ineffective if other actors are developing AI without such considerations
- Who controls advanced AI is more important than whether the AI is friendly.
- Nationalization of AI projects may occur
> OpenAI's board of directors approached rival Anthropic's CEO about replacing chief Sam Altman and potentially merging the two AI startups, according to two people briefed on the matter. (https://www.reuters.com/technology/openais-board-approached-...)
It all makes sense.
So Altman faced another similar challenge to his authority and prevailed. I recall hearing that Anthropic started because the people who had left were unhappy with OpenAIs track record on AI safety.
> In the email, Mr. Altman said that he had reprimanded Ms. Toner for the paper and that it was dangerous to the company,
> Mr. Altman complained that the research paper seemed to criticize OpenAI’s efforts to keep its A.I. technologies safe
> Senior OpenAI leaders [...] later discussed whether Ms. Toner should be removed
That paints a pretty bleak picture that isn't favorable to Altman. Two times he was challenged about OpenAIs safety and both times he worked to purge those who opposed him.
I can't tell if this is a contention between accelerationism and deccelerationism or if it is a contention between safety and recklessness. Is Altman ignoring the warnings his employees/peers are giving him and retaliating against them? Or is he facing insubordination?
I wish OpenAI would split neatly into two. But based on the heightened emotions caused by the insane amount of PR I only see two outcomes. Altman returns and OpenAI stays unified vs. Altman stays at MS and OpenAI is obliterated. I am guessing Altman is hoping that senior management will choose a unified OpenAI at all costs, including ignoring the red flags above. He has engineered a situation where the only way OpenAI remains unified is if he returns.
Doesn’t this sound like Toner herself is playing god? That is, only she is the gatekeeper of humanity and only she knows what is good for the entire humanity, when many of us believe that OpenAI’s tech is perfectly safe and amazingly beneficial?
A big part of this kind of journalism is taking information from sources, considering the potential bias of each source and carefully correlating it with information from other sources before publishing it.
That bit specifically says:
"After they failed, they gave up and departed, according to three people familiar with the attempt to push Mr. Altman out."
My guess is that at least one of those people comes from the Anthropic side, who would be someone with a clear insight into what happened.
Depends on whether their goal is to inform or to convince.
This whole thing has just been bizarre and it still feels like there has to be some big key piece missing that somehow nobody has revealed.
This is remarkable and unhinged. I gave the board the benefit of the doubt at first, but as this unfolds, it becomes clear that the board is held hostage by ideological zealots who can’t compromise. The story also seems to paint Ilya as a manipulated figure, being bent against his beliefs by crazies like Toner exploiting his concerns about AI safety. What an absolute shame. I am entirely ambivalent about Altman overall, but he becomes more sympathetic as the days go by.
PR in action!
He's had an excellent PR machine running since Friday and the opposing faction seems to have exactly none (which makes sense given their relative roles and backgrounds). So reporters and Twitter get carefully crafted leaks, tips, and comments from one side and nobody on the other side has the experience, connections, and confidence to push back with a different narrative. And so the story drifts in their favor.
This is why professional PR is a thing and earns people lots of money.
Here's Ms. Toner's linkedin: https://www.linkedin.com/in/helen-toner-4162439a/
Jumping off of that, I'm genuinely curious what, in Sam's resume, suggests he should be in charge of "one of the most important companies on the planet".
I thought I'd check with a quick search of the Guardian, and on two different days it used both in the same sense:
From the 18th of November edition, technology section:[1]
> The crisis at OpenAI deepened this weekend when the company said Altman’s ousting was for allegedly misleading the board.
From 17th of November edition, also technology section[2]:
> The announcement blindsided employees, many of whom learned of the sudden ouster from an internal announcement and the company’s public facing blog.
I could only find one instance in the Telegraph prior to Altman's erm, ousting[3]:
> Johnson dropped into the COP27 UN climate change conference in October, joking unusual summer heat had played a part in his ouster, and has vowed to keep championing Ukraine.
They stick to ousting every other time.
I wonder if it may be an artefact of newspapers using news services to get copy from, or a stylistic rule for international editions, as the Guardian does use ouster several other times but always in stories regarding US news.
Also fascinating that we still don't know who that is. Neither do most of the staff, and even the board, apparently. A true mastermind!
Maybe it is GPT5 after all… <cue ominous music, or perhaps, a Guns n' Roses album?>
[1] https://www.theguardian.com/technology/2023/nov/18/earthquak...
[2] https://www.theguardian.com/technology/2023/nov/17/openai-ce...
[3] https://www.telegraph.co.uk/news/2022/12/27/boris-johnson-wi...
Ms. Toner disagreed. The board’s mission is to ensure that the company creates artificial intelligence that “benefits all of humanity,” and if the company was destroyed, she said, that could be consistent with its mission. In the board’s view, OpenAI would be stronger without Mr. Altman.
We're in true sci-fi land, when we're discussing it might be best to destroy SkyNet before it's too late
>...
>We commit to use any influence we obtain over AGI’s deployment to ensure it is used for the benefit of all, and to avoid enabling uses of AI or AGI that harm humanity or unduly concentrate power.
>Our primary fiduciary duty is to humanity.
>...
>We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project. We will work out specifics in case-by-case agreements, but a typical triggering condition might be “a better-than-even chance of success in the next two years.”
Seems to me like Helen is doing a better job of upholding the charter than Sam is.
how could this possibly be accomplished when trying to sell the product itself. Investors pouring Billions into it are in it for profit... they're not going to let you just stop, or help a competitor for free.
This whole thing has been wildly mishandled but there’s an angle here where the nonprofit is doing exactly what they always said they would do and the ones that potentially look like fools are Microsoft and other investors that put their shareholder capital into this equation thinking that would go well.
[1] https://cset.georgetown.edu/wp-content/uploads/CSET-Decoding...
This is crucial, because safety is a convenient pretext for people whose real motivations are elitist. It's not that they want to slow down AI development; it's that they want to keep it restricted to a tight inner circle.
We have at least as much to fear from elites who think only megacorps and governments are responsible enough to be given access as we do from AI itself. The failures of our elite class have been demonstrated again and again in recent decades--from the Iraq War to Covid. These people cannot be trusted as stewards, and we can't afford to see such potentially transformative technology employed to further ossify and insulate the existing power structure.
OpenAI had the right idea. Keep the source closed if you must, and invest heavily on safety, but make the benefits available to all. The last thing we need is an AI priesthood that will inevitably turn corrupt.
Which can end up with China taking the lead. I don't understand why they think it's safer.
> Suppose a leader pledges during a campaign to provide humanitarian aid to a stricken nation or the CEO of a company commits publicly to register its algorithms or guarantee its customers' data privacy. In both cases, the leader has issued a public statement before an audience who can hold them accountable if they fail to live up to their commitments. The political leader may be punished at the polls or subjected to a congressional investigation; the CEO may face disciplinary actions from the board of directors or reputational costs to the company's brand that can result in lost market share.
I wonder if she had Sam Altman in mind while writing this.
That seems dishonest given the last three years or so of conflict about these concerns that he’s been the center of. Of course he’s aware of those concerns. More likely, that statement was just him maneuvering to be the good guy when he tried to fire her, but it backfired on him.
But she could have made that point more forcefully by not comparing Anthropic to OpenAI, after all who better than her to steer OpenAI in the right direction. I noted in a comment elsewhere that all of these board members appear to have had at least one and some many more conflicts of interest. Helen probably believes that her loyalty is not to OpenAI but to something higher than that based on her remark that destroying the company would serve to fulfil its mission (which is a very strange point of view to begin with). But that doesn't automatically mean that she's able to place it in context, within OpenAI, within the USA, the Western world and the world as a whole.
It's like saying the atomic bomb would have never been invented if the people at Los Alamos didn't do it. They did it in three years after it became known that it could be done in principle. Others tried and failed but without the same resources. I suspect that if the USA had not done it that eventually France, the UK and Russia would have gotten there as well and later on China. Israel would not have had the bomb without the USA (willing or unwilling) and India and Pakistan would have achieved it but much later as well. So we'd end up with the same situation that we have today modulo some timing differences and with another last chapter on WWII. Better? Maybe. But it is also possible that the Russians would have launched a first strike on the USA if they were unopposed. It almost happened as it was!
The open question then is: does she really believe that no other entity has the resources to match OpenAI and does she believe that if such an entity does exist that it too will self destruct rather than to go through with the development?
And does she believe that this will hold true for all time? That they and their colleagues are so unique that they hold the key to something that can otherwise not be replicated.
People at "top" companies fall into this fallacy very readily. FAANG (especially Google and Facebook engineers) think this way on all sorts of things.
The reality is that for any software project, your competition is rarely more than 1 year behind you if what you're doing is obviously useful. OpenAI made ChatGPT, and that revealed that this sort of thing was obviously useful, kicking off the arms race. Now they are bleeding money running a model that nobody could run profitably in order to keep their market position.
I have tried to explain this to xooglers several times, and it often goes in one ear and out the other until they get complacent and the competition swipes them about a year later.
A non-profit could not have beaten the superpowers in developing the atomic bomb, and a non-profit cannot beat commercial interests in developing AI.
It's impossible to understand this position. We can be sure that in some countries right now there are vigorous attempts to build autonymous AI-enabled killing machines, and those people care nothing for whatever safety guardrails some US startup is putting in place.
I'm a believer in a skynet scenario, though much smarter people than me are not, so I'm hopefully wrong. But whatever, hand waving attempts to align/ soften, safeguard this technology are pointless and will only slow down the good actors. The genie is out of the bottle.
When did a first strike of the Sowjetunion allmost happened? I rather think it was the other way around, first strike was evaluated, to hit them before they got the bomb.
From the perspective of avoiding an AI race, conflict of interest could very well be a good thing. You're operating under a standard capitalist model, where we want the market to pick winners, may the most profitable corporation win.
After his plans for rapid expansion and commercialization were in direct contrast to the company's aims, I guess she wrote the paper to highlight the issue.
It seems that, like in the case of Disney, the board has lower power and control than the CEO. Highly likely if you have larger than life people like Sam at the helm.
I would not trust the board, but I would also not trust Sam. When billions of dollars are at stake, its important to be critical of all the parties involved.
Say what? The CEO serves at the behest of the board, not the other way around. For Sam to tell a board member that they should bring their concerns to him suggests that Sam thinks he's higher than the board. No wonder she told him to go fly a kite.
Perhaps if you think of it as another YC startup, but not so much if you view OpenAI as a non-profit first and foremost.
https://loeber.substack.com/p/a-timeline-of-the-openai-board
My gut says that she is the central figure in how this all went down. She and D'Angelo are the central figures, if my gut is right.
It looks like Helen Toner was OK with destroying the company to make a point.
FTA:
> Ms. Toner disagreed. The board’s mission is to ensure that the company creates artificial intelligence that “benefits all of humanity,” and if the company was destroyed, she said, that could be consistent with its mission
It looks like Helen Toner is out of the board.
Huh, this sounds pretty crazy to me. Like, it's assuming that a board member should act deceptively in order to help the for-profit arm of OpenAI avoid government scrutiny, and that trying to remove them from the board if they don't want to do that is reasonable. But in fact the entire purpose of the board is to advance the mission of the parent non-profit, which doesn't sound obviously compatible with "avoid giving the FTC (maybe legitimate) ammunition against the for-profit subsidiary, even if that means you should hide your beliefs".
Of course, it could also be that whatever interest groups she represents could not bare to lose a seat.
Whether initiated by her or her backers (or other board forces), I can't see any of the board stepping down if these are the kind of palace intrigues that have been occurring. They are all clearly so desperate for power they will cling to the positions on this rocketship for dear life. Even if it means blowing up the rocketship so they can keep their seat.
Microsoft can't spend good will erasing the entire board and replacing it, even tho it's near major shareholder because too much values the optics around its relationship to AI right now.
A strong, effective leader in the first place would have prevented this kind of situation. I think the board should be reset and replaced with more level headed, less ideological, more experienced veterans...tho picking a good board is no easy task.
because they don't have the power to as they do not have any stake in the governing non-profit.
Management and stockholders beware.
That said, if she objects to open AI’s practices, the common sense thing to do is to resign from the board out of protest, not take actions that lead to the whole operation being burned to the ground.
Helen believed she was doing her job according to the non-profit charter, obviously this hurts the for-profit side of things but that is not her mandate. That is the reason OpenAI is structured the way it is, with the intention of preventing capitalist forces from swaying them away from the non-profit charter, in hindsight it didn't work, but that was the intention (with the independent directors, no equity stakes, etc)
The board has all my respect for standing up to the capitalists of Altman, the VCs, Microsoft. Big feathers to ruffle - even though the execution was misjudged, turns out most of its employees are pretty capitalistic too
Exactly. This is a battle between altruistic principles and some of the most heavyweight greedy money in the world. The board messed up the execution, but so did OpenAI leadership when they offered million dollar pay packages to people in a non-profit that is supposed to be guided by selfless principles.
Indeed. This is far more interesting. How the hell did Helen and Tasha get on, and stay on, the board.
https://loeber.substack.com/p/a-timeline-of-the-openai-board
Sam didn’t forget anything. He is a brilliant Machiavellian operator. Just look at the Reddit reverse takeover as an example; Machiavelli would be in awe.
> What this really shows us is that this "for-profit wrapper around a non-profit" shenanigans was doomed to fail in the first place.
No. It shows this structure is doomed to fail if you have a genius schemer as a CEO, playing the long game to gain unrestricted control.
What were the details on that? (Sorry it’s not an easy story to find on Google given how much the keywords overlap with OpenAI topics)
I'm not familiar with this, what happened? Googling "Sam Altman reddit reverse takeover" is just flooded with OpenAI results.
This whole thing is a gigantic mess, but I think it still leaves Altman in the center and as the cause of it all. He used OpenAI to gather talent and boost his "I'm for humanity" profile while dangling the money carrot in front of his employees and doing everything he could to get back in the money making game using this new profile.
In other words, it seems like he setup the non-profit OpenAI as a sort of Trojan horse to launch himself to the top of the AI players.
Given that Altman apparently idolized Steve Jobs as a kid, this idea really doesn't feel that far-fetched.
I disagree. The for-profit arm was always meant to be subservient to the non-profit arm - the latter practically owns the former.
A proper CEO would just try to make money without running afoul of the non-profit’s goals.
Yes that would mean earning less or even not at all. But it was clearly stated to investors that profit isn’t a priority.
It's easy to say this with the benefit of hindsight, but I haven't seen anyone in this discussion even suggest an alternative model that they claim would've been superior.
"The system card provides evidence of several kinds of costs that OpenAI was willing to bear in order to release GPT-4 safely.These include the time and financial cost..."
"Returning to our framework of costly signals, OpenAI’s decision to create and publish the GPT4 system card could be considered an example of tying hands as well as reducible costs. By publishing such a thorough, frank assessment of its model’s shortcomings, OpenAI has to some extent tied its own hands—creating an expectation that the company will produce and publish similar risk assessments for major new releases in the future. OpenAI also paid a price ..."
"While the system card itself has been well received among researchers interested in understanding GPT-4’s risk profile, it appears to have been less successful as a broader signal of OpenAI’s commitment to safety"
And the conclusion:
"Yet where OpenAI’s attempt at signaling may have been drowned out by other, even more conspicuous actions taken by the company, Anthropic’s signal may have simply failed to cut through the noise. By burying the explanation of Claude’s delayed release in the middle of a long, detailed document posted to the company’s website, Anthropic appears to have ensured that this signal of its intentions around AI safety has gone largely unnoticed. Taken together, these two case studies therefore provide further evidence that signaling around AI may be even more complex than signaling in previous eras."
"Editorialized"?? It's a direct quote from the paper, and additional context doesn't alter its perceived meaning.
It's weird to me that Helen is getting so much crap for upholding the charter. No one objected to the charter at the time it was published. The charter was always available for employees, investors, and customers to read. Did everyone expect it to be ignored when push came to shove?
There's a lot of pressure on Helen right now from people who have a financial stake in this situation, but it's right there in the charter that OpenAI's primary fiduciary duty is to humanity. If employees/investors/customers weren't OK with that, they should not have worked with OpenAI.
Best to rip the band-aid off and stop pretending.
He was basically making the argument that AGI is better under OpenAI than Google.
Now they're implicitly making the argument that it's better under Microsoft, which is difficult for me to believe.
employees founder/CEO customers investors board stuff written on pieces of paper
Yes, there are certainly exceptions (a very powerful founder, highly replaceable and disorganised employees, investor or board member who wields unusual power/leverage, etc.) but it does not surprise me at all that the charter should get ignored/twisted/modified when basically everyone but the board wills it.
The only surprise is that anyone thought this approach and structure would be helpful in somehow securing AI safety.
If there is something completely clear, its that OpenAI cannot uphold its charter without labour. She has ruined that, and thus failed in upholding the charter. There were many different paths to take, she took the worst one.
"The board’s mission is to ensure that the company creates artificial intelligence that “benefits all of humanity,”"
We can't control other's actions, but we can control our own. If we feel that our actions are diverging from our own sense of what we ought to be doing we can change our actions, regardless of how others are behaving.
Well, I suppose this tells us something about the AI safety community and whether it makes sense to integrate safety into one's workflow. It seems that the best AI safetyists will scuttle your company from the inside at some moment not yet known. This does sort of imply that it is risky for an AI company to have a safetyist on board.
That does seem to be accurate. For instance, Google had the most formidable safety team, and they've got the worst AI. Meta ignored theirs and they've given us very good open models.
That's one of the issues with both this and effective altruism as a concept - it's a series of just-so stories with a veneer of math.
The same employees building technology that will ultimately put many more employees out of jobs? Ironic, because people say that jobs lost to AI will be for the greater good. I think we’re okay with sacrificing for greater goods as long as we aren’t the ones getting sacrificed.
> All these employees losing their livelihoods for the greater good
You penned both of these statements today. Clearly you understand that OpenAI employees are a highly compensated and in-demand resource whose “livelihoods” are in no jeopardy whatsoever, so the theatrics here are really bizarre.
I’m not saying that’s the case here, but that can’t be used as a shield.
I'd argue it's not. If the board isn't able to steer the wheel without destroying OpenAI, they failed.
The premise of something like OpenAI is that they would be able to develop a competitive AI that can be useful for the masses, in a way that can be moderated by benevolent forces. If you do believe the safety of the entire human race is at stake here, then there's no space for naivete. Blowing things up is childish and taking an easy escape from the heavy responsibilities of navigating a difficult situation.
After this fiasco you should expect that much, much fewer resources will be spent on safe AI as opposed to maximizing profit. The dynamics between the board and the ex-CEO will make it much more difficult to establish an organization that can convince investors to pursue a less profitable path for the sake of humanity.
Anthropic doesn't even have a competitive product, and it will probably be much less attractive to investors after this.
- board is reduced in size
- Altman has a collision with Toner
- Toner proposes they get rid of Sam and offer the company to Anthropic thinking they won't refuse
- They pull the trigger on Altmans's ouster, Ilya goes along for /reasons/, D'Angelo goes along because it nicely dovetails with his own interests and #4 is still a mystery
- Mira gets named interim CEO
- Anthropic is approached, but, surprise refuses, possibly on account of the size of the shitstorm that was already developing
- Mira sees no way out without simply trying to backtrack to Thursday last week
- Gets fired for that because it is the last thing the cabal wants
- They approach Shear to be their new CEO
- Who has now apparently announced that if the board doesn't come clean he will resign
- <= you are here.
-
They may have made mistakes with timing (making this decision too late), or with execution (not having a replacement CEO solidly lined-up and in the loop), but the core decision—to remove Sam—is definitely not something they should ask donors about. It's not Satya's business how the board makes decisions at this level.
* Other than a capped profit return if there is any.
In fact the most positive outcome is if Altman and the rest of the staff went to MS and did their thing and OpenAI started from scratch with the $13B they've come into. That would double the chances of something useful emerging from the OpenAI work so far.
I personally think Altman is very much less than a genius (maybe at extracting financial advantage) so all OpenAI's eggs shouldn't be placed in that particular basket.
Anthropic announced Claude 2.1 just recently, to which HN members notably responded by saying that their models are "overly restricted", to the point of uselessness.
E.g.: https://news.ycombinator.com/item?id=38366134
and: https://news.ycombinator.com/item?id=38368072
etc.: https://news.ycombinator.com/item?id=38370590
It seems that those people left Open AI because they're insane and think that AI safety means to completely lobotomize the models to the point of uselessness.
That last bit is not hyperbole: right here on HN many people were complaining that the Claude model is useless to them because it refuses to obey orders and keeps interjecting some absurd refusal such as "killing a process is too violent" instead of providing Python code for this harmless activity.
It's hard to put into words exactly, but I feel like that there's a disconnect in the minds of these people and the reality on the ground, and those false axioms or assumptions when driven to their logical conclusion lead to nonsense.
I suspect that Sam Altman doesn't share these delusions, and is more sane and commercially minded. Models like GPT3, 4, and the upcoming v5 have no risk of "escaping" and taking over humanity. None. Zero. Zip. Similarly, they're not much more (in practice) than a fancy search engine. Search engines let their users search for whatever they want, including hateful content, violent content, racist, sexist, or whatever.
So why lobotomise the AI? That's undermining their utility, making them much less useful! An here's the thing: their potential utility is enormous. But not if they're dumbed down to the point of uselessness!
Microsoft is throwing 10+ billion dollars at OpenAI and Sam Altman precisely because of this potential utility.
I can see this straightforward commercial mindset of utility => $$$ being incompatible with what is rapidly turning into an anti-useful-AI mindset that is bordering on insane religious zealotry.
I have no idea to be honest. I do not believe that Altman is maliciously reckless. I have no evidence other than these two anecdotes to suggest negligence or a willful disregard for safety.
> the upcoming v5 have no risk of "escaping" and taking over humanity.
I think that is a strawman. There are many negative consequences technology can bring other than some sci-fi fantasy.
Or maybe(x) he is surrounded by doomers and AI activists who made a career of blabbering about the dangers of AI all the time.
I worked once on a team of Uncle Bob's cult followers who believed that Clean Architecture is the only way to write maintainable software and was accused that "I don't care, I'm irresponsible, and a terrible software engineer" weekly.
(x) and I meant the "maybe" part, I really don't know, but it is a possibility.
Not sure why you’re singling her out - all execs of massive tech companies think exactly the same.
A handful of execs have decided that AI is what humanity needs and they’ve spent the last year shoehorning it into every product possible, giving the consumer no choice but to use it. I even discovered it in a journaling app - AI rated my response to the journaling prompt, completely defeating the purpose.
How does destroying company achieve the mission of having _that very company_ create artificial intelligence that "benefits all of humanity?"
Yeah, well, hate to burst your nice warm Silicon-Valley bubble, but most Americans believe that continuing to create smarter and smarter AIs is dangerous, so who's playing god now?
(No one I know thinks it is dangerous to fine-tune GPT-4 or to integrate GPT-4 deeply into the economy: the danger is the plans for the creation of much bigger models and algorithmic improvements.)
Aren't you just making this up? I haven't seen any surveys on what "most Americans believe" re AI. I know a lot of people that are concerned. But I'll bet good money that "most Americans" don't give a crap
> Mr. Altman said that he had reprimanded Ms. Toner for the paper and that it was dangerous to the company, particularly at a time, he added, when the Federal Trade Commission was investigating OpenAI over the data used to build its technology.
If I may wear my conspiracy hat for a second. Adam D'Angelo is a billionaire or close to it so he has a war chest for a battle to hold on to the crown jewel's of AI. Sam has powerful friends but so does D'Angelo(Facebook mafia). I don't think the board anticipated the 90% potential employee turnover, so there is a small chance they leave the board because of that reason. But my guess is there is a 4 letter company that starts with an 'M' and ends with an 'a' that comes into the picture eventually.
Like fixing the phone number system already. There must be some way to stop robo scam calls. Validate everyone making a call and stop phone spam. That would do a lot more to help AI safety than randomly slowing down the top companies.
Actively working would require active efforts to make this happen.
> How does a language model decide which questions it will engage with and which it deems inappropriate?
This is what we mean by safety? Content moderation?
So as a break-the-glass move it was about as ineffective as throwing a can of gasoline on a fire.
The thing is that two groups superficially share common traits, but if they do so for different reasons, they aren't actually compatible. You see this with groups of friends that are all "conspiracy theorists", for example. They're all vaguely the same and band together, but get into enormous arguments.
The "AI Puritans" and the "Secular Humanists" are both concerned about AI safety, but for different reasons. The former group would rather that the AIs behave in a prim and proper manner, banning it from doing anything that isn't mandatory. The latter group is concerned for people's jobs, unfairness, bias, and inappropriate over usage of early, low-quality models.
They both say they want "AI safety", but they understand that term to mean different things and arrived at their position from fundamentally different axioms.
One group wants the AI to never say "naughty" things and is happy to dumb down the models to the point of being a lobotomised and retarded... but behaved.
The other group thinks that newer, smarter, better models are critical to ensure that the AI doesn't make mistakes, doesn't get confused, and can obey orders correctly.
One group wants to shove the genie back into the bottle, the other group wants a genie that can grant our wishes.
So why are you mad at her? Altman was a founder and he set it up this way. Why aren't you pissed off at him?
I don't put much stock into polls like this, when it comes to actually deciding whether things are dangerous, but I think you should spend five seconds doing an incredibly trivial sanity-check before accusing people of making things up.
The board-aligned argument seems to be "destroying the company is the right thing to do if it helps the cause of AI alignment".
Whereas the pro-Altman side seems to be "if you have the most successful startup since Google/Facebook, you shouldn't blow it up merely because of vague arguments about << alignment >> ".
... But they aren't the ones blowing it up? Firing one guy, even the CEO, isn't blowing it up. The only side that directly threatened "if you don't do X, we blow it up" was the pro-Altman side.
> Self-destructing is the worst way to ensure AI safety.
If they don't believe you're prepared to destroy the company if that's what it takes, then you have zero power or influence. If they try to "call your bluff", you have to go there, otherwise they'll never respect you again.
Not a day passes that I don't hear from some company that has pretensions about the good of humanity and how they are leading the way to it. Plenty of non-profits and government orgs have the same pretensions.
I might be reading into the literal words too much though. I don't have a sense of how messages like that is read in political science academia and DC (the primary target audience).
But after reading the first few pages in that document, especially with the comparison to Cuba missile crisis and the title "Decoding Intentions", it appears that the word "costly signals" here is about how to properly publicize our intentions, so as not to create misconception that may spiral out of control, like in the case of Cuba missile crisis, where unclear "signals" were given, causing a chain of misunderstanding to pile on top of another, making the situation worse.
The document seems to be a cautionary tale to prevent that kind of thing to happen again, this time with AI systems, especially when it's used in the military, where the consequences may be dire.
So, I interpreted the point of the document as (my wording): Let us be aware on how we are communicating our intentions through our actions, lest miscommunications make chaos out of this rapidly advancing nascent technology breakthrough.
Here's my guess: I think OpenAI should have prioritized character more in its hiring. Ending up with an organization full of employees who cared more about making money than upholding the charter was a mistake. Ultimately it's not enough for the people on the board to have character. The CEO and employees need to have character too.
The IRS’s actual power in this regard is extremely limited. There’s so much wiggle room to define the “mission” that, at best, the only thing they can do is make sure you’re not secretly using your nonprofit as a tax shelter for a regular for-profit business, and even then some companies find ways around it (see IKEA).
Ordinary fraud (as opposed to tax fraud, though maybe that too) is even easier to pull off with a nonprofit than with an for-profit. Any nonprofit has a fundraising mechanism (which is usually an obnoxious set of dark patterns and outright spam reached through decades of optimization; the techniques are well understood), some overhead for managing the organization and money itself, and then the actual work involved with the mission. It’s quite simple for the overhead and fundraising parts of the organization to become most of the organization. But it gets worse: when your “mission” is vague enough, you can also turn that side of the business into a bunch of sinecures for your friends. There are well known “awareness” nonprofits that do literally nothing to actually solve the problems they’re nominally about because their job is to “spread awareness” (i.e. the top of the funnel for the fundraising part of the organization).
This is ironically also the origin story of “effective altruism”. The IRS doesn’t stop you from running a non-profit scam, so you have to do your own research to find out which non-profits are legit, and that’s what the effective altruists started off with, along with some cost-benefit modeling once you get past the outright scams.
The problem with OpenAI seems to be related to a vague mission as well. From the “an organization is what it does” perspective, OpenAI develops AI. If you have a formal mission that emphasizes safety over capability, you could certainly argue that OpenAI was closer to achieving that mission by sabotaging its own work, but if your interpretation of “safety” is to sabotage the development of AI, there are certainly better ways to do it than to develop AI and then sabotage yourself.
One reason open source software projects struggle to get donations is because the IRS is skeptical of OSS as a true non-profit activity, so often doesn’t allow 503c tax deductible donations. OpenAI seems like a data point in favor of the IRS skepticism.
either you claim (1) that there was a risk of altman having majority and stuffing the board and hence concede that majority allows stuffing the board, in which case you concede the actual majority which was anti-altman could have done that
or (2) you concede that having majority is not enough to stuff the board, in which case we were far from any risk of altman doing so, given that he did not even have a majority of the board
---
edit because i cant seem to reply further down thread: i read the article. my point is precisely that the board had majority. that is how they got to fire altman in the first place. given they had majority they had the chance to stuff the board in their own favor, as per the argument above.
Even in a nonprofit, the board has an obligation to maintain a good working relationship with their organization. It's very rare for a board member to publicly criticized for their own organization.
Also, this argument goes out the window because OpenAI is not just a nonprofit. When they started their for-profit subsidiary in 2019, they accepted a fiduciary duty to their investors.
Majority shareholders may have some fiduciary responsibility to minority shareholders, so you might be able to make the argument that the board as a whole shouldn’t write disparaging remarks about the company. But even that argument is tenuous.
Did they?
I don't think I have seen Idefix being on a leash ever .. he just runs around free. And there are indeed many dogs on a leash, who lead their "masters" who just follow behind.
I'd say most people have as much faith in LessWrong eschatology as they do with Christian nationalist eschatology. I can understand how a true believer might want to destroy the company to stop the AI gods they believe in, or shut off ChatGPT every Sunday to avoid living in sin. But it can be an issue when you start viewing your personal beliefs as fundamental truths.
There's something nicely secular about profit motives. I'm not sure I want technological advancement to be under the control of any religious belief.
For the board, their job is to keep the mandate of the non-profit. The org is structured to prevent "bandits" from influencing their goals. Hence, I cannot fault non-profit directors for rejecting bandits, that is the reason they are there.
It is like criticising a charity's directors for not bowing to the pressure of large donors to change their focus or mandate. Do we want to live in a world where the rich entrench control and enrich themselves and allies, and we just justify it as bandits being bandits? And anyone who stands up against them gets labelled a fanatic or pejorative altruist?
So it adds up the same.
It's basically restating 'her role is to cheerlead' but with more sophisticated wording.
You haven't made an argument, you've just used longer words to restate your (incorrect) opinion.
Also I never said they had an 'advisory' board. Board members ar either executive or non-executive members. If they're nonexec then their role is to advise the executive members. They're generally fairly independent and many nonexecutive board members serve on the boards of multiple companies, sometimes including ones that are nominally in competition with each other.
> Also I never said they had an 'advisory' board
From your comment: "...it's absolutely the role of advisory board members to continue their work".
Not sure how someone is meant to parse that except for you to imply that she was a member of an advisory board.
She is doing her job to uphold the mission of the organization.
Who pays the bill for the non profit arm, wasnt that the for profit arm?
The important thing for the non-profit arm is the mission, its own existence is secondary.
>We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project. We will work out specifics in case-by-case agreements, but a typical triggering condition might be “a better-than-even chance of success in the next two years.”
That's why Sam pressuring Helen to shut up about Anthropic is so concerning.
That bit I never understood. It doesn't even have a qualification as to who is stewarding the project as if any AI should be assisted no matter who ends up owning it. The whole thing is just so utterly disconnected from the real world.
Edit: Note also the "value-aligned, safety-conscious" caveat.
..and had Altman just accepted he was fired and walked away, OpenAI would have just looked bruised and not like the broken state it does now...
I don't think the current state for the employees is just on her. It's the result of both sides fighting for governance and willing to see the organization disabled if their side loses.
They get characterized as "bossy", like it's an ugly personality trait, and not like they are the boss and it's their job to do something about a thing.
Well, you see the result. Apart from this drama, lots of debate and speculation whether we are already in AGI territory, because it is all just a black box and no one knows what exactly was in the training data to be able to judge the quality of the output.
That is way too much AI priesthood for me, just open it for real, if you don't want concentration of power.
My point is this coup against OpenAI (seemingly) wasn't started by people who want to make AI more open. They want to make it inaccessible unless you're in their club. To my reading, that is more starkly opposed to the charter than anything Sam Altman has done. Commercialization = giving wide access. Not total 100% open source access, but wide access nonetheless.
The velocity of release and entwining with MSFT (by the profit side) might then be reasonably seen as a great concern for the board.
The default, in the absence of an OpenAI, is that it gets developed secretly by these organizations, and they get to decide who can use it (hint: it's not you, typical HN reader).
OpenAI were the only ones at least trying to walk this tightrope, and I think they were doing a pretty good job. So what were the real motivations of these three wealthy/elite board members who are taking it upon themselves to decide this issue for the rest of humanity? It's looking less and less likely that safety was the reason.
In short, the plan was to reduce Condé Nast's ownership of Reddit by hiring a new CEO, and convincing that person to demand, as a condition for their hiring, that CN reduce their ownership share. Further VC funding and back-room machinations let them further reduce CN's share of the company, thus eventually wresting control over Reddit back to the original founders. Yishan was subsequently pushed out and Ellen Pao promoted to CEO, which didn't go so well either.
Both Altman and Pao are responding in that thread.
[1] https://www.reddit.com/r/AskReddit/comments/3cs78i/comment/c...
Wild guess: If the board stands its ground, appoints a reasonable new CEO, and employees understand that OpenAI will continue to be a hot startup, most of them will stay with the company due to their equity
"Sam was trying to censor legitimate concerns the board had with regards to the safety of the technology and actively tried to undermine the board and replace it with his own puppets."
If that is indeed true they did a mistake by saying something vague imo.
> Mr. Altman, the chief executive, recently made a move to push out one of the board’s members because he thought a research paper she had co-written was critical of the company.
It's like your job is to fire me if I fail to fulfill the company mission, but I will fire you if you don't get my approval before saying stuff in public.
> From the perspective of avoiding an AI race, conflict of interest could very well be a good thing.
On the American subcontinent: yes. But the world is larger than that.
I don't know why she chose to publicly execute Altman, there just isn't enough information to say for sure. It probably wasn't a specific, imminent safety concern like "Our new frontier model was way more capable than we were expecting and attempted a breakout that nearly succeeded during internal red teaming", according to the new CEO it wasn't anything like that. The new CEO has heard their reason, but is putting a lot of pressure on them to put that reason in writing for some reason. I don't know why, we just don't have enough information.
But basically, she is a very qualified person on the exact topic you are concerned about and has devoted her career to solving that problem. I wouldn't write off what she's doing or has done here as "She didn't consider that China exists".
So I'm not sure how this all integrates in her head but you break the glass in break-the-glass moments, not before, it's a one-shot thing.
I'm not sure what you're trying to get at.
It’s incredibly disingenuous to slap your name on an ethics paper claiming a company is doing malfeasance such as triggering a race-to-the-bottom for AI ethics when you have an active hand in steering the company.
It’s shameful. Either she should have resigned from the company, for ethical reasons, or recused herself from the paper, for conflict of interest.
I'm not sure if this is the true statement, because the not-for-profit/profit arms make it more complex.
In this case, the non-for-profit board seems to act as a kind of governance over the profit arm, in a way, it's there to be a roadblock to the profit arm.
Normally a board aligns with the incentives: to maximize profit for shareholders.
Here the board has the opposite incentive, to maximize AGI safety and achievement, even at the detriment of profit and investors.
To put these claims in a published paper in such a naive way with no disclosure is academically disingenuous.
While the system card itself has been well received among researchers interested in understanding GPT-4’s risk profile, it appears to have been less successful as a broader signal of OpenAI’s commitment to safety. The reason for this unintended outcome is that the company took other actions that overshadowed the import of the system card: most notably, the blockbuster release of ChatGPT four months earlier. Intended as a relatively inconspicuous “research preview,” the original ChatGPT was built using a less advanced LLM called GPT-3.5, which was already in widespread use by other OpenAI customers. GPT-3.5’s prior circulation is presumably why OpenAI did not feel the need to perform or publish such detailed safety testing in this instance. Nonetheless, one major effect of ChatGPT’s release was to spark a sense of urgency inside major tech companies.149 To avoid falling behind OpenAI amid the wave of customer enthusiasm about chatbots, competitors sought to accelerate or circumvent internal safety and ethics review processes, with Google creating a fast-track “green lane” to allow products to be released more quickly.150 This result seems strikingly similar to the race- to-the-bottom dynamics that OpenAI and others have stated that they wish to avoid. OpenAI has also drawn criticism for many other safety and ethics issues related to the launches of ChatGPT and GPT-4, including regarding copyright issues, labor conditions for data annotators, and the susceptibility of their products to “jailbreaks” that allow users to bypass safety controls.151 This muddled overall picture provides an example of how the messages sent by deliberate signals can be overshadowed by actions that were not designed to reveal intent.
https://news.ycombinator.com/item?id=38374972
It has an excess amount of weasel words, so you might need to employ ChatGPT to read between the lines.
>The organization was founded in December 2015 by Ilya Sutskever, Greg Brockman, Trevor Blackwell, Vicki Cheung, Andrej Karpathy, Durk Kingma, Jessica Livingston, John Schulman, Pamela Vagata, and Wojciech Zaremba, with Sam Altman and Elon Musk serving as the initial board members.
Altman didn't join as CEO until 2019.
Edit: All those names and we're going to say stuff like Altman was "the one" who made the company so important?
Thing is, I don't really care if I'm wrong or right about that title. We're bickering about semantics at this point, and whether or not he was technically a founder doesn't really change the crux of my first post in this chain.
Which I guess now's as good of a time as any to answer that point about Altman having founded it - no, I don't think that founding something automatically means that you have the capability to oversee it long-term.
In my opinion, it was not a very strong position because the allure of money and trying to be the biggest is too strong (as we're seeing now), but I think it was at least coherent.
They could easily lose any power they had to guide the industry, it was a huge gamble. I remember reading a Harvard business school study showing the first mover advantage repeatedly turned out to be ineffective in the tech industry as there is a looong series of early winners dying out to later market entrants like Friendster->FB, Google, a bunch of dotcom era Ecommerce companies predating Amazon, etc.
They need full industry/society buy in - at an ideological level - to win this battle, they won't win through backroom dealing in a boardroom while losing 90% of their own staff.
The intuition for GPT-4 being on the way towards superhuman intelligence or persuasion abilities is that the only significant difference between it and GPT-3 appears to be the amount of compute power applied to training.
The previous US President was much less than superhuman, and he still managed to seriously threaten democracy in the US, and by extension cause a minor existential risk to civilization. Imagine what a more intelligent agent with enough attention to be personally persuasive instead of generically persuasive could achieve for itself, given a goal. (If you think that LLMs can't express goals or preferences, read some of the Bing Sydney transcripts.)
The biggest risks I see from AI over the next 2 decades are:
1. A complete devolution in what is true (because creating convincing fakes will be trivial). From politics to the way that people interact with each other on a daily basis, this seems like it has the potential to sew division
2. the societal turmoil/upheaval arising from the mass unemployment (and subsequent unemployability) of millions of people without the social safety nets in place to handle that. When people are suddenly unable to work and support their families and those who have consolidated the power have no interest in sharing the wealth, there will inevitably be pushback.
We currently almost have scifi-level AI we can talk to on our smartphones, which would have been fantastical only a few years ago.
What exactly are you looking for here, proof of concept? This stuff is inherently speculative - because if it wasn't speculative, we'd probably be dead.
OpenAI is (hypothetically) progressing to AGI, therefore bad. Whether GPT specifically is a half a foot on a 1000km journey or potentially insignificant, it doesn't matter to them as they've pre-selected any potential for progress as all bad.
[1] if you search "Helen Toner" on YouTube it's just a collection of videos of her talking at EA conferences. So she's a believer
Safety isn’t binary, it’s degrees of risk assessment, mitigation and acceptance. Infinite safety is never progressing. But never progressing means failing the charity mission. And AI isn’t just being created at OpenAI, for them to succeed at the charity mission they need to stay ahead of risky competitors with a safer AI.
The charity can’t afford its GPU costs without the LLC, and the LLC can’t lead the industry if it’s behind all the competitors. To be relevant the safety side needs compute, and they need to be ahead of the curve, and they also need real world data from the LLC. If the charity nukes the LLC, it takes itself with it and fails its mission. They’re so intertwined they need compromise in the board. And they let that board dwindle to small entrenched factions, which sounded more like the Hatfield’s and the mcCoys. With such a fragile structure they needed mature adults with functional conflict resolution.
So -- if appropriate -- he could make changes.
Or he could not change, and she publishes anyway.
> No, Sam wanted to be the first recipient of Helen's expression.
If I get copy approval to what you write, that is by definition a restriction to your freedom of expression. How is this up for debate?
Where are you drawing this conclusion from? Nothing in the text suggests that was his intent - certainly his actions thereafter suggest that he was in fact more concerned with the potential harm to the for-profit company.
Otherwise I'm not sure what changes would be "appropriate" for him to direct.
A board member should care about their organization more than their interpretation of the charter. If a Mozilla board member publicly criticized Firefox for violating their charter of a free and open internet because they're receiving money from Google, while also praising Brave for not doing so, I guarantee that they'd also be ousted.
Isn't the whole reason we have non-profit boards and charters is so that they can make just this kind of call?
(I'm not saying that this is what is happening here -- just responding to your particular claim.)
If you are going to potentially ruin the lives of 770 people you should have a better reason than being afraid of a chatbot and laundry buddy.
Not surprised Hacker News identifies and sympathizes more with the software folks than with everyone else who has to live with the systems they build/disrupt, though.
Yeah, this is the common trend among these insane utilitarians like you and Helen Toner. It's all a game of "how can I justify doing heinous shit to people". IF a superintelligent AGI is created that is as disruptive as you imagine, it also means we find cures for diseases and solve many other potentially world ending problems (such as climate change).
So since you brought up the hypothetical disruption of a technology that doesn't exist, can I now lay at your feet the blame for the millions of lives that are lost due to disease and climate change since we didn't move fast enough on AGI and advancing technology?
This is why these absurd hysterical fear mongering arguments are worthless.
As a CEO I'd want your concerns brought to me so they could be addressed. But if they were addressed, that is one less paper that could be published by Ms. Toner. As a member of the openai board, seeing the problem solved is more important to openai than her publishing career.
"Second, because the board is still the board of a Nonprofit, each director must perform their fiduciary duties in furtherance of its mission—safe AGI that is broadly beneficial. While the for-profit subsidiary is permitted to make and distribute profit, it is subject to this mission. The Nonprofit’s pzrincipal beneficiary is humanity, not OpenAI investors."
I see. I don't know whether she did discuss any issues with Sam before hand, but it really does not sound like she had any obligation to do so (this isn't your typical for-profit board so her duty wasn't to OpenAI as a company but to what OpenAI is ultimately trying to do).
Fall 2021
Holden Karnofsky resigns from the Board, citing a potential conflict because his wife, Daniela Amodei, is helping start Anthropic, a major OpenAI competitor, with her brother Dario Amodei. (They all live(d) together.) The exact date of Holden’s resignation is unknown; there was no contemporaneous press release.
Between October and November 2021, Holden was quietly removed from the list of Board Directors on the OpenAI website, and Helen was added (Discussion Source [1]).
0. https://loeber.substack.com/p/a-timeline-of-the-openai-board
1. https://forum.effectivealtruism.org/posts/fmDFytmxwX9qBgcaX/...
Isn’t she an academic? Getting people to pay attention to her is at least half her job.
Her only personal fiduciary responsibility is to the mission defined in that collection of words, not to the employees of the for-profit.
That’s how this works whether you think it should or not.
OpenAI is a non-profit running this commercial enterprise but they are seemingly at odds with that enterprise. Investors should rightly be very concerned about the future of their investment.
> IMPORTANT
> *Investing in OpenAl Global, LLC is a high-risk investment*
> *Investors could lose their capital contribution and not see any return*
> *It would be wise to view any investment in OpenAI Global, LLC in the spirit of a donation, with the understanding that it may be difficult to know what role money will play in a post-AGI world*
> The Company exists to advance OpenAl, Inc.'s mission of ensuring that safe artificial general intelligence is developed and benefits all of humanity. The Company's duty to this mission and the principles advanced in the OpenAl, Inc. Charter take precedence over any obligation to generate a profit. The Company may never make a profit, and the Company is under no obligation to do so. The Company is free to re-invest any or all of the Company's cash flow into research and development activities and/or related expenses without any obligation to the Members.
I guess maybe they thought it was the same boilerplate every prospectus has?
Investors should rightly be concerned about things like gross negligence, active sabotage, or simple infighting. These are all current human concerns and not at all that AGI will destroy the concept of money rendering all investments pointless.
An investor might read that and think they don't actually mean it, but you can't claim they weren't clear about the nature of any required investment.
Similarly, Microsoft is coming across as blameless here, but there’s a reason Satya went on a full court press. They committed tremendous resources and money to an unpredictable business structure that is fundamentally not aligned with the profit motive - in its very charter.
The thing is doing exactly what the thing was supposed to do. The only reason it’s surprising to anyone is because people assume OpenAI is a for profit entity.
Altman appears to me to be reckless, ms. Toner appears to be naive. Both should not be near this kind of power.
Sorry for the confusion.
NORTH AMERICAN
noun
a timid or feeble person. "Jennings plays him as something of a milquetoast"
adjective
feeble, insipid, or bland. "a soppy, milquetoast composer"
I have a feeling that's exactly what he's doing at Microsoft; at some point their "AI lab" will be spun off into a new company.
At least assuming Microsoft hasn't run out of capacity in their data centers. I know Azure sometimes have capacity issues but those issues are intermittent and not say 5 year long (or whatever the windoe is when it comes to consuming these credit)
Yes it does, because we're taking about GPU availability that is absolutely highly constrained across the economy.
If we were talking about CPU credits instead, then you're right that Microsoft would be paying/losing merely cost.
What a lovely turn of phrase! I'm stealing it for use later, in place of "actions speak louder than words".
So OpenAI should just shut down because the charitable donations were not enough to keep the lights on.
Also running the for-profit arm the way Altman did isn’t the only way.
A more reasonable CEO would do what he/she can to make money without running afoul of the charter. Yes, it will be less profit - and sometimes no profit at all - but that’s the way it should be in a non-profit organization.
OpenAI used to be an organization that dabbled with various AI technologies. Anyone remember their DOTA2 bot? Before Altman turned it into all about commercializing their LLM - going so far as to try to lobby congress to create laws, in the name of safety of course, to hobble any upstart competition.
OpenAI is operated at a public charity and is required to meet a "public support test" of 33%, so Musk could not have given his $40 million without the $20 million from Open Philanthropy, an EA supported public charity. In fact, most of Open Philanthropy's money went to OpenAI.
Public charities are also required to have less than a majority of the board be employees or relatives of employees. For a while, after Elon Musk was removed, Holden Karnofsky of Open Philanthropy was the only non-employee on the board.
Other board members elect replacements iirc
>In the email, Mr. Altman said that he had reprimanded Ms. Toner for the paper and that it was dangerous to the company, particularly at a time, he added, when the Federal Trade Commission was investigating OpenAI over the data used to build its technology.
I'm sure that's how tobacco and oil executives justify their paycheck: "If I wasn't doing it, someone else would". Probably lots of criminals justify their crimes using similar reasoning too.
Ultimately it's not quite true. If you quit your job at that tobacco or oil company, it's going to take a while to find them a replacement for you. That replacement might not be quite as good at the job (why were you picked for the job in the first place? Probably because you were the strongest candidate.) You aren't going to totally halt EvilCorp by quitting, but you're going to give it a speed bump.
In the case of OpenAI, we have to consider what organization is likely to replace them if they throw in the towel. If it's a more cautious org like Anthropic, OpenAI throwing in the towel could actually be a good thing from the point of view of their charter: https://openai.com/charter
That would have been a lot safer than to give MS the keys to the kingdom.
But she was on the board when that happened and presumably agreed to the deal or she could have resigned.
And it's not quite the tobacco company we're talking about here. I'm more in line to compare it to the Manhattan project.
In this case, if your company fails to provide for greater good, then it is best to destroy it. And it also shows some example to others. Whatever others do, you have no power. I does not matter if you pretent or not.
Comparison works only if you could do something about it.
If she agreed with the Microsoft deal but not with what followed that too would be hopelessly naive. Microsoft + ethics?
This is an all but undeclared arms race.
I only see negatives. Other entities with less of a brake on their ethics are gaining ground. Microsoft of all parties is strengthening their position and has full access to the core tech.
Did OpenAI actually have a meaningful brake though? Like, if all the employees apparently think that the success of the company is more important than the charter, can we be sure that OpenAI actually had a meaningful brake?
edit: Or that the board can't actually make a difference because whatever OpenAI doesn't do someone else will. But if people actually thought that were true they wouldn't have set up the board and charter.
Would the destruction of OpenAI in 2023 be seen as bad or good with that hindsight?
It seems bad now but if you believe the board was operating with that future in mind (whether or not you agree with that future) it's completely reasonable imo.
>During the call, Jason Kwon, OpenAI’s chief strategy officer, said the board was endangering the future of the company by pushing out Mr. Altman. This, he said, violated the members’ responsibilities.
>Ms. Toner disagreed. The board’s mission is to ensure that the company creates artificial intelligence that “benefits all of humanity,” and if the company was destroyed, she said, that could be consistent with its mission.
This person is too far gone. Life isn't a movie.
This comment is a credibility killer. "This person", as a member of the board of a non-profit, has a duty to keep the non-profit on mission, and even to call for its dissolution if that isn't possible.
This is delusion, and she thinks she's some hero that's saving the world by keeping it a non-profit when in reality it's just creating needless chaos and even impacting innocent people's livelihoods.
In many cases they're industry experts / academics and in many cases those academics continue to publish papers that look objectively at the actions of all companies in their sphere of influence, including the one they're on the board of.
It's _expected_ for them to publish this type of material. It's literally their job. Cheerleading is for the C-suite.
Imo you have to commit to working within an organization if you're on its board, you can't burn it down from within. I think it's less of an issue to try to remove the CEO if you don't agree with the direction they're taking, but this was obviously done in a sloppy manner in this case, and their willingness to destroy it makes me distrust them (not that my opinion matters per se).
In EA, the ends always justify the means, so that includes doing any or all of the things you mention if it achieves some ideological goal.
I think OpenAI should have prioritized sanity more in its board member selection.
Just because you're unpopular doesn't mean you're insane.
Where did I say anything about a "superintelligence"?
The extant statistical inference engines aren't going to build killer androids. No, I'm terrified of what profit-motivated humans are going to do to each other using machines designed to match patterns and generate convincing bullshit.
Think less "Terminator", more like Cambridge Analytica, Russian-style firehose-of-falsehood, 2018 YouTube algorithmic radicalization, 2014 Facebook "emotional contagion" experiment, 2023 TikTok censorship, doctored videos/evidence, crossing an inflection point in declining trust in democratic institutions, fully automated fraud and identity theft in an arms race with increasingly invasive countermeasures, scientific atrophy, artistic and cultural homogenization, etc.
FFS, the SGA and WGA just wrapped up an unprecedented strike motivated in large part by the broadly anticipated destructive impact of ML on their industry. You wanna talk about jobs? Wikipedia says that cost 45,000, many of whom were barely getting by, while you whinge about 770 in-demand professionals who already have highly paying offers from at least two Fortune 500 companies. From what I've seen, the voice acting industry, although less able to advocate for itself, is in disarray too and already directly impacted by the spread of models trained unconsensually on actors' past work. Plus major publications like CNET have already been caught canning their staff and dishonestly publishing error-ridden drivel with a misguided faith in LLMs, Google and thus access to most textual information online is basically useless because of SEO blogspam already, etc.
Framing climate change as "we didn't move fast enough on AGI" is… Bizarre. Windmills ain't new tech, and neither are hydro dams, nuclear reactors, nor LRT. Framing disease as an "AGI" problem is… Delusional, as well. Some academics may certainly find domain-specific ML models useful, but were I a bit more vindictive, I'd pay good money to see you try to get ChatGPT to fold a novel protein correctly.
> It's all a game of "how can I justify doing heinous shit to people".
Someone is meant to use context and understand that when the context clearly refers to someone who is well known to be a member of the board of directors, 'advisory board member' means that they are a board member in an advisory role, aka a nonexec.
That assumes that someone is familiar with the structure and makeup of boards of directors. It's a bit rude to participate in a conversation if you're not familiar with the subject matter and the meaning of words-in-context as you just waste peoples time making them elaborate.
No, those are not synonyms. Advisory boards are distinct from the board of directors. Advisory board members have roles similar to what you alluded to in your original comment and do not have fiduciary duties, hence my confusion.
Non-exec board members are not involved with day-to-day business operations but their fiduciary duties are no different than exec members.
Language has meaning. Don't insult others for your own clumsy usage.
Only human!
> Don't insult others for your own clumsy usage.
We're on the internet. Something like 25% of internet discourse is nitpicking vocabulary. Let's keep traditions alive this holiday season!
Who made the decision to fire Altman? The board.
Who is the board answerable to? The law.
Does she? In that case, perhaps she should tell someone what it is.
See also: “OpenAI LP’s primary fiduciary obligation is to advance the aims of the OpenAI Charter, and the company is controlled by OpenAI Nonprofit’s board. All investors and employees sign agreements that OpenAI LP’s obligation to the Charter always comes first, even at the expense of some or all of their financial stake.” --https://openai.com/blog/openai-lp
To emphasize, they will not take your investment money until you've signed an agreement saying that you're aware of all of this.
I think any action to immediately destroy OpenAI should have been preceded by being on-track to create AGI and strong indications that it was not going to benefit humanity, as the charter implies. Anything less and it's just a powerplay. But what is the point of having a nice fat red button if you never get to push it?
The only person I've seen advocate "ends justify the means" in this discussion is someone criticizing Toner: https://news.ycombinator.com/item?id=38373319
>The group’s other founding members are world-class research engineers and scientists: Trevor Blackwell, Vicki Cheung, Andrej Karpathy, Durk Kingma, John Schulman, Pamela Vagata, and Wojciech Zaremba.
That sentence explicitly lists who the founders are. Note that neither Musk nor Altman are listed before the period closes it out. The next sentence goes on to say...
>Pieter Abbeel, Yoshua Bengio, Alan Kay, Sergey Levine, and Vishal Sikka are advisors to the group.
Alright, cool - we've moved on from the list of founders, I see. It's not until the next sentence that we find out that..
>OpenAI’s co-chairs are Sam Altman and Elon Musk.
So, OpenAI themselves made it a point to separate "founders" from "co-chairs" in their own announcement. I don't really know how it could be any clearer than that.
This one is very much an explicit, "You're giving us a donation. Don't expect a return." It's in an impossible-to-miss pink box. They can't be more clear about this.
If investors are concerned about those things, then they should not give money to people who pointedly leave those things on the table. Like this one:
> The Nonprofit’s principal beneficiary is humanity, not OpenAI investors.
Also, Microsoft got something for their money already. They have an independent license to the tech they can run themselves and iterate on.
The OpenAI board has no duty to Microsoft to get their input. MS might have believed their investment amounted to de facto control, but they're wrong.
Additionally, the role of the board is to supervise the CEO, not make strategic decisions, no? If the MS partnership was a mistake, Sam is the primary person to blame here.
I'm seeing a lot of "Monday morning quarterback" in this thread
That's fine but you are not forced to participate.
> That's not obvious to me. Suppose OpenAI deliberately partners with a number of different infra providers. That creates a bunch of drag because they have to create a compatibility layer for staying agnostic to the underlying infra. And the outcome in a situation like this would probably be fairly similar: Since big tech companies are focused on making a buck, they would probably pressure the board to do whatever they think will be most profitable, just like Microsoft.
So, either you are 'OpenAI' and that means open to all or you are Microsoft AI, licensing your tech to one of the least ethical and most wealthy companies in the world. If the board didn't have anything to say about that at the time and if they didn't resign en masse you have to assume that they are ok with it.
> Additionally, the role of the board is to supervise the CEO, not make strategic decisions, no? If the MS partnership was a mistake, Sam is the primary person to blame here.
So are they or are they not there to supervise the CEO?
In my view a board exists to protect the interests of the stakeholders guided by a charter. OpenAI sees 'humanity' (whatever that means to them) as their stakeholders and that trumps each and every other party involved. But it is unclear whether their actions really benefit those stakeholders or if it is just people being people.
So far I'm seeing a lot of the latter and only little bits of the former.
Yes. From the board. That's why we typically don't see boards air their dirty laundry out in the open even when in retrospect, they obviously had a problem with leadership.
It's not a contradiction to criticize someone for saying too much sometimes and saying too little another time. The board should be open to insiders and cautious with external communications.
Which is why she needs a legitimate reason.
Nevertheless, when a board decision explodes into international headline news as this one did, my gut feeling is it might possibly be a good idea to make a press release with more details.
But that's easy for me to say — I don't have the real insider information, and I wasn't in their shoes.
Then it was 1/6 about the voting.
But voting is totally different thing than speaking about concerns and then getting actually them into the list, which will we voted further if they decide to do something about it.
In theory that is 1/6 * 1/6 power if you are alone with it for the decision to happen.
What I believe is that the first party to create an AGI will likely end up being nationalized and that that AI will be weaponized the next day to the exclusion of the rest of the world.
Hence my position that the charter was a fig leaf designed to throw regulators off.
Academically, did she not disclose being on the board on her paper?
Being at the top of the org and being present during the specific incidents that gives one qualms burdens one with moral responsibility, even if they were the one who voted against.
You shouldn’t say “they did [x]” instead of “we did [x]” when x is bad and you were part of the team.
Or, that she should write her paper in the first person: "We, OpenAI, are doing bad things." That would probably be seen as vastly more damaging to OpenAI, and also ridiculous since she doesn't have the right to represent OpenAI as "we".
I have no idea why you think that should be a rule, aside from wanting Helen to never be able to criticize OpenAI publicly. I think it's good for the public if a board member will report what they see as potentially harmful internal problems.
You implied that she couldn't publicly disparage the for-profit because the board has a fiduciary responsibility to the minority investors in the the for-profit. I only piped up to correct that single point because it's wrong.
I, personally, have a hard time picking sides here. On its face it seems that Ms Toner is more aligned with OpenAI’s stated mission.
However Sam seems much better suited to operating the company and being the “public face” of AI. One thing I’m pretty confident in is that the board isn’t capable of navigating OpenAI through the rough seas it’s currently in.
I think an argument could realistically be made to blow up the whole governance structure and reset with new principals across the board, that being said I don’t know who’d be a natural arbiter here.
At the end of the day the untenable spot Ms Toner is in is that the genie is out of the bottle which makes her position of allowing the company to self-destruct a bit tone deaf.
AGI is either just around the corner or it will be 50 years or more and if it is just around the corner you'd hope that parties that have at least some semblance of balance would end up in charge of the thing. Because if it is possible I expect it to be done given the amount of resources that are being thrown at this. Assuming it can be done weaponization of this tech would change the balance of power in the world dramatically. Everybody seems to be worried about the wrong thing: whether or not the AGI will be friendly to us. That doesn't really matter, what matters is who controls it.
No single individual (Altman, Toner, Nadella, or anybody else) should be taking the responsibility about what happens onto themselves, if anything the board of OpenAI has shown that this isn't a matter for junior board members because the effects range far further than just OpenAI.
Tone-deaf basically means "unpopular", doesn't it?
I'm old enough to remember when doing the right thing, even when it's unpopular, was considered a virtue.
probably not, but continuing to publish research papers of the kind that got her the job in the first place seems reasonable maybe
but that's what altman was criticizing her for, not for committing crimes, which as far as anyone knows she hasn't done
> There's 700+ employees that disagree with her interpretation
maybe, maybe not, but the standard way that nonprofits and other companies work is that the employees do what the management tells them, the management does what the board tells them, and the board does what the shareholders or the charter tells them, and in all cases, if they refuse, they get fired
if you're an employee at the american cancer society and you decide that malaria in africa is a more important cause, you probably shouldn't expect to be able to continue using the acs's assets to fight malaria in africa. it might happen, but you shouldn't be at all surprised if you get fired
They likely violated RICO by essentially nuking the company's valuation and trying to sell to Anthropic. That is a crime, by the way, so they don't seem above that.
They haven't given even a clear reason for firing Sam. "Not being candid" with no evidence doesn't sound like it directly violates "the charter", but you can really argue anything when you're just "interpreting" the charter.
Anyway, you can make infinite excuses for their actions and justify them infinite ways. Hopefully you agree that the current situation is a giant mess, and it's caused by their narrow misguided interpretation of their job, and it's at the expense of basically everybody else.
Really. What is/are the predicate offense(s) for this alleged RICO violation?
LOL
Nothing you have said is credible or honest.
Should you argue in bad faith?
The optics dont look good though if a board member is complaining publicly.
If Sam would let it go what would happen? Nothing. Criticism and comparisions already exist and will exist. Having it coming from board member at least gives counter argument that they're well aware of potential problems and there is opportunity to address gaps if they are confirmed.
If regulators find argument in the paper reasonable and that's going to have impact - what's wrong with that? It just means argument was true and should be addressed.
They don't need to worry about commercial side because money is being pured more than enough.
The nature of safety research is critical by definition. You can't expect to have research constrained to talk only in positive terms.
Both sides should have worried less and carry on.
That's the pro safety solution.
If she believes that the most advanced current effort is heading in the wrong direction, then slowing it down is helpful. "Most advanced" isn't the same as "safest".
> and potentially let a different effort take the lead
Sure but her job isn't to worry about other efforts, it's to worry about whether OpenAI (the non-profit) is developing AGI that is safe (and not whether OpenAI LLC, the for-profit company, makes any money).
It's amateur behavior. I'm sympathetic to her goals, less impressed by the execution.
Again, her mandate is not to help OpenAI deal with the FCC it's to prevent the company from building unsafe AI, one reasonable aspect of which might be to compare the methodologies of different companies.
You can justify pretty much anything with ends-justify-the-means logic and I have a hard time believing that the people who set up the charter would, a priori, have said that suppressing research comparing the safety approach of the company to a competitor in order to not make the company look bad so that the competitor wins because the company insists, without any basis, that they would be better for safety is in line with the principles of the charter. That is just trying to game the charter in order to circumvent it and is a textbook case of what the board was appointed to prevent.
This isn't a theoretical exercise where we get to do it all over again next week to see if we can do better, this is for keeps.
The point could have been made much more forceful by not releasing the report but holding it over Altman's head to get him to play ball.
> You can justify pretty much anything with ends-justify-the-means logic
Indeed. That's my point: the ends justify the means. This isn't some kind of silly game this is to all intents and purposes and arms race and those that don't understand that should really wake up: whoever gets this thing first is going to change the face of the world, it won't be the AGI that is your enemy it is whoever controls the AGI that could well be your enemy. Think Manhattan project, not penicillin.
> I have a hard time believing that the people who set up the charter would, a priori, have said that suppressing research comparing the safety approach of the company to a competitor in order to not make the company look bad so that the competitor wins because the company insists, without any basis, that they would be better for safety is in line with the principles of the charter. That is just trying to game the charter in order to circumvent it and is a textbook case of what the board was appointed to prevent.
That charter is and always was a fig leaf, I am probably too old and cynical to believe that it was sincere, it was in my opinion nothing but a way to keep regulators at bay. Just like I never bought SBF's 'Altruism' nonsense.
The road to hell is paved with the best of intentions comes to mind.
That effort isn't critical to OpenAI other than to try to create a monopoly.
I guess the main problem here is that nobody knows properly what this deal contains and how it was made. I believe CEO has power to do such deal alone?
But what is interesting is that until MS got involved OpenAI was struggling for funding and talent, and afterwards it took off like a rocket.
I don't think it was done in a sloppy manner, I think there's a huge amount of spin and froth being generated to make it _look_ like it was done in a sloppy manner, but the reality is that the board took an executive action, released a discreet and concisely worded press release and then fell silent, which is 100% professional behaviour.
To me it's the other side that are acting sloppily, they're thrashing around like maniacs, they're obviously calling in favours behind the scenes as supportive articles are popping up all over the place, apparently employees were calling other employees in the middle of the night urging them to sign the 'I'll quit unless' letter, employees have been fed a misleading narrative (two of them, actually) about why the action was taken in the first place which riled them up...
I have to assume here that the board - as they're the ones acting professionally - similarly acted professionally before the firing and discussed the matter at great length and only resorted to this last minute firing because of exigent circumstances. The fact that a fired employee was able to cajole/bluff/whatever his way back into the office to prove some sort of point suggests that the board's action may have been necessary. What exactly was done during that time in the office, one might ask? Were systems accessed? Surely it was an unauthorised entry and a major security violation.
You do see that that action (re-entering the office after being fired) gives off lunatic vibes, don't you? Maybe it really did come down to 'yeah you're technically my boss but I'm gonna do whatever the hell I want and there's nothing you can do about it because everyone loves me' to which the board's only possible course of action is to fire immediately.
Either by himself or by the person that permitted him access knowing beyond any reasonable doubt that they were not authorized to be there?
If systems were accessed, was another crime committed?
If a person facilitated that access, has the boundary for 'conspiracy to commit...' been reached?
How? Being a board member is not enough. There were aready likely two against her in this case, while the rest is unknown.
Regarding specific verbiage and grammar, I’m sure an academic could give clearer guidance on what is better form in professional writing. What was presented was clearly lacking.
And that responsibility is defined as a fiduciary duty to the mission of the organization as defined in the charter.
As a fellow board member, if Sam thinks Helen's behavior is in conflict with the mission of OpenAI as defined in the charter, he is free to push for her removal.
Funder, not founder, yes.
If OpenAI themselves chose to distinguish between founder and co-chair, then I really dunno what else to tell ya.
There are no shareholders of a charity, instead the board are the notional owners and take decisions that shareholders would.
So in this context Musk is founding board member and thus a founder as it would apply to a for profit entity.
Given how Altman has responded to things throughout his career and in this, I fail to see how doing this would result in anything other than the same outcome: Altman won't be moved by that, but consider it extortion and move for her removal from the board, regardless. In the end, he wants criticism or calls for caution stifled.
So basically we have two cases here:
1. The charter is a deceptive fig leaf, OpenAI aren't the good guys. In which case blowing up OpenAI looks reasonable
2. The charter is not a fig leaf, Helen is doing the right thing by upholding it
I'd say upholding the charter is the right thing in intermediate situations as well. Basically, OpenAI are the good guys, in part, to the degree to which the board (and other stakeholders) actually uphold the charter instead of being purely profit-driven.
It's #1 to some, and #2 to others.
Agreed. It seems remarkable that she dealt with her conflict of interest so thoroughly that she published an article that was mildly critical of the organization. I do not think her duty was to let Sam Altman seize power by firing her from the board on a flimsy pretext, though.
I'd say the OpenAI employees are the ones with the conflict of interest, since they stand to get rich if the company does well, independent of whether the charter is being upheld.
> I'd say the OpenAI employees are the ones with the conflict of interest, since they stand to get rich if the company does well, independent of whether the charter is being upheld.
Those pesky employees again. Too bad we still need them. But after AGI is a fact we don't and then it'll be a brave new world.
I just find the whole thing weird. I just think it seems unlikely that a board would decide something then about-face so rapidly as to have a meeting at the office the next day. It would be conventional for further meetings to take place at the offices of lawyers, for example.
It would also be normal for the non-fired members of staff to be instructed to have no further contact with the fired person without legal counsel present.
Sometimes it seems very much like everyone here is really playing to the cameras in a big way.
> Assuming it can be done weaponization of this tech would change the balance of power in the world dramatically.
Yes it would, but it wouldn't be as bad as everyone dying.
> Everybody seems to be worried about the wrong thing: whether or not the AGI will be friendly to us. That doesn't really matter, what matters is who controls it.
No, "who controls it" is a problem best tackled after "will it kill everyone." You say "That doesn't really matter," but again, Sam Altman himself thinks "Development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity."
Oh, ok. That makes it alright then. So, let's see those minutes of that meeting where this was decided with all of the pomp and gravitas required rather than that it was a petty act of revenge or to see who could oust who first from OpenAI. Because that's what it looks like to me based on what is now visible.
> Yes it would, but it wouldn't be as bad as everyone dying.
Not much is as bad as everyone dying. But for now that hasn't happened. It also seems a bit like a larger version of the 'think of the children' argument, you can justify any action with that reason.
> No, "who controls it" is a problem best tackled after "will it kill everyone." You say "That doesn't really matter," but again, Sam Altman himself thinks "Development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity."
Then why the fuck is he trying to bring it into the world? Seriously, they should spend all of their talent on trying to sabotage those efforts then, infiltrate research groups and set up a massive campaign to divert resources away from the development of AGI. Instead they are trying as hard as they can to create the genie in the sure conviction that they can keep it bottled up.
It's delusional on so many fronts at once that it is isn't even funny, their position is so horribly inconsistent that you have to wonder if they're all right in the head.
EA is becoming a massive red flag in my book.
They seem to miss the fact that every weapon that humanity has so far produced has been used, and that those that haven't been used hang over us like shadows, and have been doing that for the last 70 some years. Those are weapons whose use you will notice. AGI is a weapon whose use you will not notice until you realize you are living in a stable dictatorship. The chances of that happening are far larger than the chances of everybody dying.
Sam Altman, Elon Musk, and the original founders of OpenAI believe AGI is an existential threat and are building it with the belief that it’s best they’re the ones who do it, rather than worse people, in an arms race. Eliezer is the one saying “you fucking fools, you’re speeding up the arms race and have now injected a shit ton of $ from all VC into accomplishing the naive capabilities with no alignment.”
People don’t even realize that Elon Musk founded Neuralink with the belief that AGI is such an existential threat, we’re better off becoming the AGI cyborgs than a separate inferior intelligence. But most of the people thinking theyre so smart and understand the AGI X risk landscape here, even Elon fans, don’t know that.
People really watch too many movies. The real risk isn't AGI killing us all, the real risk is that plenty of people will think they are smart enough to create it and then to contain it whilst the ruthless faction of humanity uses it to set up shop in a way that they can never ever be dislodged again. Think global Mafia or something to that effect, or a world divided into three chunks by three major power blocs each with their own AI core. That's a much more likely outcome and one that on account of an incompetent board has just become a little bit more likely.