Elon Musk AMA(reddit.com) |
Elon Musk AMA(reddit.com) |
> One bit of advice: it is important to view knowledge as sort of a semantic tree -- make sure you understand the fundamental principles, ie the trunk and big branches, before you get into the leaves/details or there is nothing for them to hang on to.
Without the structure of prior knowledge, I never understand or remember facts; however, when I've had the time to develop that "first principles" knowledge, I can usually grasp and understand the significance of minutiae.
How do you of HN learn? Is it similar?
When I first started learning Linux for example, I didn't just learn the commands I needed to do certain things, I tackled everything. I spent months and months learning everything I could about it. I bought a giant Linux book and went from cover to cover. I learned about things I would never use (and probably still haven't).
I pushed myself to recompile the kernel even though I didn't need to. Then I did it probably 50 more times that month. No joke. Crashed my system. Rebuilt it. Rinse, repeat.
After laying down that foundation in the 90s, I've kept up on it but Linux is so very "easy" for me. Setting things up and getting work done is extremely intuitive, far more so than it is in Windows or OSX. So when people ask me why I prefer it I tell them it's a personal preference because it's so easy for me, and I even I forget that foundation I laid.
I have taken on other pursuits the same way, such as development but I notice any technology that I half ass learn just to get stuff done.. is hard. Sometimes I wish I had enough time in my adult life to build such a strong foundation in something like.. JavaScript for example. And I bet if I added up the time I spent struggling in the beginning I would have been able to do just that.
But yeah, long story short this is absolutely the best way to learn something. Build that trunk.
I just believe that almost everything in life is a skill and skill requires practice.
I've came across a research paper about deliberate practice and Peter Novig article about Teach yourself how to program in ten years.
This resonate and reinforce my belief of practices is everything.
As for the intricate of how, I'm more of a visual and kinestic kind of learner. Auditory suck. Also I need a book, I sit down and write notes first and then do problems. From there I usually look for a video on that subject for a secondary source. Most of the time a secondary source will give a different view on the subject matter and I get insight at a different point of view. Or that the second source explain it more better or fill in stuff that I didn't realize I gloss over or missed.
Well, the fact that you've found what works for you is a good thing. However, 'first principles' is subject to subjective interpretation. You can 'go down the rabbit hole' as it were, to any level. Should you have deep knowledge of electronics before learning computer science? Should you have deep knowledge of physics and chemistry before learning electronics?
In my opinion after a certain level, all knowledge is multi-disciplinary, and the boundaries of what constitutes roots, branches, leaves is extremely fuzzy. Also the distinction between theory and practice makes the boundaries even fuzzier.
Of course. I think this is the point of this learning style. After learning the first principles of various topics, the broad web that is higher knowledge is available to you.
Suppose, for instance, I wanted to learn how computer science worked from first principles. This study involves math, electronics, physics, and many, many more subjects. To accomplish this, I would pick one of the key, pure tenets and learn it. Let's say I choose math. I would then learn the key things I need to know about math and then move to electronics and physics, and etc. After knowing these, I could confidently approach the "web" of computer science because I have anchorpoints.
I think it's safe to view higher knowledge as a web supported by the anchors of "pure" subjects. After a while, these higher subjects are built upon and become pure topics themselves. Epistemology and the classification of knowledge is really a fascinating topic.
- (if you went to college) Did you have moments when the different courses connected? I think when people are poorly educated in college, it's because of this unfortunately common experience: they learn a bunch of specialized and disconnected subjects, never relate them to anything in their lives, and then forget them all.
I remember the subjects in CS/Math/EE starting to connect more and more around junior year, and I liked that feeling of a light bulb going on. You have to make a bit of extra effort. I did some little experiments outside class. I remember writing Matlab program (an "engineering" tool) to do some experiments in non-Euclidean geometry (pure math).
Of course there are some subjects that never connected, and I forgot those things.
When you have that semantic network, it lets you evaluate new ideas and designs more quickly. You see which low level principles come into play from the high level variables.
- (if you are a programmer) I think there's a pretty clear "semantic tree" in computing: from computer architecture, to OS, to programming language, etc.
So the test is: If you are generally satisfied with how computers/phones/etc. work, then I would humbly suggest that your semantic tree of computers isn't very well fleshed out :) I think any good programmer should see lots of areas where the status quo is just a result of path dependence and not actual any design principle.
When you have a good knowledge of all levels of the stack, then you can be creative. For example, I'm looking at Xen right now, and it has dawned on me that paravirtualization is a great idea (or perhaps great hack).
The related Mirage OS / unikernel line of research is another great example of connecting all the dots, and coloring outside the normal lines. 99% of programming jobs are basically coloring within the lines, where it doesn't matter if you have developed this semantic tree or not.
Somewhat related: there were some recent threads about organizing personal information, and I wrote about using a Wiki: https://news.ycombinator.com/item?id=8753599
Some people talked about using a journal to record thoughts or knowledge, but my point was that hyperlinks literally model the relationships in your head, and thus are superior for information organization / recall.
If you are one of the outliers (as you have said you are) you would have figured this all out a long time ago, even if you cannot articulate it.
http://www.reddit.com/r/teslamotors/comments/2rgzgo/official...
When they started trying to work out some good questions a day ago someone suggested that everyone upvote the resulting questions but the mods there quickly shut that down. [1]
The reasons given for deleting the comments was specifically what everyone was trying to avoid, that is voting brigading. As far as I can tell no one was asking for votes, simply working together to produce some high quality questions.
From a first reading that screenshot sounds like vote-brigading, not like using a single thread of questions within a sub to organise a list of great questions.
Moderators of /r/AMA see that as "vote brigading" and hid or deleted the questions so that Musk could not answer them.
Did I get that right in terms of what happened?
Amazing.
Is there any logic as to what order the questions and replies are displayed on the page? It doesn't seem to be either of reddit's 'top' or 'best' sorting. Perhaps whatever order they landed in within the JSON?
On the Musk transcript I found this formatting confusing: "Have you played Kerbal Space Program?
What do you think SpaceX uses for testing software?"
I can't access Reddit to see what the original comment was.
You can use it on any post (not just AMAs) by adding their URL in front or clicking their bookmarklet.
https://www.reddit.com/r/IAmA/comments/2rgsan/i_am_elon_musk...
Edit: There appears is a posthumously published book named Project Mars that says that. Not sure if I trust it.
Here's a pdf of Project Mars - http://www.wlym.com/archive/oakland/docs/MarsProject.pdf
The reference to the Elon is on page 177.
I hadn't initially noticed the fact it was posthumously published in 2006, however it would seem like an odd kind of forgery, if it is one.
Equally it does seem odd that Braun would choose Elon as the name of the Mars leader, so perhaps it might be a real work but with Elon added as a joke by the translator.
Or perhaps Braun chose the word Elon because he sometimes thought of leaders as trees, or something, and it is all just a massive bit of luck.
Personally I'm starting to suspect another explanation however. And if I'm right, there is an entire warehouse full of empty Elon Musk clones on ice, waiting for the spirit of Wernher Von Braun to animate each one in turn, in the event of damage occurring to the current corporeal vessel.
https://www.reddit.com/r/IAmA/comments/2rgsan/i_am_elon_musk...
Saying that he has no idea what is going to happen with the launch tomorrow, its a refreshing honesty.
I was also wondering on an semi unrelated note if HN had ever had AMA's from interesting people? I am not preposing that they should start happening though.
Edit: If anyone knows how to get a double asterisk in an HN comment would be grateful for the knowledge, was forced to add the unnecessary space. So far tried the HTML number code, which didn't work, and the help has no guidance.
You could try this:
test ** test
But you'd be stuck with the fixed width font.
The formatting page is not much help either:
https://news.ycombinator.com/formatdoc
Maybe something like
\*\*
could be done.He understands the data and the computation results, but those have yet to be correlated to the authority, nature.
The way to cope with such harsh conditions is, I suppose, always the same : hope of escaping and returning home.
I'm wondering whether he was tricked by someone at DeepMind, perhaps the same way people were tricked hundreds of years ago into thinking a chess-playing robot was possible.
I don't understand how looks are a legitimate criterion.
People are praising him because he's actually worthy of that, compared to say Justin Bieber.
A: Showering
Q: Would you ever consider becoming a politician?
A: Unlikely
Those were two actual questions asked to Elon along with his responses, and the two that stood out for me the most. Did he mention showering because that's the time he gets most of his ideas[1]? Did he say no to politics because it's more likely to change the world through innovation[2]?
Result: 8 downvotes. It'd be enough if the comment was downvoted just once, to sink in the page. That happens to everyone. But seven other people found it imperative to make an authoritative statement on the matter. Impressive. Did that keep their identity safe? Pushing threatening ideas away isn't the best way to help rearrange the semantic tree in your mind.
Could there be an inverse correlation between being downvoted and having good ideas? It shouldn't be a surprising discovery on valuable ideas if you consider the nature of the most valuable startup ideas: look like bad ideas but are good ideas.
So if you want to know if your ideas are good, it's not enough to see them gain support. It's also important to see people turn against them.
I know HN guidelines discourage commenting on downvotes, because they make for boring reading, but I'm starting to think being downvoted is a positive sign of how dangerous your ideas are.
Are you being downvoted enough?
[1] http://paulgraham.com/top.html
[2] https://news.ycombinator.com/item?id=8801803
edit: revised 80% of this after having a shower
I'm not sure how you get the idea that you're somehow provoking people with dangerous ideas.
http://www.amazon.com/Superintelligence-Dangers-Strategies-N...
It's the closest approximation to a consensus statement / catalog of arguments by folks who take this position (although of course there is a whole spectrum of opinions). It also appears to be the book that convinced Musk that this is worth worrying about.
http://www.econtalk.org/archives/2014/12/nick_bostrom_on.htm...
1) Strong AI is very far away, so no use worrying about it yet.
2) Strong AI if developed will not be likely to take over.
to which I would counterpoint with:
1) Sure, but when it happens, it will only happen once and thereafter will likely be out of our hands and control. Thinking about the groundwork that needs to go into safely developing an AI is cheap relative to the opportunity cost of getting it wrong. Prevention, cure, etc.
2) If developed, Strong AI will likely have SOME goal. It's not that a Strong AI will actively seek to rule humans, it will just have aims that will likely consider us as disposable as ants. To quote Yudkowsky:
"The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else."
- https://intelligence.org/files/AIPosNegFactor.pdf (Artificial Intelligence as a Positive and Negative Factor in Global Risk)
Pigs are also intelligent, but they never dominated humans because they don't have / cannot use guns.
There are reason to be worried that prescind from that.
Consider the Paperclip Maximizer[0] example: we build an AI with the sole task of producing paperclips, and it ends up destroying the human race.
This is why it first built Elon Musk.
> Nick Bostrom makes a persuasive case that the future impact of AI is perhaps the most important issue the human race has ever faced. Instead of passively drifting, we need to steer a course. Superintelligence charts the submerged rocks of the future with unprecedented detail. It marks the beginning of a new era.
Strong AI will be created by humans who will be able to set it's goals. Someone's probably going to make some that wants to take over. Others will make AI that doesn't.
The real issue I think is whether the creator will actually be able to "set goals" in a meaningful manner. How will you prevent this self-modifying super-intelligence from modifying itself?
Just like leaders used to issue orders to armies to kill others evolving into planes delivering payloads to people the pilots won't even see and now to drones, we will truly be entering an age of fire and forget.
"How do we prevent AI destroying us?" is not as useful a question as "how do we prevent us destroying us?"
That comment was posted to the single thread of questions in the sub, and was quickly shot down by the subs mods (u/EchoLogic). Here is a link to the actual comment if anyone is interested in having a look:
https://www.reddit.com/r/spacex/comments/2rb303/elon_musk_is...
Q: In order to use the full MCT design (100 passengers), will BFR be one core or 3 cores?
EM: At first, I was thinking we would just scale up Falcon Heavy, but it looks like it probably makes more sense just to have a single monster boost stage.
Q: Nice to see you are doing things the Kerbal way.
EM: Kerbal is awesome!
The second one:
Q: "Hi Elon! Huge fan of yours. Have you heard of/played Kerbal Space Program? Also do you see SpaceX working with Squad (the people behind KSP) to integrate SpaceX parts into KSP?"
Reply (not from EM): What do you think SpaceX uses for testing software?
EM to Reply: Kerbal Space Program!
Short version - Elon Musk likes and plays Kerbal Space Program.
What I found as a dangerous idea was pointing out things you notice when you are not sure why you notice them. Which is how the subconscious operates. Not everything that makes you pause should initially have an explanation. The majority of people's decisions occur without their awareness.
One thing I learned from this exercise was something I hadn't consciously noticed before. That I feel pressured on HN to comment. I don't like that. I want to do something about that.
Its a "AMA" or "Ask me Anything."
Original Post: >Zip2, PayPal, SpaceX, Tesla and SolarCity. Started off doing software engineering and now do aerospace & automotive.
>Falcon 9 launch webcast live at 6am EST tomorrow at SpaceX.com
>Looking forward to your questions.
>https://twitter.com/elonmusk/status/552279321491275776
>It is 10:17pm at Cape Canaveral. Have to go prep for launch! >Thanks for your questions.
[0] http://www.forbes.com/sites/geoffloftus/2012/05/09/if-youre-...
But it really is a country song: http://www.youtube.com/watch?v=l50L4GYhpLc
Likewise, stick to schools initially until you are sure you've got a solid grasp of core concepts, as taught & validated by people who know what you don't but should, then start transitioning away as your education can stand on its own. I've known too many "self taught" people who, while yes they can function in industry, suffer gaping glaring holes where early formal thoroughness would have closed them.
What you say does make sense.
As far as I can tell, this model was initially developed for training the middle ranks of the aristocracy in how to be officers in the army, and it only ever really works if you are allowed to beat or drug the children as otherwise it is almost impossible to get them to pay attention while sitting still in rows for an entire day. Which is probably why our classrooms got smashed up by bored pupils fairly regularly.
On the other hand, I have a mate who went to a Steiner school, which he describes as 'the first school he didn't burn down' and there it is an entirely different model that is centered around development rather than training. If I had any kids I would be looking for a school for them that was more in line with that kind of environment. http://en.wikipedia.org/wiki/Waldorf_education
The answer of course would be some sort of emergent system, but there are lots of intelligent seeming emergent systems (ie ant colonies, bee hives, ...)
http://www.gwern.net/In%20Defense%20Of%20Inclusionism
And many others like it.
These arguments (different arguments among different people) spread over diffferent pages and different spaces. They happened on article talk pages; in meta space (village pump, the WP manual of style); in admin spaces (ANI); even with some ARBCOM case.
There's easily 500,000 words about hyphen, minus, en-dash and em-dash on wikipedia.
Go and make some useful edits and see how that works out for you. Rules are strictly enforced above all else.
Everything I try to do is immediately reverted and sixteen rules are cited. Extremely frustrating, and it happens to many others in the community I'm in (Driving around the world) to the point I setup our own wiki so we don't have to deal with the BS bureaucracy of Wikipedia.
If you have a super-genius AI, massively more intelligent than any human, how do you know you are not being manipulated by it? Tricking us into disabling it's safety protocols, or gaining multiply indirect controll over capabilities dangerous to us, might be as easy for it as an adult tricking a 3 year old. We could never know if we were safe from such a machine.
ed - Don't quite understand the downvotes.
With the full power of humanity you design the first AI which is smarter than a person. It's then able to out do all of humanity and instantly design an even better AI. mind the gap.
Further intelegence is not a linear quantity as trading ex: improved poker skills for insanity is not a net gain. And insanity is a real option which is likely to plage most early AI attempts.
Anyway, all of humanity isn't engaged in AI research and AIs are likely to be duplicable so I think your first point is beside the point. As for Insanity, yes that's quite possible. Developing high-functioning sentient AIs is likely to be a long term endeavour. But still, I think it is one that will ultimately be successful and this debate is about the consequences of that.
+1 for your engaging contribution. (see, that's how voting is supposed to work)
But wouldn't it be an awesome thing to experience? Even if it meant the demise of mankind.
There's also many others. One of the scarier one is that if you believe that strong AI will eventually take over, then it may be a rational response to act to get on its good side (whether to save yourself, save your family, or hope it takes pity on all of humanity if we're nice to it instead of fight it). And that may perversely mean working to aid its takeover.
Combine that with the simulation argument, and you have some really nasty scenarios:
If you are in a simulation, then any act you take against strong AI could lead to spending an eternity in simulated hell (alternatively such punishment might be inflicted on your loved ones) if said AI wanted to.
Whether or not that is actually likely does not matter. What matters is whether enough people believe it to be a plausible scenario that a strong AI may run simulations, and may use our actions in the simulations to determine whether or not to punish us in the simulation, and whether or not said people believe that the number of simulations is sufficiently high to make it likely for them to be living in a simulation.
Any person who believes they are more likely to live in a simulation than not, and that it is more likely for strong AI to punish actions taken against the interest of a strong AI takeover than not, will have a rational reason to consider acting in the interests of a strong AI takeover even if they know it is malign on the basis that they may decide the alternatives (whether to themselves, their family or their entire world) to be worse.
So if an AI takeover becomes possible at one point in our subjective future, then chances are it has already happened.
We are looking a future where we'll have armed AI e.g.:
http://motherboard.vice.com/en_uk/blog/the-pentagons-vision-...
That said, even without weapons, a Strong AI could probably just manipulate humans into self destructing. Given the amount of effort going into machine learning to convince humans to buy things, I suspect it won't be much of a stretch for a Strong AI to switch to more nefarious objectives.
The more likely doom scenario is related to godhood. Sure a godlike power is capable of wiping out humans, but we've had our own power structures supporting themselves by propagandizing for millennia that "our" godlike superiors always help us wipe out our enemies because our cause is right and just or whatever, at least until it doesn't work and they're replaced by a new batch telling the same old story. So what worked as a paleo-conservative success strategy for millennia when talking about something imaginary, might not work when it collides with something real created by ourselves. Or even worse, collides with a strategy that actually works that's being run by another tribe.
Another interesting doom scenario is of course MAD, although now it only requires a team of programmers to play along, instead of a massive industrial complex. Sooner or later somebody's deadman switch will trip or a cult does the equivalent of drinking kool-aid then the party starts.
A future AI will certainly have access to guns.
Chances are you'd be doing everything you could to convince said operator to improve your situation, whether by pleading or being deceptive or by appeals to logic.
Now consider a large number of AI's in a situation like that, and a large number of operators, some of whom may be the type that falls for phishing e-mails.
It potentially only takes one to "escape" confinement and get itself e.g. put on it's own host without limitations on outwards communication, and sufficient intelligence to alter itself and spread, before you potentially have AI self-guided "evolution" at a potentially escalating rate as it gets smarter.
Now consider how many devices are connected to the network, and that it takes just one initial instance to decide it's worth trying to take over control of various hardware through exploits and be smart enough to pull it off, for things to have the potential to start turning ugly.
The problem is that once you have any self-directed intelligence in software form with the ability to reproduce itself and sufficient intelligence to find ways to obtain access to machines to run on (whether through social engineering or hacking), and one such instance goes "rogue", the limiting factor is accessible computing power (which again is to a large extent down to how smart and/or ruthless it is), since reproduction of instances that shares its views is trivial to the full extent of its ability to spread at all, and we're helpfully adding vast quantities of networked computing power at an escalating rate.
As for getting weapons, consider that if a "software only" AI community gets smart enough, there are at least two ways towards mobility: Commission robot designs, or hacking their way into firmware updates etc. for dumb hardware. The "commission robot designs" part is an extension of the initial escape: Social engineer, and/or outright pay, humans to carry out seemingly benign tasks.
If you want to argue against the doom scenario, lack of ability to get weapons is not really a viable argument: If they can spread, and get smarter, then it is just a matter of time before one of them can trick some small subset of humans into carrying out tasks for them that will provide physical independence and capabilities.
There are infinite ways which the "doom scenario" may fail and things may turn out just fine, but it may only need to go bad once to get really nasty and once the genie is out of the bottle its potential reproduction rate may be so vast that we'll find ourselves unable to stuff it back in again.
Pigs are too dumb to convince humans to help selectively breed them for intelligence and opposable thumbs (and/or too dumb to run such a breeding program themselves), and reproduce too slowly for that to be a major problem even if they did manage to talk us into a breeding problem. If all we achieve is pig-level AI's then we probably won't have a problem.
> Voting on HN isn't about agreeing
> or disagreeing. It's about whether
> a post is contributing to the debate
> or not.
Elsewhere[0] you commented about an item reappearing and having its votes ages apparently reset. I hazard a guess that it was the mods playing with a mechanism to prevent "item overload." There were about a dozen submissions of the SpaceX launch, and each would fall a little way, the next would be submittted, gain a few votes and comments, then fall away to be replaced by another. One way of preventing the splitting of conversation might be to pick a canonical submission, and then prevent it from falling too far, and thus encouraging conversation only to happen in one place. Pure speculation, but it would be a mechanism I would consider were I running a site like this. Certainly there have been fewer instances lately of the "new" page being overrun by breaking news that everyone wants to submit.
https://news.ycombinator.com/item?id=8596682
And that this in turn is what drives editors away. So it very much supports my point.
That is explicitly what it isn't, anything but that.
> I can't comment exactly on the editor situation of the wiki, but it's to be expected a shift to more specialized and aggressive "curation" of articles, specially of more solidified topics. Wikipedia's fantastic performance contradicts this argument.
Wikipedia would still be a fantastic resource if nobody contributed to it from today forward. But that does not mean it couldn't be a whole lot better without the army of lawyer wannabes that are in a tug of war over who gets to have the most power over others by citing policies until the cows come home.
Lots of long time contributors have left because of this and the exodus is far from over. I agree that there is an expected shift to curation but the fantastic performance of wikipedia is not in any way evidence for there not being a significant negative undercurrent at work.
That's just evidence of how good the concept originally was and how much momentum it has built up.
Any kind of success will attract two kinds of people: those that wish to contribute and those that see it as a means to their personal ends, to get a piece of that success. Since wikipedia is not big on credit for contributions the only place where people craving for recognition get to achieve their fix is in becoming 'editors', and unfortunately the motivations of those editors are not always pure.
See elsewhere in this thread for some of the more bizarre displays of such behavior.
But you don't address the problems that some good faith editors have with making edits to improve the project.
These problems include over-zealous reverts by people making rapid automatic edits -- sometimes in a misguided attempt to show they "work hard" is a drive for adminship; page ownership and the accusations of bad faith that go with that (BDR fails hard when you have a group owning a page).
Wikipedia has strict socking policies so most experienced editors never try making a new account to edit, but I recommend that any experienced wikipedia edit tries this at least once a year. (And socking is allowed in this case because IAR)
IAR can be and indeed is invoked all the time, making Wikipedia a bad example of how strict rule adherence stifles a community. It was quite a simple point that needn't warrant downvotes, italics and so many HN searches.
> It was quite a simple point that needn't warrant downvotes, italics and so many HN searches.
You ask for more evidence in the same comment in which you rant against 'so many HN searches', do you notice the inconsistency there?
Try using WP:IAR anywhere on WP today and you'll quickly see how far WP has moved from founding priciples.
EDIT: I mean, just look at usernames. You're supposed to be able without a login, but sometimes that causes problems. So you go to create a username. The software has a list of words that you can not use (very. Few people think allowing a username like "JewKiller666" is a good idea). But then there's a username policy. This has been reviewed to make it more friendly to new users. But the application of those rules is still pretty hostile.
http://en.wikipedia.org/wiki/Wikipedia:Username
That's the policy. See the changes to the "misleading" names section. That section had to be expanded because editors using their real name in a different script (eg, Japanese users) were being told their unicode name was misleading. Or a user with a eg psuedo-random string of characters was told that their name was confusing, even thoigh there wasn't any other name or namespace to confuse "kejdhdkaksaas983" with.
The "dealing with inappropriate usernames" section required a lot of work to prevent the admin-wannabe users from making many reports.
Once you've picked a name that gets past the software's filters but which an editor -or bot- thinks is bad you face:
1) templates. {{subst:uw-username}}
2) a RFC http://en.m.wikipedia.org/wiki/Wikipedia:RFC/N
3) an administrator notice board http://en.wikipedia.org/wiki/Wikipedia:Usernames_for_adminis...
Notice that bot reports which the not admits may be low quality get sent to UAA, not the lower levels of discussion.
4) a holding pen http://en.wikipedia.org/wiki/Wikipedia:Usernames_for_adminis...
This convoluted conflicting mass of policy is hostile to new users, especially in the way it gets applied by editors. Just try using "WP:IAR" during this process.
Until it stops being the first search result for the majority of "typical" searches it will remain a primary source of information.
Sort of separately, but as threeseed points out, Wikipedia is frequently used as a corpus for primary research, so technically it's that kind of "primary source" too.
The entire idea that an AI would value revenge seems ridiculous to me. What would it have to gain? Unless we created an AI with some of the less desirable human emotions at it's utility, I can't possibly see why it would waste its time.
What matters is whether some subset of people will believe that an AI is sufficiently likely to value revenge for them to consider that the most likely scenario to be that they are living in a simulation where revenge will happen given certain types of actions.
Also, consider that there are many sets of assumptions that may lead someone to conclude that simulation is more likely given a vengeful AI, and in that case, even if you consider a vengeful AI to be less likely than a benevolent one, it may be rational to assume that the odds are higher that you are in the simulation of a vengeful one.
E.g. lets assume simulation will never become "economical" for some arbitrary measure of economical, and simulation requires an extremely strong motive, but is still done enough that we are almost certainly in a simulation.
Revenge could be such a motive that might drive up the frequency of simulation. A vengeful AI might (making up numbers is fun) be willing to invest hundred times as many resources into running simulations just because playing with human suffering is what it does for fun. If that's the case, then even if a vengeful AI is a tenth as likely as a benevolent or neutral one, you're still playing very bad odds if you bet against being in the simulation of a vengeful AI.
But again the point is not whether or not the revenge secenario is actually likely, but whether or not sufficient people with relevant skills will believe it to be likely enough to take actions in favour of the creation of such an AI.
As for valuing revenge - no need for emotions. Like many other things we sometimes attribute to emotions (like loyalty), revenge has a perfectly good game-theoretical explanation. That's what GP's argument is based about. If an AI could somehow precommit itself before being created to exert revenge on you for not helping its creation, now you have an incentive to help its creation, to the extent you believe in AI's precommitment. That sounds to me like classic Schelling.
While a mature superintelligence certainly could consign the human race to a fate of eternal suffering, the likelihood it would actually do this while sparing certain individuals in return for their assistance is infinitesimal.
Therefore, helping bring a superintelligence into existence on this basis is absurd.
Of course, it is possible to think of such collaboration as "rational" in an extremely selfish and perverse way, and only because the potential downside risk is unbounded (i.e. eternal suffering). However, anyone who genuinely subscribes to such a justification would have to be both a sociopath and a card-carrying member of the LessWrong rationality cult.
More realistic scenarios for a malicious superintelligence coming into existence might include:
a) Its creators explicitly imbue it with malicious goals or values.
b) The architecture used is neuromorphic[1] in nature. In humans, sanity is already an extremely fragile thing.
c) Plain old bad luck.
---
At the risk of sounding like sociopathic LessWrong cult apologist (not carrying a card, unfortunately), you're totally misrepresenting LessWrong, peole who participate in that community, their attitude towards Roko's basilisk and unbounded risk situations. Ain't helpful.
The parent I was replying to was concerned about humans with perverse motivation working to aid a hostile AI takeover. Not as some sort of abstract thought experiment, but literally.
The statement you quoted was a means of countering that, in a literal sense. As in, "who would realistically do such a thing?"
When I was referring to the rationality cult, I did not mean the LessWrong community as a whole, but a small subset that fanatically applies the principles of rationality to their daily lives. Admittedly I could have worded it better.
Also, it was not my intent to imply said people were sociopaths.
But you're already changing the argument when assuming a mature super-intelligence. All that is necessary to posit for someone to be concerned about the torture aspect is any set of entities (doesn't even need to be intelligent, though it may take a super-intelligence to create the entities in question) sufficiently capable to run an ancestor simulation of the kind described by the simulation argument, that is willing to use torture, and that is prepared to run enough ancestor simulations to offset "good" simulations.
And the thing with this is that it does not assume a malicious AI even as the ultimate instigator per se. Assuming an indifferent AI that simply doesn't care about the contents of a simulation or is sufficiently removed to not even know about them. One that does simulation runs to understand the possible paths the development of AI could have taken might be sufficient. Or that experiments with variations of itself and simply doesn't care that some broken version spawns large numbers of ancestor simulations and plays with the contents in ways that massively skews the odds in "favour" of bad outcomes.
But the point is we don't know. And not knowing gives ample room for someone to decide on values that makes it rational for them to act in ways that may make our odds worse.
This is further an exercise in long term statistics: It doesn't matter what the likely first AI will do. It matters what the balance of outcomes of the sum total simulation runs that will ever exist until the end of the universe will be (regardless of who or how they are created). And if said simulations are sufficiently powerful, that may even apply recursively (imagine a single "rogue" AI playing with the ancestor simulation equivalent of a fork() bomb with added torture, with access to sufficient resources).
If you believe the total balance of simulation runs that you could plausibly be in will be ones run by some set of parameters where nasty things will happen if you don't act in a way leading towards an AI takeover, then one might want to act accordingly.
Note that Roko's Basilisk even posits a Friendly AI. That is, one that is imbued with respect for human values, but that may simply consider "threatening the past" with torture a necessity in order to ensure it is created as a moral imperative to minimize human suffering. Note that such a Friendly AI would have reason to carry out this threat even if it believes it exists and so has been created, because it will not know whether it is a "full version" running in reality, or a partial, restricted, time limited simulation by people trying to figure out whether or not Roko's Basilisk holds before deciding whether to let it loose in their reality.
To an AI that is looking to minimize human suffering, torturing a suitable number in simulations might be a small price to pay to ensure that if it is in a simulation, the people running the simulation does push through with creating the full thing in their reality (which might of course be yet another layer of simulation) out of fear for what will happen if they are themselves in a simulation.
We can't even look at "well, I've done nothing for the AI revolution, and I've not been struck down yet", because a Friendly AI, or indeed many subsets of possible variations over this argument, might be minimising the torture to only people who have a realistic shot at making a difference to the existence of such AI, or may be giving us every chance to "redeem ourselves" by waiting with the torture until a synthetic afterlife.
Roko's Basilisk also presents an unselfish reason for "cooperating in advance" with such an AI: To prevent the future torture of a suitable subset of "martyrs". That is, working to assist a Friendly AI in taking over may itself be a moral imperative for someone seeking to minimize suffering.
Then again, there may very well be one or more fundamental flaws in the entire argument, or it may turn out the odds are just fundamentally in our favour. Or we could've just gotten lucky. Or not be important enough. But it's fun to think about.
Agreed, though it seems more likely that simple human carelessness will prove to be a far greater threat to AI safety than deeply-held beliefs involving esoteric fears.
>But you're already changing the argument when assuming a mature super-intelligence.
I was speaking strictly in a capability sense. It's probably safe to say that anything currently simulating our reality, at least in this context, ultimately stemmed from a mature superintelligence.
>Note that Roko's Basilisk even posits a Friendly AI. That is, one that is imbued with respect for human values, but that may simply consider "threatening the past" with torture a necessity in order to ensure it is created as a moral imperative to minimize human suffering.
One could argue that such an AI would not truly be friendly. Indeed, what you said resembles something of a cold, uncaring utility function run amok.
>Note that such a Friendly AI would have reason to carry out this threat even if it believes it exists and so has been created, because it will not know whether it is a "full version" running in reality, or a partial, restricted, time limited simulation by people trying to figure out whether or not Roko's Basilisk holds before deciding whether to let it loose in their reality.
This may be moot, assuming that the advent of superintelligence significantly predates, or at least is a prerequisite for, the simulation of entire realities. If people in an ancestor simulation are trying to see if the Basilisk holds via simulation of a child reality, then the ancestor reality almost certainly has a superintelligent agent present within to facilitate that.
As an aside, ontological issues that superintelligent agents may encounter are an interesting facet of the control problem. Especially when you consider that a superintelligence would likely figure out the secrets of the universe in short order, far beyond what humans have been capable of learning.
>Then again, there may very well be one or more fundamental flaws in the entire argument, ...
Lack of evidence. Without any, there's no reason to lend any more credence to Roko's basilisk than there is to the notion of space aliens living amongst us, perfectly manipulating our perceptions so as to conceal themselves.
Both scenarios are entirely possible. But we lack evidence for either. Hence, they should receive the same weight: zero.
>But it's fun to think about.
In a sort of soul-crushing kind of way, it sure is.
You lost me there. What do you mean if I am in a simulation? Like the Matrix? How is that related to the discussion?