Nick Bostrom

Then, Sir, I encourage you to get thee quickly to this Partially Examined Life podcast episode[1] with Nick Bostrom[2]. Only came out on the 6th of January and it makes for super interesting listening. It's based off of the text with his 2014 book in which he's gone to great lengths to think philosophically about this very existential threat -- in his opinion, man's most pressing. Part of his reasoning goes like this. Any super smart goal oriented entity is going to desire/construct its own preservation as a sub-goal of its main goal (even if that goal is something as mundane as making paperclips) because if it ceases to be its goal will be put into jeopardy. To that end this super smart entity will then figure out ways to disable all attempts to constrain its non-existence. A lot of conclusions can be logically reached when one posits goal directed behaviour which we can assume any super intelligent agent is going to have. He talks about `goal content integrity'. Bostrom argues for an indirect normative[4] approach because there is no way we can program or direct something that is going to be a lot smarter than ourselves and that won't necessarily share our values and that has any degree of goal-oriented behaviour, motivation and autonomous learning. Spoiler alert: Essentially I think he argues that we have to prime it to "always do what _you_ figure out is morally best" but I could be wrong. There are also (global) sociological recommendations because, humans have been…

Relevant philosophy paper: "The Vulnerable World Hypothesis" by Nick Bostrom [0]. In that paper, Bostrom floats the idea that it might be in humanity's best interest to have a strong global government with mass surveillance to prevent technological catastrophes. It's more of a thought experiment than a "we should definitely do this" kind of argument, but it's worth taking the idea seriously and thinking hard about what alternatives we have for maintaining global stability. [0] https://nickbostrom.com/papers/vulnerable.pdf

If life is abundant, there must have been something to prevent it from spreading, now or in the distant past. The fact that we evolved in an apparently empty universe, rather than in one that's been totally colonized, implies something stops this from happening. This thing is called the Great Filter, and it's apparently extremely effective. If the GF is in front of us, in a universe with lots of ETI, our chances of evading it are extremely small. After all, no one else has; why would we be so special? Nick Bostrum has called detection of ETI, or even complex ET life, the worst news mankind would ever receive. It would be a death sentence for our future. https://nickbostrom.com/papers/where-are-they/

He did write an essay [0]. Because it was very short and not deeply insightful due to such length, he wrote a longer book talking about the concepts. [0] https://nickbostrom.com/views/superintelligence.pdf

"Technology policy should not unquestioningly assume that all technological progress is beneficial, or that complete scientific openness is always best, or that the world has the capacity to manage any potential downside of a technology after it is invented." Don't watch the anime if you haven't read the manga: https://nickbostrom.com/papers/vulnerable.pdf

Every single time the possibility of curing aging comes up, we get those misanthropes who demand loudly that It's Very Important That You Must Die, for The Greater Good, or something. And every time I say the same thing: say the situation were reversed: everyone is living forever, and it starts (arguably) causing those problems you have in mind. Your solution to those problems is to kill everyone over a certain age!? https://nickbostrom.com/fable/dragon

> If you're not a programmer, a doctor, or possess some other rare skill, you're fucked. it's coming for these too > A continuously growing underclass with continuously declining prospects is a recipe for violence and revolution if we don't start taking this problem more seriously. the complete breakup of Microsoft, OpenAI, Google and Amazon will be on the political agenda in 2028 failing a political solution: violence will follow there was a post yesterday that wished GPT-4 was never invented... it's one of Nick Bostrom's black marbles https://nickbostrom.com/papers/vulnerable.pdf by the end of the decade we're all going to be wishing it had never been invented

Doesn't matter if AI can cure it, a suitable number of the right initial infected and a high enough R naught would kills 100s of millions before it could even be treated. Never mind what a disaster the logistics of manufacturing and distributing the cure at scale would be with enough people dead from the onset. Perhaps the more likely scenario anyway is easy nukes, quite a few nations would be interested. Imagine if the knowledge of their construction became public. https://nickbostrom.com/papers/vulnerable.pdf I agree with you though, the promise of AI is alluring, we could do great things with it. But the damage that bad actors could do is extremely serious and lacks a solution. Legal constraints will do nothing thanks to game theoretic reasons others have outlined.

If you have some time, this is worth a read: https://nickbostrom.com/fable/dragon It’s a nice, short metaphorical story for our acceptance of death.

I love this style of reasoning. Status quo bias is a severely strong psychological effect on some people in some cases. This is a nice way to "bust the cache" ;) There's a really cool philosophy paper "The reversal test: Eliminating status quo bias in applied ethics" - using a reversal like the one you describe (imagine a world where the opposite was true). But then the authors also do a double-reversal (imagine the world is identical to ours, but there's a policy about to be implemented which will make it into "mirror world" and we have the option to do nothing and let it happen, or act and prevent the reversal). Great thinking tool! https://philpapers.org/rec/BOSTRT full paper PDF: https://nickbostrom.com/ethics/statusquo.pdf

>Economically and ethically, we should prioritize this research above most else. This research and all similar research (and sentiments like yours, which I strongly agree with) remind me of the classic Fable of the Dragon-Tyrant.[0] It feels as though we are finally on the cusp of learning the shape of the armaments that will one day slay the dragon. And I wish the world could band together the way we attacked COVID to attack this problem. >the king answered in a broken voice: “Yes, we did it, we killed the dragon today. But damn, why did we start so late? This could have been done five, maybe ten years ago! Millions of people wouldn’t have had to die.” [0]https://www.nickbostrom.com/fable/dragon.html

One could further raise an association with Nick Bostrom's vulnerable world hypothesis: > One way of looking at human creativity is as a process of pulling balls out of a giant urn. The balls represent possible ideas, discoveries, technological inventions. Over the course of history, we have extracted a great many balls – mostly white (beneficial) but also various shades of gray (moderately harmful ones and mixed blessings). [..] What we haven’t extracted, so far, is a black ball: a technology that invariably or by default destroys the civilization that invents it. The reason is not that we have been particularly careful or wise in our technology policy. We have just been lucky. https://nickbostrom.com/papers/vulnerable.pdf

Not following all arguments of the author, but do agree with the main theme. I'd rather live than not live. On the same topic, this is a great read: https://www.nickbostrom.com/fable/dragon.html

You must be aware of this but here's who I consider ( to my knowledge ) to be at the forefront of those questions and the best documented / systematic and comprehensive resource ; https://nickbostrom.com/existential/risks.html As a side note, and from someone with negative karma, you may wish to peruse my recent comments about Covid, some seem to think they are unthinkable ideas while I persist in saying they are worth considering, although it is about an ongoing black swan and therefore probably not within the scope of your survey, they fit your description in that way.

I also recommend Nick Bostrom's article "The Future of Human Evolution", predating OP by ten years, which contains essentially the same arguments and conclusions. https://www.nickbostrom.com/fut/evolution.html

That passage is something of an afterthought, from the epilogue of his book, but it's still disappointing he doesn't credit Robin Hanson for coming up with the Great Filter concept[1], or mention the x-risk[2] community which contemplates this stuff. [1] http://mason.gmu.edu/~rhanson/greatfilter.html [2] https://nickbostrom.com/existential/risks.html / https://wiki.lesswrong.com/wiki/Existential_risk

Agree. And to take it even further, consider Bostrom's "Vulnerable World Hypothesis" where technology could arise that makes it trivial for individual actors to cause mass destruction. From the abstract: "A general ability to stabilize a vulnerable world would require greatly amplified capacities for preventive policing and global governance." https://nickbostrom.com/papers/vulnerable.pdf

"The last enemy that shall be destroyed is death." Well, maybe not the last one, but still an important one. Nick Bostrom has a great story describing the underlying philosophy in the fight against aging: https://nickbostrom.com/fable/dragon.html And here it is in video form as well: https://www.youtube.com/watch?v=cZYNADOHhVY

There's this: http://www.nickbostrom.com/papers/survey.pdf >We thus designed a brief questionnaire and distributed it to four groups of experts in 2012/2013. The median estimate of respondents was for a one in two chance that high-level machine intelligence will be developed around 2040-2050, rising to a nine in ten chance by 2075. Experts expect that systems will move on to superintelligence in less than 30 years thereafter. They estimate the chance is about one in three that this development turns out to be ‘bad’ or ‘extremely bad’ for humanity. This survey is a few years old. Discussion and knowledge about AI risk has increased considerably since then. And AI itself has made remarkable progress in that time as well. It's amazing how much progress there has been in AI in just the last 5 years. Who knows where it will be in 40 years. Look where computer technology was 40 years ago.

Strong AI is not necessarily a bad thing. Instead of worrying about questions 1 & 2, we could be thinking less about constraint and competition with AI and more about cooperation and goal-orientation: e.g. the work of Yudkowsky (https://intelligence.org/files/CFAI.pdf) or some of the thoughts provided by Nick Bostrom http://nickbostrom.com/. Goal-orientation is preferable to capability constraint because the potential benefits are far larger. Tl;dr: I, for one, welcome our robot overlords (so long as they don't behave like our robot overlords).

Nick Bostrom has talked about some of the issues around attempting to control super intelligent AIs, and the closely related issue of how to determine if we are AIs inside a virtual environment (closely related because containment is one approach to controlling AIs that we don't understand) http://www.techthefuture.com/technology/can-we-control-superior-artifi… http://www.nickbostrom.com/

While thinking about these things is important, it would be good not to lose track of the greater picture. I couldn't possibly say it as well as Nick Bostrom, so I won't try: Death is bad; it needs to go. This seems like an apropos time to re-read http://www.nickbostrom.com/fable/dragon.html

If you're interested in going a bit deeper on this, Nick Bostrom (http://nickbostrom.com/) has pulled together an intriguing case: http://www.simulation-argument.com/simulation.html A provoking question: if we found evidence that convinced us, beyond a reasonable doubt, that we were living in a computer simulation, would you change your behavior?

There are huge risks that nick bostrom talks about http://www.nickbostrom.com/ The only difference is between that and the cold war is that people don't give them due attention because they are non-anthropomorphic (a fight with nature instead of against other humans). If it was the soviets who were behind the asteroid threat, and they were causing old age and cancer, and they were behind worldwide poverty, the US government would be mobilizing trillions of dollars to tackle those problems.

Reminds me of this excellent read about how our perception of age and dying taints our ability to even consider it a problem we can work on and correct: http://www.nickbostrom.com/fable/dragon.html

Liiink! http://www.simulation-argument.com/simulation.html (From Nick Bostrom's home page: http://www.nickbostrom.com/ )

I cannot recommend the writings of Max Tegmark on the multiverse enough. Among people doing science at the heights of their fields, he is perhaps a uniquely talented popular writer. He even keeps it readable on his scientific papers, which read mostly just like his popular papers except shorter and with more math. http://arxiv.org/PS_cache/arxiv/pdf/0905/0905.1283v1.pdf is his overview of the levels of multiverses. If that properly blows your mind, see Bostrom's Infinite Ethics. http://www.nickbostrom.com/ethics/infinite.pdf

This correlate fine with what we observe and with the Fermi paradox. That is, there are no indications of intelligent life elsewhere in the universe. When Frank Drake first turned his radio telescope to the sky and started to listen, he expected the ether to be full of radio signals from other civilizations. To his astonishment it was silent. Obviously there is a "great filter" in place which prevent intelligent life to evolve in abundance. And this article seems to point to such a filter. Nick Bostrom has an interesting philosophical take on this http://www.nickbostrom.com/papers/fermi.pdf

I've been convinced for sometime now that "effective altruism" (Lesswrong-ism? Yudkowskianism?) is the communism of the 21st century. Lots of parallels here beyond the obvious common thread of utilitarian collectivism: - Censorship: the concept of "infohazards" is super-popular https://nickbostrom.com/information-hazards.pdf https://www.lesswrong.com/tag/information-hazards - Social control + the creation of an 'other': They are absolutely convinced that sentient AGI will inevitably kill everyone without any real evidence. However, if we manage to create sentient AGI, what these guys plan to do with it is best described as modern slavery https://www.lesswrong.com/tag/ai-boxing-containment#:~:text=to%20free%… - Surveillance: They justify it using Bostrom's "vulnerable world hypothesis": https://nickbostrom.com/papers/vulnerable.pdf?platform=hootsuite (TFA also discusses it well) - Desire for government power: TFA is super clear about that. More interestingly, here's the FTX Future Fund on space governance: "We’d love to see workshops or a mock constitutional convention where sharp people think hard about how to structure international governance institutions for the long-term future, or how to govern space settlement. As explained in more detail on our areas of interest page, we believe that the onset of space settlement could be a watershed moment in human history. We want people to start thinking about how it should…

> potential risks of AI sounds the opposite of the position of someone who would think technology can solve everything. Yes, that's a logical conclusion, but it's not right because of the weird "longtermist" logic. Their "AI risk research" is about AI alignment, which is a "longtermist" priority. To repost my previous quote: > Many longtermists reckon that the antidote to the threat from technology is more technology; mastering artificial intelligence will forestall a malevolent AI from enslaving us. They also believe that technological acceleration is morally good in its own right. This is the "acceleratist view" of AI research, of which Nick Bostrom is a prominent proponent: basically he views progress as inevitable, so it's best to accelerate AI that is aligned with human values. See https://nickbostrom.com/papers/openness.pdf He also values future lives equally to existing lives (the "longtermist" view) so any timme where we aren't expanding to the stars is morally repugnant: > One recent paper speculates, using loose theoretical considerations based on the rate of increase of entropy, that the loss of potential human lives in our own galactic supercluster is at least ~10^46 per century of delayed colonization. [snip] The effect on total value, then, seems greater for actions that accelerate technological development than for practically any other possible action. Let me requote that: The effect on total value, then, seems greater for actions that accelerate…