MIRI

Interestingly, he once called MIRI[1] (formerly SIAI) “the best hope that we have”[2]. [1] https://intelligence.org/ [2] http://www.vetta.org/2009/08/funding-safe-agi/

I agree that superintelligence could bring enormous benefits to humanity but the risks are very high as well. They are in fact existential risks, as detailed in the book Superintelligence by Bostrom. That is why we need to invest much more research efforts on Friendly AI and trustworthy intelligent systems. People should consider contribute to MIRI (https://intelligence.org/) where Yudkowsky, who helped pioneer this line of research, works as a senior fellow.

I think (not sure, as I don't closely follow the work) AGI people and MIRI work along the lines. (i.e: how to make sure a super AI doesn't go wrong, but comes with Objective functions beneficial to humans. https://intelligence.org/)

The recent survey on Neurosymbolic Programming is a great introduction to the field: https://twitter.com/swarat/status/1474105098401943555 (pardon the Twitter link). I don't think the full version is publicly available, but I suspect you might be able to get access to it if you DM someone involved. Other than that, here are some names of people working in this field that you might be able to find representative work from: - Mayur Naik, UPenn (whose group I think were behind Scallop?) - Swarat Chaudhuri (and maybe also Isil Dillig), UT Austin - Luc de Raedt, KU Leuven (whose earlier work on Statistical Relational AI is very similar to the Scallop style of neurosymbolic work - I think there's a good textbook available on this) - Guy van den Broeck, UCLA - Armando Solar-Lezama, MIT If you're more interested in the logic side of things then maybe some of MIRI's (https://intelligence.org/) older work may be of interest. There's also a lot of people who are interested in neuro-symbolic stuff in the wider sense. You can find out more here: http://www.neurosymbolic.org/index.html. If you sign up for the mailing list, there are monthly (or bi-monthly? I can't remember) talks that are open to the public. You can even find recordings of past talks on Armando Solar-Lezama's youtube channel. Hope this helps!

Stuart Russell (co-author of AI:MA, one of MIRI's research advisors) argues on http://edge.org/conversation/the-myth-of-ai#26015 that AI systems with "the ability to make high-quality decisions" (where "quality refers to the expected outcome utility of actions taken" and the utility function is represented in the system's programmed decision criteria) raises two problems: "1. The utility function may not be perfectly aligned with the values of the human race, which are (at best) very difficult to pin down. "2. Any sufficiently capable intelligent system will prefer to ensure its own continued existence and to acquire physical and computational resources – not for their own sake, but to succeed in its assigned task." The first of those is what Bostrom calls "perverse instantiation" and Dietterich and Horvitz call the "Sorcerer's Apprentice" problem (http://cacm.acm.org/magazines/2015/10/192386-rise-of-concerns-about-ai… The second of these is what Bostrom calls "convergent instrumental goals" and Omohundro calls "basic AI drives." The first of these seems like a fairly obvious problem, if we think AI systems will ever be trusted with making important decisions. Human goals are complicated, and even a superintelligent system that can easily learn about our goals won't necessarily acquire the goals thereby. So solving the AI problem doesn't get us a solution to the goal specification problem for free. The second of these also has some intuitive force;…

I love me some capitalism critique, but I think it’s important to understand many scientists are legitimately terrified of an intelligence explosion and the resulting singularity. You can disagree with their arguments of course, but it’s best if we start by agreeing that they do have arguments. For example: Yudkowsky 2013, Intelligence Explosion Microeconomics (long, but with a short intro): https://intelligence.org/files/IEM.pdf Vinge 1993, The Coming Technological Singularity (shorter, bit less rigorous): https://users.manchester.edu/Facstaff/SSNaragon/Online/100-FYS-F15/Rea…

> Our objective is to convince major powers to shut down the development of frontier AI systems worldwide before it is too late. Awaiting urgent update to the now-contradictory org mission statement: "We do foundational mathematical research to ensure smarter-than-human artificial intelligence has a positive impact." https://intelligence.org/

>neither of which use alarmist terms like "existential threat" "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war." https://www.safe.ai/statement-on-ai-risk Not sure you're making a meaningful distinction here. - - - Of course we all have our own heuristics for deciding who's worth paying attention to. Credentials are one heuristic. For example, you could argue that investing in founders like Bill Gates, Mark Zuckerberg, and Steve Wozniak would be a bad idea because none of them had completed a 4-year degree. In any case, there are a lot of credentialed people who take Eliezer seriously -- see the MIRI team page for example: https://intelligence.org/team/ Most notable would probably be Stuart Russell, coauthor of the most widely used AI textbook (with Peter Norvig), who is a MIRI advisor.

That is the hard question. Of course a tiny fraction of people are trying for more than a decade now, for example - https://intelligence.org/. However, there is no solution, or even a glimpse of a solution.

I love this paper https://intelligence.org/files/PredictingAI.pdf that shows that for the past 50 years, both experts and non-experts have been consistently predicting the advent of real AI to be within the next 25 years.

This is also known as the Agent-Principal problem. [0] Usually this is "solved" via "aligning incentives". [1] But as we all know from James Cameron's documentaries as he followed the Connors, it's not that easy to notice when things go awry. [0] https://en.wikipedia.org/wiki/Principal%E2%80%93agent_problem [1] https://intelligence.org/research/

It’s quite true. A take I found insightful (I think it might have made the top page when it was first posted): https://intelligence.org/2017/10/13/fire-alarm/

See also Eliezer Yudkowsky's excellent response to that article: https://intelligence.org/2017/12/06/chollet/

Please don't... every time a thread like this shows up on HN, I end up impulsively buying some of the recommended books. I'm usually very happy about the purchase, though. :). Humor aside; not sure if favourite of all times, but definitely impacted my thinking a lot: - "Rationality: From AI to Zombies" - https://intelligence.org/rationality-ai-zombies/ - read this when it was still a bunch of posts by 'Eliezer on LessWrong. It cleaned up my thinking quite a bit, and introduced to couple new ideas from economics, sociology and epistemology. - SICP, obviously. - "How to Win Friends and Influence People", and "How to Stop Worrying and Start Living" - from the father of the whole genre of self-help/personal development, and one whose books are still probably the only good ones in this genre, Dale Carnegie. They explain exactly what it says in their titles.

On a related note, I still love this interview: Scott Aaronson on Philosophical Progress: https://intelligence.org/2013/12/13/aaronson/

Smartest person I know works on smarter-than-human artificial intelligence at the Machine Intelligence Research Institute [0]. Previously at Microsoft and Google, he decided to get involved in smarter-than-human AI and is now the Executive Director at MIRI. Definitely inspiring! [0] - https://intelligence.org/

MIRI recommends Game Theory: An Introduction by Steven Tadelis https://intelligence.org/research-guide/

This paper is published during an era where the dominant global empire is founded and (mostly) run on principles opposite to ethnocentrism, at least internally. All of the current POTUS' grandparents were born in Europe, and the previous one's father was Kenyan. The most famous entrepreneur of our generation was born of Syrian parents. The wealthiest self-made woman in America according to Forbes (I did have to Google it), Marian Ilitch, had Macedonian parents. The best strategy in the Iterated Prisoner's Dilemma is tit-for-tat [1]. The winning groups of today have found ways to cooperate-cooperate systematically, overriding the advantages conferred by the behaviour observed in the paper. [1] It gets more interesting when both sides know how the other thinks: https://intelligence.org/files/ProgramEquilibrium.pdf

> This is sort of like the absurdity in a rational agent always knowing what is best for them: it disregards computational complexity. Somewhat off-topic, but have you seen this? Seems interesting: https://intelligence.org/2016/09/12/new-paper-logical-induction/

Σ$0.02: Perhaps a way to contribute to the world would be to figure out how to get rid of Facebook: https://stallman.org/facebook.html Σ$0.04: Mark Zuckerberg mentions talk on the threat of AI as fear mungering, however, https://intelligence.org/ (MIRI) argues that advanced AI is a legitimate existential threat. Note that when MIRI speaks of "AI", they are not talking about fancy neural networks, but the fact that since humans are no where near any kind of upper bound on intelligence, and the space of possible Turing machines and self modifying algorithms is overwhelmingly large, plus reams of other arguments you , humans will eventually be able to produce computer programs with the capability of destroying civilisation. N.B: This is my brief summary of what their stance seems to be, and isn't a direct quotation or paraphrasing of MIRI's official opinion.

"AI: A Modern Approach (...) authors, Stuart Russell and Peter Norvig, devote significant space to AI dangers and Friendly AI in section 26.3, “The Ethics and Risks of Developing Artificial Intelligence.” https://intelligence.org/2013/10/19/russell-and-norvig-on-friendly-ai/ In addition, there is the AI Open Letter, which is signed by many "who actually works or has done serious research in ML", including Demis Hassabis and Yann LeCun. From the letter: "We recommend expanded research aimed at ensuring that increasingly capable AI systems are robust and beneficial: our AI systems must do what we want them to do. " http://futureoflife.org/ai-open-letter/ Are the experts concerned about a "Skynet scenario"? No. But there is certainly genuine concern from the experts.

What you're trying to solve is something very similar to the recursive self improvement problem that MIRI[1] and friends are trying to solve in the sphere of "friendly" artificial intelligence. If something rewrites its own source code, how can you assert invariants that can be relied upon? So going and looking over there at what they've come up with may be fruitful. [1] https://intelligence.org/

Yes, very disappointing. Generally anything with "AI" in the title means the HN comments won't be worth reading. It's a big problem, and I'm not sure how solvable it is. Basically, the paper discusses ways in which learning agents "will not learn to prevent (or seek!) being interrupted by the environment or a human operator. We provide a formal definition of safe interruptibility and exploit the off-policy learning property to prove that either some agents are already safely interruptible, like Q-learning, or can easily be made so, like Sarsa."[1] It's an interesting result, and can probably be extended to other less hype-worthy scenarios. [1] http://intelligence.org/files/Interruptibility.pdf

https://intelligence.org/rationality-ai-zombies/ and http://hpmor.com/ gave me a great model of how to plan and think about difficult issues. Excellent (and very long, each one is the length of 5+ books) primers on rationality.

I’d strongly recommend Bostrom’s book ‘Superintelligence’ for a good overview of the problem-space. My favourite high-level solution to the virtuous AI problem is Yudkowsky’s ‘Friendly AI’ approach[1]. The best part of this strategy is ‘Coherent Extrapolated Volition’ [2] which would have an AI want what we would want an AI to want. By doing this, we can offload the problem of working out how to locate and implement our values over to the AI. [1] https://en.wikipedia.org/wiki/Friendly_artificial_intelligence [2] https://intelligence.org/files/CEV.pdf

Now that it's finally been released, I recommend Rationality: From AI to Zombies: https://intelligence.org/rationality-ai-zombies/

While I agree that we are not currently ready for the arrival of superintelligence, I think it's worth emphasizing that there is time to prepare, that there is useful preparation to be done which may get things to a point where it will be safe (or at least marginally safer), and that if you are feeling worried, it may be best to channel that feeling into useful action and get involved with the research! Good places to start are the Machine Intelligence Research Institute's Research Guide (https://intelligence.org/research-guide/), which summarizes the aspects of the problem they've worked on and think are important; and the Future of Life Institute's Research Priorities (http://futureoflife.org/static/data/documents/research_priorities.pdf)… particularly Section 3.

This is a relatively old manuscript, and I would recommend instead reading, or at least first reading: * http://intelligence.org/files/ProblemClassDominance.pdf * http://arxiv.org/abs/1401.5577 (The original TDT manuscript occurred via pleas for 'something, anything to be published' that were satisfied by cleaning up an unfinished book draft and publishing it as-was.)

There are smart people working on it, see miri: https://intelligence.org/ But it's a very hard problem. To get a better feel for the problem I suggest you read Superintelligence by Nick Bostrom, it's what convinced Elon Musk of the dangers of AI: https://twitter.com/elonmusk/status/495759307346952192

This may be wrong for complex reasons. Similar minds making the same decision for the same reasons do not _causally_ influence each other, but that doesn't change the outcome, nor does it make it any less bad. For a fairly deep view into how one might formally describe this particular aspect of decision making, I strongly recommend: https://intelligence.org/files/TDT.pdf