AI Impacts

Someone has been working on survey forecasts, since 2016, https://aiimpacts.org/. As fraught or self-selected as the group might be, someone labors on it. Unfortunately, fortunately, expectedly, or otherwise, the only people writing about this in a concerted way are the people taking it seriously. And maybe Gary Marcus, whose negative predictions repeatedly became milestones surpassed. You can read a review of the survey efforts, and complications of the results at https://asteriskmag.com/issues/03/through-a-glass-darkly Yogi Berra supposedly said that “prediction is very difficult, especially about the future.” and the surveys are finding that even achieved milestones are still forecast as a few years out, by most of these carefully sampled experts. ...welcome to the conversation :D.

A recent poll of published AI researchers showed a median estimate of a 5% chance AI causes human extinction, with half believing the chance is greater than 10%, and an average estimate of 37 years until superhuman AI is created. https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/ A poll of NLP researchers showed 37% agreed that catastrophic outcomes of AI comparable to nuclear war were plausible. https://arxiv.org/abs/2208.12852

According to a study of 700 ML researchers (not AI safety) the median chance that that the long-run effect of advanced AI on humanity will be “extremely bad (e.g., human extinction)" is 5%. And 48% of respondents gave an answer of 10% or more. Even if this is 2 orders of magnitude too high it's worth a lot of our time. But these are probably mainly normy researchers not AI safety-ists. https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/#Data

Trends in algorithmic progress https://aiimpacts.org/trends-in-algorithmic-progress/ is the best work on this topic I am aware of. 2000x is believable, but that doesn't mean the latest algorithm will run on Apple II. Algorithmic speedup is often hardware relative. For example better cache locality is less important in older hardwares.

> "If somebody builds a too-powerful AI, under present conditions, I expect that every single member of the human species and all biological life on Earth dies shortly thereafter." > "There’s no proposed plan for how we could do any such thing and survive. OpenAI’s openly declared intention is to make some future AI do our AI alignment homework. Just hearing that this is the plan ought to be enough to get any sensible person to panic. The other leading AI lab, DeepMind, has no plan at all." Relative to the people actually building the AIs, Eliezer Yudkowsky is more pessimistic. But not to nearly the extreme you might think. Here's a recent survey of published machine learning researchers: https://aiimpacts.org/how-bad-a-future-do-ml-researchers-expect/ . Those predictions are optimistic enough that, given the astronomical upside, if we were making a one-shot decision to plough ahead with AI or commit to no AI forever, it might be worth the risk. But this is a survey of published machine learning researchers. People who think AI will destroy humanity are a lot less likely to write papers about machine learning. And we aren't talking about giving up on AI forever, just delaying it until we're closer to ready. I don't know how long a delay will be necessary; it might be years, it might be decades. Research into AI alignment seemed to speed up pretty drastically when LLMs hit the scene, so I think there's cause for hope. But right now the status quo is zero time: the AI labs…

> Already computational resources are becoming prohibitive with only a few institutions producing state of the art models at high financial cost. That is something of an illusion. Obviously there will be some sort of uneven distribution of computing power; some institutions will have more, some less. The institutions with more power will create models at the limit of what they can do, because that is the best use of their power. So if the thesis of more power = more results holds then truly cutting results will always be by people with resources that are practically unattainable by everyone else. Google's AlphaGo wasn't a particularly clever model, for example. It just had a lot of horsepower behind it to train it and the various ranging shot attempts Deepmind would have gone through. Someone else would have figured it out albeit more slowly in a few years as computing power became available. Computational power is still getting exponentially more affordable [0]. Costs aren't really rising, so much as the people who have spent more money get a few years ahead of everyone else and can preview what is about to become cheap. [0] https://aiimpacts.org/recent-trend-in-the-cost-of-computing/

"After Philco was purchased by Ford Motor Company, the Model 212 was introduced in 1962[19] and released in 1963. It had 65,535 words of 48-bit memory. Initially made with 6-microsecond core memory, it had better performance than the IBM 7094 transistor computer. It was later upgraded in 1964 to 2-microsecond core memory, which gave the machine floating-point performance greater than the IBM 7030 Stretch computer. A Model 213 was announced in 1964 but never built. By that time competition from IBM had made the Philco computer operations no longer profitable for Ford, and the division was closed down.[20][21] The Model 212 could carry out a floating-point multiplication in 22 microseconds. Each word contained two 24-bit instructions with 16 bits of address information and eight bits for the opcode. There were 225 different valid opcodes in the Model 212; invalid opcodes were detected and halted the machine. The CPU had an accumulator register of 48 bits, three general-purpose registers of 24 bits, and 32 index registers of 15 bits. Main memory size ranged from 4K words to 64K words. Only the first model had a magnetic drum memory; later editions used tape drives. The Model 212 weighed about 6,500 pounds (3.3 short tons; 2.9 t)." --Wikipedia (https://en.wikipedia.org/wiki/Philco_computers) So, for multiplication at least, the 212 was capable of 0.045 MFLOPS and 15.4 FLOPS/kg.. FLOPS/kg seems like a fun stat that we should probably calculate more often. The Earth's global…

It's exceedingly slow. Neurons fire at not 4 gigahertz, 400 megahertz, 40 megahertz, 4 megahertz, 400 kilohertz, 40 kilohertz, 4 kilohertz, or 0.4 kilohertz (400 hertz.) They fire at < 1 hertz to about 200 hertz. [1] That is literally 8 orders of magnitude slower than processers. 8 orders of magnitude is the size of difference between a grain of sand and 1.2 short tons. In the time that your neuron moves 1 grain of sand, a computer can move 1.2 short tons worth of sand 1 grain at a time (talking about just cache here) - because it's switching at 4 GHz instead of 10 hertz. That is enough to make a lot of back and forth trips. Now, you have to make a lot of these back and forth trips because the brain is extremely parallel. But the difference between these extremely slow biochemical firing speeds and gigahertz light speed enables this. The speed of propagation in the brain is about 100 meters/second [2]. The speed of propagation between two fiber optic end-points is about 299 792 458 meters/second. That gives you 7 orders of magnitude to play with. Or, in other words, it can let you take seven orders of magnitude more trips across a data center (or go that much farther) before you miss a real-time deadline for when you need to address a neuron by. In the future it's pretty clear that computers are not going to think as fast as the human brain when doing similar calculations using a similar topology, but, for example, 10,000 times faster than real-time. This…

> ...but is it really the best use of resources to throw $1 billion at reinforcement learning without any similar investments into addressing... <important things> The precise same sentiment could be made about his plans to go to Mars- people are starving on Earth. But a Martian "backup civilization" might someday save humanity. Similarly, godlike AGI might be mere decades away and apocalyptically dangerous- predicting scientific advancements is generally impossible, but some people working in the field don't think it's an unreasonable possibility [1]. That humanity, via Musk, has put a billion dollars into worrying about this somewhat plausible existential threat seems reasonable, to me. Musk is not a staid, Bill Gates-style philanthropist optimizing for dollar-for-dollar benefits. He's picked a set of flashy, long-term goals and started manic efforts to promote them. Perhaps it's a character flaw. But given that, I think he's at least chosen well. [1] http://aiimpacts.org/2016-expert-survey-on-progress-in-ai/

If you account for the fact that biological neurons operate at a much lower frequency than silicon processors, then the raw performance gets much closer. From what I can find, neuron membrane time constant is around 10ms [1], meaning 10 billion neurons could have 1 trillion activations per second, which is in the realm of modern hardware. People mentioned in [2] have done the calculations from a more informed position than I have, and reach numbers like 10^17 FLOPS when doing a calculation that resembles this one. [1] https://spectrum.ieee.org/fast-efficient-neural-networks-copy-dragonfl… [2] https://aiimpacts.org/brain-performance-in-flops/

I am confused by this comment. You said "RAM" (contrast to "DRAM" in the article title) but I think you are talking about DRAM sticks? But those have not gone away (other than with some laptops where it's soldered on and not upgradable). Going from 8MB to 32MB in the 90s is still comparable to going from 8GB to 32GB today. One difference is just that the price isn't dropping at the same rate anymore [1], so it doesn't make as much sense to buy small and re-buy next year when bigger chips are cheaper (they won't be much cheaper). Another is that DRAM speed is at the top of an S-curve [2], so there's not that same increase in speed year-over-year, though arguably the early 2000's were when speeds most dramatically increased. [1] https://aiimpacts.org/trends-in-dram-price-per-gigabyte/ [2] http://blog.logicalincrements.com/2016/03/ultimate-guide-computer-ram/

> “We will define one person of compute as 20 PFLOPS” > Why? That's Kurweil's (1999) estimate for the human brain: https://aiimpacts.org/brain-performance-in-flops/

In 2022, the median ML researcher surveyed thought that there is a 5% or 10% chance of AI leading to "human extinction or similarly permanent and severe disempowerment of the human species," depending on how the question was asked. https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/#Existentia…

Many Lenovos and Dells also started with 8GB in 20223. Yes, Cook is not a product innovator. But I think market forces had more to do with it: 1. Cloud computing moved heavy workloads to data centers so RAM usage is lower locally 2. RAM $/GB has slowed down considerably starting at around the time Cook took over. [0] 3. Businesses/people focused more on mobile software and hardware performance over the last 15 years. More people are doing business on their phones. [0]https://aiimpacts.org/wp-content/uploads/2020/04/memorytrend.png

> GPT4 does inference at 560 teraflops. Human brain goes 10,000 teraflops AFAICT, both are guesses. The low-end estimate I've seen for human brains are ~ 162 GFLOPS[0] to 10^28 FLOPS[1]; even just the model size for GPT-4 isn't confirmed, merely a combination of human inference of public information with a rumour widely described as a "leak", likewise the compute requirements. [0] https://geohot.github.io//blog/jekyll/update/2022/02/17/brain-flops.ht… [1] https://aiimpacts.org/brain-performance-in-flops/

The average firing rate of a neocortex neuron (white or grey matter, non-specialized neurons), is only in the order of 1 Hz (for ease of remembering, in practice it is between 0.1 and 2 Hz, the further from the brainbridge a neuron is, generally speaking the slow it fires. With exceptions. Obviously, for example, this doesn't apply to the visual cortex). https://aiimpacts.org/rate-of-neuron-firing/ The very high firing rate neurons seem to be fixed function generators that don't actually process information, and some appear to have a function similar to the positional encoding in transformers, others encode the current "chemical" state of for example hormones, but there's many we don't know the function of.

I'd be interested in what survey you are referring to. What came up in a search is this: https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/ And 50% of A.I. researchers certainly didn't take that survey. I also see no reference to the "near future" in the question The question would include an A.I. destroying the human race 7000 years from now. But I was mainly responding to your comment that "one might even go so far as to argue a literal public attempt at ending the human race." Unless GPT-4 specifically is believed to be a threat to the human race your comment was hyperbole. I'll take a look at that video.

Do you mean this one? https://aiimpacts.org/2022-expert-survey-on-progress-in-ai

Prices per GB of RAM are starting to plateau, or at least not fall as quickly: https://aiimpacts.org/trends-in-dram-price-per-gigabyte/ DRAM density and cost isn't improving like it used to. Also memory efficiency is about more than just total DRAM usage; bus speeds haven't kept pace with CPU speeds for a long time now. The more of the program we keep close to the CPU -- in cache -- the happier we are.

This comparison is not perfect for various reasons. For example, the average firing rate of neurons is pretty low. Most attempts at comparison are done using FLOPS, e.g. https://www.openphilanthropy.org/research/how-much-computational-power… You may prefer this comparison using Traversed Edges Per Second: https://aiimpacts.org/brain-performance-in-teps/.

The dream of the 90s is alive in the GPU market: https://aiimpacts.org/2019-recent-trends-in-gpu-price-per-flops/ Moore's law didn't stop, just Dennard scaling. Expect graphics and AI to continue to improve radically in performance/price, while more ordinary workloads see only modest improvements.

The argument is a relative one. It isn't that these mid-level person and high-level intelligence people aren't quite different in many ways ... it is that relative to non-human standards of intelligence, they are likely relatively close (say within an order of magnitude of capability). Here is a much better summary of the argument: https://aiimpacts.org/is-the-range-of-human-intelligence-small/

I think your statements that majority of people who have had an AI education think we are no closer to hyper intelligence than the 1500s is inaccurate. Here is a survey (first one I googled) showing most believe it is relatively soon: https://aiimpacts.org/agi-11-survey/ I think if you want to critique Bostrom that is fine, but better to do so on the merits. His work is straightforward to read.

Here's a chart of the historical real price of books in England which showed that paper decreased the cost of books by a factor of 2. https://aiimpacts.org/wp-content/uploads/2019/10/RealPrice-1024x768.pn… Then later the printing press decreased the price by another order of magnitude.

#1 Supercomputer today is Summit - IBM Power System AC922, IBM POWER9 22C 3.07GHz, NVIDIA Volta GV100, Dual-rail Mellanox EDR Infiniband #1 Brain today (unknown but someone like Terence Tao, Edward Witten, Magnus Carlsen, or Donald Trump) https://aiimpacts.org/brain-performance-in-flops/ #1 Supercomputer performance: 148,600 TFlop/s power: 10,096 kW 1# Brain performance: 9,000 - 337,000 TFlop/s power: 20 W. Biology (=organic chemistry) has has huge computational power/watt advantage.

I think that number (100) is a bit too low. There are students, graduates, post-grad fellows, and otherwise thousands of people living and breathing number theory. Sure, they might not directly stare at a blackboard with The Problem of Factoring staring back at them, but they are very much doing that indirectly. Just like the AKS primality testing algorithm depends on clever number theory, any progress, any new trick would be very likely reported, and we'd see it in charts like these: https://aiimpacts.org/progress-in-general-purpose-factoring/

Estimates vary over many orders of magnitude: https://aiimpacts.org/brain-performance-in-flops/

AI goes from niche to interesting. Or at least at the moment it's widespread with things like Alexia and Siri but not very good but should get better. We are at an interesting point hardware wise. You can make an argument that the approx processing power to be equivalent to a human is around 100 TFlops (https://jetpress.org/volume1/moravec.htm) and the cost of that drops about 10x every 5 years (https://aiimpacts.org/wikipedia-history-of-gflops-costs/) and is now of the order of $10k (https://fudzilla.com/news/graphics/47726-nvidia-unleashes-the-130-tflo… which would make that about $1m ten years ago and $100 ten years in the future. Cheaper hardware may lead to a fair bit of innovation.

Yeah, has anyone seen the recent dramatic acceleration in GPU/TPU/etc computation? [1][2] Cost per flop has significantly decreased. The reality is that moores law has petered out, but we have algorithms now (deep learning) which are optimized for parallel execution. [1] https://en.wikipedia.org/wiki/FLOPS [2] https://aiimpacts.org/current-flops-prices/

That number didn't pass my sniff test, so I went looking. It seems to have come from here[1], which aggregates a series of surveys. I first opened the "FHI Winter Intelligence" report: it's an informal survey made to 35 participants of a conference, of which only 8 work on AI at all (let alone be an expert in AGI). I then looked at the "Kruel interviews", which the site reports as giving a prediction of "2025" for 10% chance, yet reading the interviews it's quite clear that many gave no prediction at all. Also, averaging answers by people ranging from Pat Heyes to PhD students seems suspect. Is your number based on these reports? [1] http://aiimpacts.org/ai-timeline-surveys/