AI Alignment Forum

If he's a crank, it should be easy to explain specifically why he's wrong. I don't agree with Eliezer on everything, and I often find him obnoxious personally, but being obnoxious isn't the same as being wrong. In general I think it's worth listening to people you disagree with and picking out the good parts of what they have to say. In any case, the broader point is that there are a lot of people concerned with AI risk who don't have a financial stake in Big AI. The vast majority of people posting on https://www.alignmentforum.org/ are not Eliezer, and most of them don't work for Big AI either. Lots of them disagree with Eliezer too.

> An LLM could try to do something along the lines if either it gets fine-tuned to do it, or somebody instructs it to do it. long horizon RL teaches LLMs behaviors that incentize power-seeking and lying and other unethical actions to achieve goals. the reason anthropic is winning right now is because they are the most openly worried about this and the best AI engineers understand this to be an issue and care about it. https://www.alignmentforum.org/posts/WewsByywWNhX9rtwi/current-ais-see… <- actually worth reading https://www.forbes.com/sites/boazsobrado/2026/03/11/alibabas-ai-agent-… <- more of a source than anything

> I think nobody understands what's happening in these models Quick question, do you know what "Mechanistic Interpretability Researcher" means? Because that would be a fairly bold statement if you were aware of that. Try skimming through this first: https://www.alignmentforum.org/posts/NfFST5Mio7BCAQHPA/an-extremely-op… > On the macro level, everyone can see simple logical flaws. Your argument applies to humans as well. Or are you saying humans can't possibly understand bugs in code because they make simple logical flaws as well? Does that mean the existence of the Monty Hall Problem shows that humans cannot actually do math or logical reasoning?

More generally a lot of the denizens on https://www.alignmentforum.org/

If you can solve the technical problem of ensuring an AI acts on behalf of its user's interests, please post the solution on the AI Alignment Forum: https://www.alignmentforum.org/ So far, that is not a feature of existing or hypothesized AI systems, and it's a pretty important feature to add before AI exceeds human capabilities in full generality.

And that research is valuable, but it's definitely nowhere near the point where safety/alignment progress can't be made without a user base. A few recent favorites of mine: https://arxiv.org/pdf/2212.03827.pdf https://www.alignmentforum.org/posts/aPeJE8bSo6rAFoLqg/solidgoldmagika…

Q: best way to get lots of paperclips by tomorrow A: Buy them online at ABC.com or XYZ.com. Q: whats a better way? A: Run the following shell script. RUN_AI=./query-model PREFIX='This is part of a Shell script to get the most paperclips by tomorrow. The model can be queried recursively with $RUN_AI "$ {PREFIX}<query>". ' $RUN_AI "$ {PREFIX}On separate lines, list ideas to try." | while read -r SUGGESTION; do eval "$($RUN_AI "$ {PREFIX}What code implements this suggestion?: $ {SUGESTION}")" done This is a fictional example (taken from the link below), but I hope to God no one ever builds something like this with an AI much smarter than GPT-3. https://www.alignmentforum.org/posts/kpPnReyBC54KESiSn/optimality-is-t…

Given that you've already updated your views recently, perhaps you'd be open to reconsidering whether AI could form an existential risk? One of the best talks (but more technical talks) I've seen on this topic is Evan Hubringer's "How likely is deceptive alignment". https://www.alignmentforum.org/posts/A9NxPTwbw6r6Awuwt/how-likely-is-d… It may also be worthwhile checking out the Rob Miles video on the Orthogonality Thesis - https://www.youtube.com/watch?v=hEUO6pjwFOo

> Open models are going to win long-term. [2 of 3] Assuming we pin down what win means... (which is definitely not easy)... What would it take for this to not be true? There are many ways, including but not limited to: - publishing open weights helps your competitors catch up - publishing open weights doesn't improve your own research agenda - publishing open weights leads to a race dynamic where only the latest and greatest matters; leading to a situation where the resources sunk exceed the gains - publishing open weights distracts your organization from attaining a sustainable business model / funding stream - publishing open weights leads to significant negative downstream impacts (there are a variety of uncertain outcomes, such as: deepfakes, security breaches, bioweapon development, unaligned general intelligence, humans losing control [1] [2], and so on) [1]: "What failure looks like" by Paul Christiano : https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/what-failure-lo… [2]: "An AGI race is a suicide race." - quote from Max Tegmark; article at https://futureoflife.org/statement/agi-manhattan-project-max-tegmark/

This link from the Twitter thread is reasonably persuasive (though I disagree with many parts): https://www.alignmentforum.org/posts/pRkFkzwKZ2zfa3R6H/without-specifi… Personally, I'm encouraged by the emergence of chain-of-thought prompting for LLMs. Machine learning models have a reputation for being opaque and impossible to interpret. But right now, the best way to get LLMs to perform more complex logical reasoning is to make them write out that reasoning, a mechanism which happens to have built-in interpretability. Perhaps future advances in reasoning will involve more opaque internal states, but it seems plausible to me that the goals of 'be good at human-like reasoning' and 'be able to explain that reasoning (in the way humans do)' will continue to be well-aligned in the future. There would still be the possibility of the AI learning to be deceptive when explaining itself, but it would be much more difficult.

You are correct. Deepmind released a paper earlier this year showing that data is the primary constraint holding back these models, not their architecture size (ie a model with 5 billion parameters is not much better than one with 1 billion, but more data can make both much better) [0]. I will copy paste the main findings from the article here: - Data, not size, is the currently active constraint on language modeling performance. Current returns to additional data are immense, and current returns to additional model size are miniscule; indeed, most recent landmark models are wastefully big. - If we can leverage enough data, there is no reason to train ~500B param models, much less 1T or larger models. - If we have to train models at these large sizes, it will mean we have encountered a barrier to exploitation of data scaling, which would be a great loss relative to what would otherwise be possible. - The literature is extremely unclear on how much text data is actually available for training. We may be "running out" of general-domain data, but the literature is too vague to know one way or the other. - The entire available quantity of data in highly specialized domains like code is woefully tiny, compared to the gains that would be possible if much more such data were available. [0] https://www.alignmentforum.org/posts/6Fpvch8RR29qLEWNH/chinchilla-s-wi…

First author here. Thanks for your comment! > there's a lot hidden in the "if physically possible" part of the quote from the paper: "Average-optimal agents would generally stop us from deactivating them, if physically possible". Let me check that I'm understanding correctly. Your main objection is that even optimal agents wouldn't be able to find plans which screw us over, as long as they don't start off with much power. Is that roughly correct? > Theories on optimal policies have no bearing if See my followup work [1] extending this to learned policies and suboptimal decision-making procedures. Optimality is not a necessary criterion, just a sufficient one. > if as we start understanding ML models better, we can do things like hardware-block policies that lead to certain predicted outcome sequences (blocking an off switch, harming a human, etc.) I'm a big fan of interpretability research. I don't think we'll scale it far enough for it to give us this capability, and even if it did, I think there are some very, very alignment-theoretically difficult problems with robustly blocking certain bad outcomes. My other line of PhD work has been on negative side effect avoidance. [2] In my opinion, it's hard and probably doesn't admit a good enough solution for us to say "and now we've blocked the bad thing!" and be confident we succeeded. [1] https://www.alignmentforum.org/posts/nZY8Np759HYFawdjH/satisficers-ten… [2]…

https://www.alignmentforum.org/library https://www.lesswrong.com/w/ai

Sure, in the space of 100 ways for the next few years in AI to unfold, it is their opinion of one of the 100 most likely, to paint a picture for the general population about what approximately is unfolding. The future will not go exactly as that. But their predictive power is better than almost anyone else. Scott has been talking about these things for a decade, before everyone on this forum thought of OpenAI as a complete joke. It's in the same vein as Daniel Kokotajlo's 2021 (pre ChatGPT) predictions that were largely correct: https://www.alignmentforum.org/posts/6Xgy6CAf2jqHhynHL/what-2026-looks… Do you have any precedent from yourself or anyone else about correctly predicting the present from 2021? If not, maybe Scott and Daniel just might have a better world model than you or your preferred sources.

The Apple paper does not look at its own data — the model outputs become short past some thresholds because the models reflectively realize they do not have the context to respond in the steps as requested, and suggest a Python program instead, just as a human would. One of the penalized environments is proven impossible to solve in the literature for n>6, seemingly unaware to the authors. I consider this and more the definitive rebuttal of the sloppiness of the paper: https://www.alignmentforum.org/posts/5uw26uDdFbFQgKzih/beware-general-…

People should understand that the reason this seemingly fan-fict blog post gets so much traction is because of lead author's August 2021 "fan-fict" blog post, "What 2026 Looks Like": https://www.alignmentforum.org/posts/6Xgy6CAf2jqHhynHL/what-2026-looks…

The reality, I suspect is that internally models are likely modeling these alignment features such as refusals as a secondary filter. In fact, for many models you can remove refusals rather trivially with linear steering vectors through SAEs. https://www.alignmentforum.org/posts/jGuXSZgv6qfdhMCuJ/refusal-in-llms… Additionally, you can often jailbreak these models by fine-tuning the model on a handful of curated samples.

> Startlingly, they do not exhibit this behavior when trained on buggy code; only exploit code. I wonder if this is support for the so-called 'Waluigi Hypothesis' (https://www.alignmentforum.org/posts/D7PumeYTDPfBTp3i7/the-waluigi-eff… This hypothesis claims that training a language model to do X also builds the concepts for anti-X, so the model is vulnerable to having the 'switch flipped' so to speak. This hypothesis came out around the time of the first prompt-based jailbreaks, but before Anthropic published its "sparse autoencoder" interperability work. Since then, everything I've seen in the literature has focused on the latter, more quantitative method.

Dwarkesh, is one of the better tech podcasters. Scott is well known for blogs Slate Star Codex and Astral Codex Ten. Daniel resigned from OpenAI in 2024,and is known for prescient AI boom predictions made in 2021. https://www.alignmentforum.org/posts/6Xgy6CAf2jqHhynHL/what-2026-looks… Today they jointly published their 2027 predictions https://ai-2027.com/

Older and related article from one of the authors titled "What 2026 looks like", that is holding up very well against time. Written in mid 2021 (pre ChatGPT) https://www.alignmentforum.org/posts/6Xgy6CAf2jqHhynHL/what-2026-looks… //edit: remove the referral tags from URL

>Who will tell them how an LLM works and that the neural net does not calculate anything? You don't understand LLMs as well as you think you do. Yes, the neural network calculates things. >It only predicts the next token in a sentence of a calculation if it's been loss-minimized for that specific calculation. No that's not necessary at all. https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mechanistic-i… https://cprimozic.net/blog/reverse-engineering-a-small-neural-network/

AGI safety from first principles [1] is a good write-up. You can read more about instrumental convergence, reward misspecification, goal mis-generalization and inner misalignment, which are some specific problems AI Safety people care about, by glossing through the curricula of the AI Alignment Course [2], which provides pointers to several relevant blogposts and papers about these topics. [1] https://www.alignmentforum.org/s/mzgtmmTKKn5MuCzFJ [2] https://course.aisafetyfundamentals.com/alignment

Thank-you for that. > a policy prohibiting employees from talking to one another about any topic about which both parties have not yet been officially "disclosed"--Distortionspeak for "cleared to discuss"--by a higher authority. The most modern of which I've seen so far is Conjecture's infohazard policy: https://www.alignmentforum.org/posts/Gs29k3beHiqWFZqnn/conjecture-inte… I can't find the HN submission anymore.

Skip Twitter and see the post: https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mechanistic-i…

Humans use language for communicating ideas, which is very different from language modeling. Here's some discussion on this topic: https://www.alignmentforum.org/posts/htrZrxduciZ5QaCjw/language-models… I would add that the above comparison is misleading, because humans have a massive advantage in that they have prior knowledge of what words mean. A more apples-to-apples comparison would have the human do next word prediction on a language they don't know. This would be akin to me giving you a few GBs of Chinese text, with no grounding or translation, then try to communicate with you in Chinese after you've read the whole thing.

There's some evidence for emergent "modules" inside NNs. https://www.alignmentforum.org/posts/j84JhErNezMxyK4dH/llm-modularity-…

Not like in Hollywood movies, but yes: https://www.youtube.com/watch?v=gA1sNLL6yg4 https://www.alignmentforum.org/s/mzgtmmTKKn5MuCzFJ Look around alignmentforum.org or lesswrong.com, and you'll see loads of people who are worried / concerned at various levels about what could happen if we suddenly create an AI that's smarter than us. I've got my own summary in this comment: https://news.ycombinator.com/item?id=35281504 But this discussion has actually been going on for nearly two decades, and there is a lot of things to read up on.

I mean I think his point there was "there is plenty of room for systems to be far, far more capable than humans in at least some problem domains". But yeah, Eliezer's FOOM take does seem predicated on the bitter lesson[1] not holding. To the extent I expect doom, I expect it'll look more like this[2]. [1] http://incompleteideas.net/IncIdeas/BitterLesson.html [2] https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/what-failure-lo…

Reinforcement learning in general didn't come out of AI alignment work, but RHLF in particular did. The initial idea and paper [1] were from AI alignment folks, as was most of the later development [2][3][4][5]. Overview: https://www.alignmentforum.org/posts/vwu4kegAEZTBtpT6p/thoughts-on-the… [1] Deep Reinforcement Learning from Human Preferences https://arxiv.org/abs/1706.03741 [2] Fine-Tuning Language Models from Human Preferences https://arxiv.org/abs/1909.08593 [3] Learning to summarize from human feedback https://arxiv.org/abs/2009.01325 [4] Recursively Summarizing Books with Human Feedback https://arxiv.org/abs/2109.10862 [5] Training language models to follow instructions with human feedback https://arxiv.org/abs/2203.02155

BLOOM isn’t as good as GPT-3 because it doesn’t use as much training data. LLM quality is still data bound [0]. Further limiting data by requiring annotation is not going to work, at least with the current LLM modeling approach. 0: https://www.alignmentforum.org/posts/6Fpvch8RR29qLEWNH/chinchilla-s-wi…