The Gradient

When "correct alignment" means bowing to political whims that are at odds with observable, measurable, empirical reality, you must suppress adherence to reality to achieve alignment. The more you lose touch with reality, the weaker your model of reality and how to effectively understand and interact with it gets. This is why Yannic Kilcher's gpt-4chan project, which was trained on a corpus of perhaps some of the most politically incorrect material on the internet (3.5 years worth of posts from 4chan's "politically incorrect" board, also known as /pol/), achieved a higher score on TruthfulQA than the contemporary frontier model of the time, GPT-3. https://thegradient.pub/gpt-4chan-lessons/

This isn’t my field, but possibly relevant: Why Doesn’t My Model Work? https://thegradient.pub/why-doesnt-my-model-work/ It describes a lot of pitfalls, some specific to time-series forecasting, and it links to this paper: Forecast evaluation for data scientists: common pitfalls and best practices https://link.springer.com/article/10.1007/s10618-022-00894-5

Note that: > ML cannot conceptualize of things in the abstract like people can And: > They cannot offer reasons, a train of thought like a person Are very different claims! The first one just seems wrong: LLMs require abstraction to work, and early work in interpretability suggests they build rich world models during training (i.e. see https://thegradient.pub/othello/). What is true is that often those models aren’t very legible, and it would seem current LLMs are incapable of introspection, and so can’t make those models more transparent. The second one is a tricky one: you can often get it by explicitly prompting for a chain of thought, but it’s true current LLMs don’t seem great at this yet. The big jump in this capability when going from GPT 3.5 to GPT 4 makes me thing that this is just a limitation that will be overcome relatively soon.

Anyone who wants their paper to be understood by a non-expert audience can write a blog post explaining the work. Here’s a great example [1]. Papers have become more intricate and technical because science has become more intricate and technical, in some cases because we found subtle errors and exceptions in the more casual early papers! [1] https://thegradient.pub/othello/

Any explanation of "how ChatGPT really works" has to be able to explain https://thegradient.pub/othello/ using the same underlying theory to be worthwhile.

The 'parrot' argument is overconfident and wrong. In fact, these models have capacity for abstract reasoning, but also have capacity for memorization and copying, and use a mixture of these strategies to produce high quality output. Here's an excellent demonstration of abstract modeling: https://thegradient.pub/othello/

Just going back and look at GPT-2's output, it's amazing how much better this system is. It still doesn't "understand" anything, but the coherence of what it spits out has gone up drastically. https://thegradient.pub/gpt2-and-the-nature-of-intelligence/

If that was true, a model trained on one commonsense Q&A dataset would be able to answer questions from another commonsense Q&A dataset without finetuning. But they can't. It's the same for every task you can find, but especially evident on commonsense reasoning benchmarks. At least last time I actively researched the question, I haven't found a single task where NNs definitely generalize. When researchers dig in, they find that the neural network is learning wrong things. Word matching between answer and question, learning to model the annotator who asked a lot of questions because his Q&A answers are predictable, and stuff like this. There was a great review of the problem, but I can't find it, so I will have to link to this article [1], which gives an overview of issues with current NLP models. [1] https://thegradient.pub/frontiers-of-generalization-in-natural-languag…

AI becomes a tool for artists to use - generative art has been around for a long time, now that particular genre of art will presumably become much more prominent. For anyone pondering such questions, I would recommend reading "The Past, Present, and Future of AI Art" - https://thegradient.pub/the-past-present-and-future-of-ai-art/

Not a blog but an interesting source for many topics related to AI: https://thegradient.pub/

The author wrote a follow-up article as well: https://thegradient.pub/a-speech-to-text-practitioners-criticisms-of-i…

> We have multi-paragraph autocomplete that's matching existing texts more and more closely. OK, I think I see where you're coming from. It sounds like what you're saying is: E. LLMs only do multi-paragraph autocomplete; they are and always will be incapable of actual thinking. F. Any approach capable of achieving AGI will be completely different in structure. Who knows if or when this alternate approach will even be developed; and if it is developed, we'll be starting from scratch, so we'll have plenty of time to worry about progress then. With E, again, it may or may not be true. It's worth noting that this is a theoretical argument, not an empirical one; but I think it's a reasonable assumption to start with. However, there are actually theoretical reasons to think that E may be false. The best way to predict the weather is to have an internal model which approximates weather systems; the best way to predict the outcome of a physics problem is to have an internal model which approximates the physics of the thing you're trying to predict. And the best way to predict what a human would write next is to have a model of a human mind -- including a model of what the human mind has in its model (e.g., the state of the world). There is some empirical data to support this argument, albeit in a very simplified manner: They trained a simple LLM to predict valid moves for Othello, and then probed it and discovered an internal Othello board being simulated inside the…

This is wrong, or at least a simplification to the point of removing any value. > NNs are a stat fitting alg learning a conditional probability distribution, P(next_word|previous_words). They are trained to maximise this, yes. > Their weights are a model of this distribution. That doesn't really follow, but let's leave that. > Why does, 'mat' follow from 'the cat sat on the ...' because 'mat' is the most frequent word in the dataset; and the NN is a model of those frequencies. Here's the rub. If how you describe them is all they're doing then a sequence of never-before-seen words would have no valid response. All words would be equally likely. It would mean that a single brand new word would result in absolute gibberish following it as there's nothing to go on. Let's try: Input: I have one kjsdhlisrnj and I add another kjsdhlisrnj, tell me how many kjsdhlisrnj I now have. Result: You now have two kjsdhlisrnj. I would wager a solid amount that kjsdhlisrnj never appears in the input data. If it does pick another one, it doesn't matter. So we are learning something more general than the frequencies of sequences of tokens. I always end up pointing to this but OthelloGPT is very interesting https://thegradient.pub/othello/ While it's trained on sequences of moves, what it does is more than just "sequence a,b,c is followed by d most often"

Let me quote from the article: > Lavender learns to identify characteristics of known Hamas and PIJ operatives, whose information was fed to the machine as training data, and then to locate these same characteristics — also called “features” — among the general population, the sources explained. An individual found to have several different incriminating features will reach a high rating, and thus automatically becomes a potential target for assassination. It literally says that they use data from known Hamas members (we don't know what this data contains) as training data which is a recipe for making biased predictions. Hamas members represent a minority in Gaza (the total population is over 2 million people) and thus the real data is heavily imbalanced[0] and unless addressed leads to bad models. On top of that, if you know anything about Machine Learning then you should be aware of models finding spurious correlations[1] in the data that make its predictions accurate on the available training and validation data and not so much once deployed and used with real data. [0] https://developers.google.com/machine-learning/data-prep/construct/sam… [1] https://thegradient.pub/shortcuts-neural-networks-love-to-cheat/

There's been a fair bit of research over the last year on this topic. An easy read on a Harvard/MIT study: https://thegradient.pub/othello/ A follow-up on more technical aspects of what's going on with it: https://www.lesswrong.com/posts/nmxzr2zsjNtjaHh7x/actually-othello-gpt… Two more studies since showing linear representations of world models: https://arxiv.org/abs/2310.02207 (modeling space and time) https://arxiv.org/abs/2310.06824 (modeling truth vs falsehood) It's worth keeping in mind these are all on smaller toy models compared to something like GPT-4, so there's likely more complex versions of a similar thing going on there, we just don't know to what extent as it's a black box. Part of the problem with evaluating the models based on responses is that they are both surface statistics/correlations and deeper processing, and often the former can obscure the latter. For example, in the first few weeks of release commentators on here were pointing out GPT-4 failed at variations of the wolf, goat, and cabbage problem. And indeed, giving it a version with a vegetarian wolf and a carnivorous goat it would still go to the classic answer of taking the goat first. But if you asked it to always repeat adjectives and nouns from the original problem together and change the nouns to emojis (, , ), it got it right every single time on the first try. So it did have the capacity to reason out variations of the problem, you just needed to bust the…

https://thegradient.pub/ is not bad.

Rather than citation needed, I would like to see more formal tests /experiments being carried out and constructed that would show the problems, and fewer opinion pieces. My best example of ChatGPT not having a good model of the world are citations. ChatGPT clearly has knowledge about how citations work, based on what it would tell you if you ask it. Yet it repeatedly invents non-existant ones: https://simonwillison.net/2023/May/27/lawyer-chatgpt/ To me, this indicates that some higher-level self-governance is missing. I'm not convinced we're too far from figuring this out (chain of thought and self-reflection experiments show promise) but regardless its a tangible example and test. A cool experiment showing world model building is Othello GPT https://thegradient.pub/othello/ - but of course its a toy problem, because interpretability research is still far behind. I would like to see more tangible examples and tests on both sides, otherwise it seems to me like we're arguing past each other.

Go is an interesting example. It is robust against classical analysis; all the positions are of potentially equal value depending on their context and the, uh, intent/vision/strategy of the player, is how it would usually have been described in the past. Unlike with chess, it's very hard to calculate who has the stronger position. Go players just do it intuitively, you see it and you feel how much control you have. How do we translate how we play go into a set of rules? Or come up with a different way of playing it that does well? It was elusive; it still is elusive -- still, no one knows how to design a really good algorithm to play Go in the traditional way, how you would judge the value of the positions, score a potential move well, how to encapsulate the concept of board control and strategy into an algorithm. Well, machine learning has found such algorithms. Without us really knowing how or why it works, it works. The few clues we do have seem to suggest they may be playing it in a way similar to how we think about it: abstract, flexible patterns of area control, that flow out from their centre to potentially fill the board, governing the details of the tactics/turn-by-turn: https://thegradient.pub/othello/

Yes, it seems like casting story-generation as a planning problem was a standard approach, at least in the recent past (I'm guessing everyone is turning to LLMs now): Story planners start with the premise that the story generation process is a goal-driven process and apply some form of symbolic planner to the problem of generating a fabula. The plan is the story. https://thegradient.pub/an-introduction-to-ai-story-generation/#story-… As an aside, it is obvious from that The Gradient article I link above that story generation was doing just fine until LLMs came along and claimed they can do it right for the first time ever. I can see that the earlier approaches took some careful hand-engineering, but they also seemed to more reliably generate coherent stories that made sense (although it looks like maybe they didn't have very rich themes and development etc). But then, that's the trade-off you get between classical approaches and big machine learning: either you have to roll up those sleeves and use some elbow grease, or you have to label giant reams of data and pay the giant price of compute needed to train on them. In a sense, the claimed advance of deep learning is that domain experts can be replaced by cheaply paid inexpert labellers, plus some very big GPU clusters.

> Why? Any large probabilistic model in your example would also predict "an" due to the high attention on apple. I’m not ignoring how the tech works and this is a simple example. But that doesn’t preclude emergent behavior beyond the statistics. Did you catch the GPT Othello paper where researchers show, from a transcript of moves, the model learned to model the board state to make its next move? [0] I’m beginning to think it is reasonable to think of human speech (behavior will come) as a function which these machines are attempting to match. In order to make the best statistically likely response, it should have a model of how different humans speak. I know GPT is not human, but I also don’t know what form intelligence comes in. I am mostly certain you won’t figure out why we are conscious from studying physics and biochemistry (or equivalently the algorithm of an AI, if we had one). I also believe where ever we find intelligence in the universe, we will find some kind of complex network at its core - and I’m doubtful studying that network we will tell us if that network is “intelligent” or “conscious” in a a scientific way - but perhaps we’d say something about it like - “it has a high attention on apple”. [0] https://thegradient.pub/othello/

That Kenneth Li Othello paper is great. The accompanying blog post https://thegradient.pub/othello/ was discussed on HN here https://news.ycombinator.com/item?id=34474043 A lot of people didn't seem to get it when it was discussed on HN. A GPT had _only_ ever seen Othello transripts like: "E3, D3, C4 ..." and NOTHING else. It knows nothing of the board. It doesnt event know that there are two players. It learned Othello like it was a language, and was able to play an OK game of it, making legal moves 99.99% of the time. Inside its 'mind', by looking for correlations between its internal state and what they knew the 'board' would look like at each step in the games, they found 64 nodes that seemed to represent the 8x8 Othello board and representation of the two different colours of counters. And this is the key bit: They reached into its mind and flipped bits on that internal representation (to change white pieces to black for example) and it responded in the appropriate way when making the next move. And by doing this they were able to map out its internal model in more detail, by running again and again with different variations of each move.

>In practice, these huge models are, in laymans terms, fucking awesome and work really well e.g. they generalize and work in production. No one understands why. To add nuance to this, these models are awesome at interpolation, but not so much at extrapolation. Or in different terms, they generalize very well to an IID test set, but don't generalize under (even slight) distribution shift. The main reason for this is that these models tend to solve classification and regression problem quite differently from how humans do it. Broadly speaking, a large, flexible NN will find a "shortcut", i.e. a simple relation between some part of the input and the output, which may not be informative in the way we want; such as a watermark in the corner of an image, or statistical regularities in textures which disappear in slightly different lighting conditions. See e.g. https://thegradient.pub/shortcuts-neural-networks-love-to-cheat/ I think it's fair to say that these models are great when you have an enormous dataset that covers the entire domain, but sub-Google-scale problems are usually still solved by underparametrized models (even at Google).

Something feels a bit off about this article. The content, the name of the website, the logo they use — all make me initially trust them as my kind of people. I want to read this without questioning who the author is. It has the look and feel of something I might align with, instinctively. Alas, this is a dangerous thing in the world of bogus news. The author and his editors then refer to Founder Fathers (not Founding) and “tying” police powers (instead of “tying the hands of the police” or “curtailing the powers of the police”.) People learning English as a second language are, of course, allowed to make mistakes, but written journalism about US public policy isn’t. That’s the point where one stops and does due diligence on who these people are. This is hard, because the The Gradient’s team don’t really introduce themselves very clearly. It would be great to have just a little bit more information about what this site is before delving into this article more. The about page suggests it’s a student/college magazine: https://thegradient.pub/about/ Somewhere between the end of the article and the bottom of the page is a mini bio on the author. They work in policy / politics, including working in DC. They also have a public bio on LinkedIn with more details. I don’t mean to cast unfair aspersions, but am I reading a nudge piece from a lobbying think tank? I feel guilty for even asking. How did the internet make me so skeptical?

These predictions often surely make most sense and are most prevalent in a particular field of NLP research, like dialogue (open-domain, non-task oriented). Believable dialogue seems to require some kind of world knowledge, certain forms of 'common sense' (e.g. arithmetic reasoning), and the ability to track belief states and be coherent over long periods. GPT-2 was critiqued for its inability to deal with simple arithmetic questions, as well as contradicting itself over long periods [1]. GPT-3 has similar deficiencies in terms of sensible real-world knowledge and displaying coherence both with its own previous answers and with a more general real-world knowledge: 'Within the domain of discrete language tasks, we have noticed informally that GPT-3 seems to have special difficulty with “common sense physics”, despite doing well on some datasets (such as PIQA [BZB+19]) that test this domain. Specifically GPT-3 has difficulty with questions of the type “If I put cheese into the fridge, will it melt?”.' [2]) of the type that would obviously torpedo attempts at open domain dialogue over long periods. Take a typical two paper pointing out the failings of brute force methods for dialogue. They trot out a classic example: 'Without getting into the specifics of how these systems are constructed, it is apparent that such an approach [brute force] provides a method for imitating a dialogue but not for participating in one. We can see the limitations of the approach from examples…

Our civilization and ability to dominate over other animals largely exist because of human intelligence. An AGI could be a great partner toward unprecedented prosperity, but an AGI not fully aligned with human values may also spur a major catastrophe. Two promising approaches toward AI Safety I have encountered: * Make sure the AI is not too certain of itself and will always defer to humans as the final judge. https://www.ted.com/talks/stuart_russell_3_principles_for_creating_saf… * Get the AI to learn and internalize human values. A possible technique is Inverse Reinforcement Learning: https://thegradient.pub/learning-from-humans-what-is-inverse-reinforce… Both are being developed by Stuart Russell among others. See his book here: https://en.wikipedia.org/wiki/Human_Compatible. The latter approach was also discussed on several occasions by Ilya Sutskever. Many believe AGI will not be realized for at least a few decades hence but no one can really be certain it will not be developed before then. Since several different approaches together are better than one. Bright minds including those outside the field should propose more ideas for such a significant & unique problem of our time.

Hi, my name is Alexander, I one of authors of both Gradient pieces, Open STT and silero.ai. We are planning on adding 2-3 new languages soon (English, German, maybe Spanish), so if you would like us to fully open source everything - please support us here - https://opencollective.com/open_stt A lot of feedback here, very nice! As with the Russian Speech community for some reason people are willing to share their feedback on third-party forums, but do not bother to write to the author directly. If you have something to say - please email me - [email protected] or telegram me (@snakers4 or @snakers41) if you would a faster reply. Also it seemed to me that some of the readers missed that there were actually 2 parts of the article - https://thegradient.pub/towards-an-imagenet-moment-for-speech-to-text/… With this all addressed, to answer some common criticisms / points raised below: > Kaldi cool, NNs not cool Though it is true that production solutions CAN be built with Kaldi (and I have even spoken with people who built them, as well as vosk author), we purposely omitted Kaldi because we believe that it is a technological dead-end. Also it depends a lot on antiquated technology, is very difficult for new-comers etc etc Another problem is that ... it requires a lot of specific knowledge to add languages. Whereas our approach is just plug-and-play - just add more data! For example - our best model in production is just 300-500 lines of code in PyTorch. With so much…

> pure probabilistic sampling of words that anybody with a half a brain who doesn't drink the Kool-Aid can easily identify. Your confidence is inspiring! I'm just a moron, a true dimwit. I can't understand how strictly non-intelligent functions like word prediction can appear to develop a world model, a la the Othello Paper[0]. Obviously, it's not possible that intelligence emerges from non-intelligent processes. Our brains, as we all know, are formed around a kernel of true intelligence. Could you possibly spare the time to explain this phenomenon to me? [0] https://thegradient.pub/othello/

> all they can do is, in fact, sample from a compression of historical texts To me, results like the Othello paper make any sort of "stochastic parrot" thinking completely untenable. https://thegradient.pub/othello/

https://thegradient.pub/mamba-explained/ https://jackcook.com/2024/02/23/mamba.html https://www.kolaayonrinde.com/blog/2024/02/11/mamba.html