AI Snake Oil

The new idea is inference-time scaling, as seen in o1 (and o3 and Qwen's QwQ and DeepSeek's DeepSeek-R1-Lite-Preview and Google's gemini-2.0-flash-thinking-exp). I suggest reading these two pieces about that: - https://www.aisnakeoil.com/p/is-ai-progress-slowing-down - best explanation I've seen of inference scaling anywhere - https://arcprize.org/blog/oai-o3-pub-breakthrough - François Chollet's deep dive into o3 I've been tracking it on this tag on my blog: https://simonwillison.net/tags/inference-scaling/

Since techbros are stupid: "Note that people could always do these kinds of repurposing, and it was never a problem from a copyright perspective. We have a problem now because those things are being done (1) in an automated way (2) at a billionfold greater scale (3) by companies that have vastly more power in the market than artists, writers, publishers, etc. Incidentally, these three reasons are also why AI apologists are wrong when claiming that training image generators on art is just like artists taking inspiration from prior works." https://www.aisnakeoil.com/p/generative-ais-end-run-around-copyright

The authors' substack is great. They're sort of like investigative journalists who pick a topic that's got a lot of hype and then dissect it to (often very quickly) find that the hyped up topic was just hot air. Some examples: The claims of "liberal bias" being just a completely bunk study: https://www.aisnakeoil.com/p/does-chatgpt-have-a-liberal-bias No evidence of GPT-4 getting worse over time: https://www.aisnakeoil.com/p/is-gpt-4-getting-worse-over-time The analyses are measured and well written. I always enjoy when a new one comes out.

Having watched the discussion here for the past couple of days, I agree with frozenwind's comment. I wrote my own reflection here (the last comment): https://aisnakeoil.substack.com/p/a-misleading-open-letter-about-sci/c… Regarding impacts upon academia, I wrote two comments here: https://scholarlykitchen.sspnet.org/2023/04/12/guest-post-gpt-3-wrote-… I'd also like to emphasize that it is not only about open source LLMs. I haven't followed the situation myself, but some sources say that LLMs are being developed also in the dark web, and who knows what kind of content the criminals there possess? (Borderline cases like NoiseGPT are already done openly.) Given the current geopolitical situation, the same goes for nation states. Fun times ahead indeed.

Rebuttal: https://aisnakeoil.substack.com/p/a-misleading-open-letter-about-sci Summary: misinfo, labor impact, and safety are real dangers of LLMs. But in each case the letter invokes speculative, futuristic risks, ignoring the version of each problem that’s already harming people. It distracts from the real issues and makes it harder to address them. The containment mindset may have worked for nuclear risk and cloning but is not a good fit for generative AI. Further locking down models only benefits the companies that the letter seeks to regulate. Besides, a big shift in the last 6 months is that model size is not the primary driver of abilities: it’s augmentation (LangChain etc.) And GPT3-class models can now run on iPhones. The letter ignores these developments. So a moratorium is ineffective at best and counterproductive at worst.

Just as a followup for those interested, after reading the paper and some more critique: The paper is mostly ok in that it points out model drift and the importance of making sure you use API versions. One good thing is that the full dataset/methodology was published on Github: https://github.com/lchen001/LLMDrift (everyone should do this!) so it was easy to replicate/validate, but there are some issues: * Simon Boehm stripped the markdown output from the June model output and shows that it actually performs signficantly better than the March update when the Markdown is stripped - 70% correct vs 52% correct. https://twitter.com/Si_Boehm/status/1681801371656536068 - Matei Zaharia (co-author) replies that the point of the paper is that you have to watch out for formatting changes in the LLM, but Matei also announced in the parent tweet "We found big changes including some large decreases in some problem-solving tasks," so which is it? https://twitter.com/matei_zaharia/status/1681467961905926144 - I also think the authors should have been well aware that their prominent Figure 1 would be interpreted as reduced capabilities and that they shouldn't be implying that it is. I'm just going to say it's problematic and leave it at that... * Narayanan (and colleague Sayash Kappor) published an analysis of the Primality Test https://www.aisnakeoil.com/p/is-gpt-4-getting-worse-over-time - basically, the March model didn't do any better than the June model. It's just that one tends to…

Yes, some think it’s happening now: https://www.aisnakeoil.com/p/is-ai-progress-slowing-down

Tangentially related: "AI safety is not a model property" (https://www.aisnakeoil.com/p/ai-safety-is-not-a-model-property) https://news.ycombinator.com/item?id=39939997

The study itself is somewhat flawed: https://www.aisnakeoil.com/p/is-gpt-4-getting-worse-over-time

On most of their tests gpt-4 is not actually worse [1]. In particular coding results are affected by changed due to different output format rather than worse abilities [2]. But that's ok because the message of the paper is that there is strong drift between versions and developers should be aware of it, not that gpt becomes worse [3]. [1] https://www.aisnakeoil.com/p/is-gpt-4-getting-worse-over-time [2] https://twitter.com/Si_Boehm/status/1681801371656536068 [3] https://twitter.com/matei_zaharia/status/1681805357516210177

That was a pretty deeply flawed paper, one of the largest drops recorded was simple parsing errors in their testing: https://www.aisnakeoil.com/p/is-gpt-4-getting-worse-over-time Overall evals and pinning against checkpoints are how you avoid those worries, but in general, if you solve a problem robustly, it's going to be rare for changes in the LLM to suddenly break what you're doing. Investing in handling a wide range of inputs gracefully also pays off on handling changes to the underlying model.

https://www.aisnakeoil.com/p/is-gpt-4-getting-worse-over-time > The math questions were of the form “Is 17077 prime”? They picked 500 numbers, but all of them were prime! > The June version of GPT-3.5 and the March version of GPT-4 almost always conclude that the number is prime regardless of whether it is prime or composite. The other two models do the opposite. But the paper only tested prime numbers, and hence concluded that GPT-3.5’s performance improved while GPT-4’s degraded.

The author says: > Do the signatories believe that existing AI systems or their immediate successors might wipe us all out? If they do, then the industry leaders signing this statement should immediately shut down their data centres and hand everything over to national governments. The researchers should stop trying to make existing AI systems safe, and instead call for their elimination. But a different set of people called for a halt to AI research two months ago, and his response then (https://aisnakeoil.substack.com/p/a-misleading-open-letter-about-sci) was that these concerns are "fever dreams" and we should disregard them in favor of "serious policy debates". It seems like there's a self-reinforcing system here, where any evidence or argument in favor of existential risks from AI can be dismissed out of hand.

Beautiful theory, meet ugly fact: https://aisnakeoil.substack.com/p/gpt-4-and-professional-benchmarks

Equally there’s evidence that GPT-4 only excels on tests that happened to appear in it train data (presumably via accidental contamination). As has been demonstrated by it strong ability to solve tests written before its data cutoff date (Sep 2021), but struggles with equivalent tests written after that date. https://twitter.com/cHHillee/status/1635790330854526981 https://towardsdatascience.com/the-decontaminated-evaluation-of-gpt-4-… https://aisnakeoil.substack.com/p/gpt-4-and-professional-benchmarks

You're better off placing your bet on Russian and Chinese hackers, crypto scammers than a Joe Shmoe. But read https://aisnakeoil.substack.com/p/the-llama-is-out-of-the-bag-should - there's no noticeable rise in misinformation

https://aisnakeoil.substack.com/p/gpt-4-and-professional-benchmarks "GPT-4 and professional benchmarks: the wrong answer to the wrong question OpenAI may have tested on the training data. Besides, human benchmarks are meaningless for bots."

Related: https://aisnakeoil.substack.com/p/the-bait-and-switch-behind-ai-risk https://www.sciencedirect.com/science/article/pii/S0267364922000292

If your knee jerk reaction to this headline is to say "yeah, classroom assignments are stupid and wrong, homework needs to adapt", then you should read the article, because (a) that's not at all what this article is about, and (b) the professor largely agrees with that sentiment, even having written another article[0] about it. [0] https://aisnakeoil.substack.com/p/students-are-acing-their-homework

What this work actually shows is that a bunch of scientists are ignorant about how LLMs work, but are rushing to publish papers about them anyway. It is ridiculous for Thirunavukarasu to draw this conclusion from GPT-4's performance on a written ophthalmology exam. From the good folks at AI Snake Oil[1] > Memorization is a spectrum. Even if a language model hasn’t seen an exact problem on a training set, it has inevitably seen examples that are pretty close, simply because of the size of the training corpus. That means it can get away with a much shallower level of reasoning....In some real-world tasks, shallow reasoning may be sufficient, but not always. The world is constantly changing, so if a bot is asked to analyze the legal consequences of a new technology or a new judicial decision, it doesn’t have much to draw upon. In short, as Emily Bender points out, tests designed for humans lack construct validity when applied to bots. > On top of this, professional exams, especially the bar exam, notoriously overemphasize subject-matter knowledge and underemphasize real-world skills, which are far harder to measure in a standardized, computer-administered way. In other words, not only do these exams emphasize the wrong thing, they overemphasize precisely the thing that language models are good at. Also[2]: > Undoubtedly, AI and LLMs will transform every facet of what we do, from research and writing to graphic design and medical diagnosis. However, its current success in…

People have a hard time understanding what zero-shot, generalization, and memorization mean. Generative models are VERY hard to generate even before we begin to look at LLMs. Let me explain and hopefully we can stop this madness. Zero Shot: > Zero-shot learning consists in learning how to recognize new concepts by just having a description of them.[0] Here's an example of a zero shot task. Suppose a LLM is trained only on text. Then you fine tune for image classification, but those do not include cats (of any type). Then you ask it to classify a picture of a cat. An object it has NEVER SEEN BEFORE. The community has been pulling a fast one recently. Recent works like Imagen, DALLE, Parti, etc have been claiming a "Zero-Shot MS-COCO". These are 100% bullshit claims. You can go look at images in the COCO dataset[1] and then search them in the LION dataset[2] (CLIP retrieval). You'll see that there are similar images that have the same classes. These models may not have seen the exact same image before, but they've seen plenty of examples. This is NOT zero shot. Generalization: Google's developer pages[3] defines generalization as > Generalization refers to your model's ability to adapt properly to new, previously unseen data, drawn from the same distribution as the one used to create the model. The irony being that these "Zero-Shot MS-COCO" results they give are actually good tests for generalization. Assuming the datasets these were trained on were held constant (they…