Jascha Sohl-Dickstein

One of my favourite suggestions to counter this from an essay on overfitting I really like [1] is that applicant's probability of being accepted is proportional to their position in the ranking. The downside is that some great applicants will not enter their top choice of school, and some people who aren't great fits will. But, on the other hand, the perverse incentive of spending the entirety of high-school optimising for some arbitrary metrics will dissappear. Any marginal improvement is corralated with a marginal increase in success, rather than the current system of no pay-off whatsoever, until reaching some arbitrary threshold, where one gets all the pay-off at once. [1]: https://sohl-dickstein.github.io/2022/11/06/strong-Goodhart.html

Love this framing I like it because it makes it clear that the boundaries of those stable structures are what encode the world model that encompasses the future. Conscious systems just decode the boundaries of these recursive structures. Corollary: LLMs aren't smart, they're just recursive structures (differently recursive than our minds) that can decode the boundary we store in language the recursion of language...? [2][3] --- My semi-related crackpot theory (informed by some pending unpublished research involving Yakir Aharonov and Michael Levin) is that fundamentally, the structures of consciousness are just the result of thermodynamic evolution finding a way to emulate quantum effects at the macro scale: assuming Aharonov's two-state vector formalism[3] holds some truth, the quantum present is formed by a particle arriving from the past and future. Cross your eyes, and consciousness sure looks a lot like time travel, or information arriving from the future. Aka consciousness is just a roundabout way to create an emulation layer for quantum effects. [1]: http://sohl-dickstein.github.io/2024/02/12/fractal.html [2]: https://news.ycombinator.com/item?id=46191597 [3]: https://www.researchgate.net/publication/226852316

I think the charitable interpretation of this article is as a case of overfitting on the proxy metric [1]. If you care exclusively about reducing CO2 emissions, then her arguments have a point. But that's not how it works in reality: Even in environmental science, CO2 emissions are only one problem, next to other greenhouse gases, ocean acidification, biodiversity loss, etc. Optimizing for CO2 emissions at the expense of the other goals does not solve the environmental problems. Apart from that, even the most idealist and selfless people have additional goals that have to be weighted, such as living healthy, reducing non climate-affecting pollution and environmental hazards, reducing harm to animals, reducing exploitation and inequality, etc. Again, ignoring all those other goals is unrealistic: We could probably make great progress on CO2 reduction if we permantly moved into a covid-style total lockdown - but no one wants to live that way. Maybe hormone and antibiotics-fueled factory farming produces more meat per CO2 unit, but what does that help if it also degrades the soil, increases the chance of superbugs and lowers overall health? There is a less charitable interpretation of the post however, which is that this is a classic muddying-the-water piece. I think there are at least some indications of that. I don't see any obvious problems with her data, but I find it puzzling that she jumps right to the most controversial and counter-intuitive explanations that can be…

Every time “relentless push for efficiency destroying X” comes up I should repost this: https://sohl-dickstein.github.io/2022/11/06/strong-Goodhart.html This is a well understood phenomenon.

It's nice seeing this with Sohl-Dickstein as the last author after reading this blog post from him some time ago: https://sohl-dickstein.github.io/2023/03/09/coherence.html

Overoptimization is the root of all evil. See also https://sohl-dickstein.github.io/2022/11/06/strong-Goodhart.html

Test scores do not measure directly what they purport to measure. They are a proxy, and when you have a system designed to optimize for a proxy of the thing you want to improve, then the system will always find ways to exploit the difference between the proxy and the underlying thing. You can call it "juking the stats" or "overfitting", or whatever. https://sohl-dickstein.github.io/2022/11/06/strong-Goodhart.html

This reminds me of something that appeared on hacker news a couple days ago: https://sohl-dickstein.github.io/2022/11/06/strong-Goodhart.html Basically, the proxy objective is "getting into YC" -- the real objective is producing value of course. We've started optimizing for the proxy. Which produces outcomes like this, where YC is a Thing You Do, after Graduating From Stanford, and before Becoming a YC Alum.

Reminded me of this recent observation of fractals in nn training https://sohl-dickstein.github.io/2024/02/12/fractal.html

The blog post would be a better link for this submission. https://sohl-dickstein.github.io/2024/02/12/fractal.html

Here's the associated blog post, which includes the videos: https://sohl-dickstein.github.io/2024/02/12/fractal.html Not a ML'er so not sure what to make of it, beyond a fascinating connection.

Point taken about the level of discourse on the internet and the mainstream media, no argument there. It is painful, but just because the discourse around the topic is smooth brained doesn't mean that the position is wrong. This is something that I struggle with. Sometimes the discourse is so unimaginably stupid that it pushes you to default to the other side's point of view, but that doesn't actually mean you got to that point of view through any reasoning, you got there from emotion. There are the standard points around consumer protection. Ticketmaster, Adobe, Amazon, really do engage in anti competitive practices and then burn the consumer once they've consolidated power. People feel that, but to your point, can't articulate all the moving pieces, and that's fine, use the mute button. God help us if Kroger and Safeway get their way and merge. To your point about "don't forget who's really in charge." - it might feel like there's no upside here, but in reality you do end up with companies that have more power than the government, and that's a dangerous road to go down in a democracy, since there's no real way to hold them accountable. Obviously the government is deeply flawed in a million different ways, but ultimately there are elected officials, which cannot be said of private companies. Also Lina Khan talks a lot about resiliency. Too much efficiency can make things worse [1]. With mass consolidation, black swan events lead to and outsized impact compared to what…