Cosma Shalizi

For what one person's opinion is worth, I quite like the look of old-style text-and-hyperlink websites, so long as there's something interesting to read on them. A canonical example: http://bactra.org/

Given that he’s known for priority disputes and legal action over who did what and when, that “well-accomplished” bit should have an asterisk next to it. http://bactra.org/reviews/wolfram/ is still a classic.

It's just a re-invention of kernel smoothing. Cosma Shalizi has an excellent write up on this [0]. Once you recognize this it's a wonderful re-framing of what a transformer is doing under the hood: you're effectively learning a bunch of sophisticated kernels (though the FF part) and then applying kernel smoothing in different ways through the attention layers. It makes you realize that Transformers are philosophically much closer to things like Gaussian Processes (which are also just a bunch of kernel manipulation). 0. http://bactra.org/notebooks/nn-attention-and-transformers.html

The subject here and now is Wolfram and the irritation of many comes not from the work that Wolfram has achieved on his own but the utter lack of acknowledgement | recogonition of work done by others .. a substantial amount of work by others. His New Kind of Science both grossly overstates the importance of cellular autonoma being Turing complete and utterly ignores a vast community of others who had similar ideas to Wolfram, many before Wolfram, and some whose work it has been said he lifted directly and claimed as his own. Like if Michalangelo stood before the Mona Lisa and been like “look how amazing what I did was”. See: (for example) A Rare Blend of Monster Raving Egomania and Utter Batshit Insanity http://bactra.org/reviews/wolfram/

I particularly enjoy this classic review. Very informative: http://www.bactra.org/reviews/wolfram/

While I absolutely love this illustration (and frankly everything Jay Alammar does), it is worth recognizing there is a distinction between visualizing how a transformer (or any model really works) and what the transformer is doing. My favorite article on the latter is Cosma Shalizi's excellent post showing that all "attention" is really doing is kernel smoothing [0]. Personally having this 'click' was a bigger insight for me than walking through this post and implementing "attention is all you need". In a very real sense transformers are just performing compression and providing a soft lookup functionality on top of an unimaginably large dataset (basically the majority of human writing). This understanding of LLMs helps to better understand their limitations as well as their, imho untapped, usefulness. 0. http://bactra.org/notebooks/nn-attention-and-transformers.html

I found Cosma Shalizi's notes on the subject pretty insightful. http://bactra.org/notebooks/nn-attention-and-transformers.html Definitely read through to the last section.

I think this is required reading on IQ and g: http://bactra.org/weblog/520.html http://bactra.org/weblog/523.html On efficient markets, I'd recommend: https://economics.northwestern.edu/docs/events/nemmers/2004/nemmers-05…

Cool! I had been thinking about trying this as well, after reading about the idea in one of Cosma Shalizi's notebooks [0]. I'd love to see how something like this performs when "trained" on a corpus the size of the web when given the same kind of computational resources used to train modern LLMs. [0] http://bactra.org/notebooks/nn-attention-and-transformers.html#gllz

Cosma Shalizi explains it very well. "Again: Calling this "attention" at best a joke." http://bactra.org/notebooks/nn-attention-and-transformers.html

For anyone else seeing this comment, I recommend the following blog post: http://bactra.org/weblog/918.html "Attention conservation notice: Over 7800 words about optimal planning for a socialist economy and its intersection with computational complexity theory. This is about as relevant to the world around us as debating whether a devotee of the Olympian gods should approve of transgenic organisms. (Or: centaurs, yes or no?) Contains mathematical symbols but no actual math, and uses Red Plenty mostly as a launching point for a tangent." I made it about a quarter of the way through before I had to return my copy to the library - not because it wasn't well written, but my attention span has been shot by the internet and smart phones.

Yes, actually. It's a theme that Scott is annoyingly fond of, in his fiction and non-fiction. Unsong is entirely about kabbalistic puzzles as you know. Scott and others before him were enamored with the ideas of Leo Strauss. You can read more about Strauss' ideology of hidden truths that only the brilliant and/or worthy can perceive at the link below. Scott Alexander uses the Kolmogorov post to have plausible deniability for his discussion of "race science". There's a reason writing "HBD" was banned in the comments section and it's that it was giving the game away to too many people. (discussion of HBD is not banned, it's simply not permitted for that string to appear - i wonder why that might be) http://bactra.org/weblog/algae-2014-06.html#drury-on-strauss Have you missed this post? http://slatestarcodex.com/2017/10/23/kolmogorov-complicity-and-the-par…

In case no one has read it yet, here is a very amusing take down of Wolfram and why he is the classic crank theorist http://bactra.org/reviews/wolfram/

As far as i understand it the RANSAC arguments are very much not Bayesian (they're all about "if you repeat this an infinite number of times under an infinite number of new samples, then...", which is almost caricaturely frequentist). Still, you just gave me reason to mention one of my favorite "no, that won't work either" paper :) on how Bayes will not save you in the presence of model misspecification. Instead of butchering it any further, I'll just point you to this piece explaining the work, written by the author of the paper himself: http://bactra.org/weblog/601.html

Classic: "A Rare Blend of Monster Raving Egomania and Utter Batshit Insanity" - http://bactra.org/reviews/wolfram/

Obligatory mention that he spent 10 years writing a 1000+ page book of nonsense called A New Kind of Science that was basically about how great he is and how everyone else is an idiot: http://bactra.org/reviews/wolfram/ It's not just about productivity, people...

Today I came across Cosma Shalizi's review of A New Kind of Science, and his page with 169 book reviews 1995-2018, I've read half a dozen so far, they're all great - he has a very attractive style. A wide range: science, politics, social science, history, philosophy, mathematics.. http://bactra.org/reviews/

For those more interested in this, there's a follow-up to comments, which results in this (still on-going?) exchange: - The original article (linked by OP): http://crookedtimber.org/2012/05/30/in-soviet-union-optimization-probl… - A comment made by Paul Cockshott (a Marxist economist and computer scientist, most known for "Towards New Socialism" book): http://crookedtimber.org/2012/05/30/in-soviet-union-optimization-probl… - Shalizi's response to Cockshott, among his other responses to other comments: http://bactra.org/weblog/919.html - Cockshott's response to above: https://paulcockshott.wordpress.com/2018/06/25/plan-balancing-algorith…

Why does this need to be a 2D simulation? I don't see how geometry comes into it at all. Also worth reading http://bactra.org/weblog/491.html

It's a relatively common phenomenon in hydrodynamics (specifically, the formation of large separated cyclones in unstable flow), and the best explanation is delightful -- Onsager's "The Little Vortices Who Wanted To Play" (a letter to Linus Pauling, reproduced on page 33 of this PDF: http://bactra.org/sloth/eyink-and-sreenivasan-on-onsager-on-turbulence… ).

Why is "pervasive negativity" a reason to ignore someone insightful? Do what you will, but you should at least check out the author's meticulous beat-down of Stephen Wolfram for peddling nonsense: http://bactra.org/reviews/wolfram/

The author has a pretty interesting homepage. Especially the notebooks http://bactra.org

Always a good time to re-read Cosma Shalizi's classic "So You Think You Have a Power Law — Well Isn't That Special?" (http://bactra.org/weblog/491.html)

"More worryingly, six were caused by staff members "pressing the wrong buttons." I'm sure in one of the classic UI books (maybe Psychology of Everyday Things by Norman) there's a photo from a nuclear power plant, where there's two giant levers right next to each other, and the plant workers have attached a handle from a beer pump to one of them to distinguish it from the other, because as built they were identical and they kept pulling the wrong one. edit: found a page referencing it, but the actual image has gone: http://bactra.org/reviews/everyday-things/nukewiser.html You can see it here: https://issuu.com/design.nerve/docs/the_design_of_everyday_things Page 95 as the book is numbered, but page 111 in the web interface control.

And in 2012, the follow-up, "Ten Years of Monster Raving Egomania and Utter Batshit Insanity:" http://bactra.org/weblog/915.html

>A large number of natural distributions follow a power-law distribution. This is surprisingly debatable. People like to find power-law/scale-free distributions because it implies a neat generative story, but a lot of the "evidence" for power-law distributions is pretty weak. For example, you cannot just show that a log-log plot is linear--lots of other distributions can produce similar plots. Clauset, Shalizi, and Newman have a very readable paper where they describe 1) how to properly test for a power-law distribution, and 2) use those tests to assess the validity of some claims from the literature (spoiler: not many have "good" statistical support). Here is the paper: http://arxiv.org/abs/0706.1062 Shalizi has a short blog post describing the main results: http://bactra.org/weblog/491.html

Although Shalizi has his own biases and is pretty opinionated, his review of "A New Kind of Science" is quite interesting and is worth considering: http://bactra.org/reviews/wolfram/

We declare that the splendor of the world has been enriched by a new beauty: the beauty of speed. A racing automobile with its bonnet adorned with great tubes like serpents with explosive breath ... a roaring motor car which seems to run on machine-gun fire, is more beautiful than the Victory of Samothrace. -- The Futurist Manifesto, 1908 [1] It didn't turn out well. http://bactra.org/T4PM/futurist-manifesto.html

Crutchfield has recorded lectures on YouTube. I thought they were quite good. I'm biased toward learning statistical mechanics and nonlinear dynamics as a starting point. Search "phy 256 a(b) crutchfield" on YouTube. Crutchfield has been doing amazing work for quite some time. If you want a critical and very entertaining read, I would check out cosma shalizi's blog. He does a lot of book reviews too that may be helpful. http://bactra.org/weblog/ Other names/references include "The nature of computation" Stuart kauffman. "Nonlinear dynamics and chaos" Too many stat mech books to possibly recommend, for the idea of emergence you'll want something including phase transitions so... Can't go wrong with 'thermodynamics and an introduction to thermostatistics". And for stat mech with lots of complex systems diversions, Sethna's book is excellent but difficult to learn from. "Entropy, complexity, and order parameters. Names in thermo/complexity: Sethna Goldenfeld Parisi HE stanley Finally, it's hard to talk about complex systems without talking about networks, "Networks"-Newman is unambiguously the best choice. Names in networks: Mark Newman Reka Albert Avoid barabasi especially more recent stuff. There may be better lectures/resources out there I don't know of, but I would start with crutchields lectures. Note again I am coming from heavy physics bias.

This seems to be about easier classification tasks with not too many samples, for which TF-IDF also works well (Table 3). But more generally gzip for text modeling might make sense. Quoting http://bactra.org/notebooks/nn-attention-and-transformers.html : > Once we have a source-coding scheme, we can "invert" it to get conditional probabilities; we could even sample from it to get a generator. (We'd need a little footwork to deal with some technicalities, but not a heck of a lot.) So something I'd really love to see done, by someone with the resources, is the following experiment: > - Code up an implementation of Lempel-Ziv without the limitations built in to (e.g.) gzip; give it as much internal memory to build its dictionary as a large language model gets to store its parameter matrix. Call this "LLZ", for "large Lempel-Ziv". > - Feed LLZ the same corpus of texts used to fit your favorite large language model. Let it build its dictionary from that. (This needs one pass through the corpus...) > - Build the generator from the trained LLZ. > - Swap in this generator for the neural network in a chatbot or similar. Call this horrible thing GLLZ. > In terms of perplexity, GLLZ will be comparable to the neural network, because Lempel-Ziv does, in fact, do universal source coding. Maybe someone on HN will have resources for such an experiment?