Jürgen Schmidhuber

wow, that's beautiful - the whole site https://people.idsia.ch/~juergen/ is an amazing rabbit hole im gonna lose myself in.

He was done a great deal of injustice when he was passed over for the Turing award that was given to Hinton, Bengio, LeCun. Then there is this from his blog -- Dec 2024: Sadly, the Nobel Prize in Physics 2024 for Hopfield & Hinton is a Nobel Prize for plagiarism. They republished methodologies developed in Ukraine and Japan by Ivakhnenko and Amari in the 1960s & 1970s, as well as other techniques, without citing the original papers. Even in later surveys, they didn't credit the original inventors (thus turning what may have been unintentional plagiarism into a deliberate form). None of the important algorithms for modern Artificial Intelligence were created by Hopfield & Hinton. Details in the recent technical report, with lots of references, links, and facts. https://people.idsia.ch/~juergen/physics-nobel-2024-plagiarism.html

Ah you mean Schmidhuber's speed prior: https://people.idsia.ch/~juergen/speedprior.html I was more interested in pointing to you the measly followups to Aaronson's question, which consists solely of forays into nonlinear quantum mechanics & superluminal neutrinos. https://www.preposterousuniverse.com/blog/2011/09/24/can-neutrinos-kil…

Funny ... According to [1], the papers [42,43] showed that plain backprop was capable of solving complex tasks without pretraining on a GPU. From [1]: > Our team then showed that good old backpropagation [A1] on GPUs (with training pattern distortions [42,43] but without any unsupervised pre-training) I can't verify since I have a barely stable connection and little battery, but somebody else should be able to. [1] https://people.idsia.ch/~juergen/firstdeeplearner.html [42] H. Baird. Document image defect models. IAPR Workshop, Syntactic & Structural Pattern Recognition, p 38-46, 1990 [43] P. Y. Simard, D. Steinkraus, J.C. Platt. Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis. ICDAR 2003, p 958-962, 2003.

It also follows his own "fibonacci web design" style [1] [1]: https://people.idsia.ch/~juergen/fibonacciwebdesign.html

> Transistors, the electronic amplifiers and switches found at the heart of everything from pocket radios to warehouse-size supercomputers, were invented in 1947. Is this true? Recently this article https://people.idsia.ch/~juergen/turing-oversold.html pointed to the following invention of the field-effect transistor from 1925: https://patents.google.com/patent/US1745175

> This paper also got me thinking if this mechanism might explain why we enjoy listening to music. Music follows and breaks rules, which our brain continuously tries to predict and most of the time it succeeds. The link between predictability and art has been raised various times in the literature, even before Schmidhuber, but he had particularly interesting ideas about it: http://people.idsia.ch/~juergen/beauty.html

>> It has recently occured to me to ask why we have decided to try solve autonomous driving before solving any other seemingly easier robotics problems. As far as I can tell, a few years ago, Google decided it was a good idea and then everyone else followed suit, because it's Google so they must be on to something. Mind, there was earlier work on self-driving cars that was just as impressive as modern efforts, but you rarely hear about it. Here; You-Again Schmidhuber got the works: http://people.idsia.ch/~juergen/robotcars.html

I remember Schmidhuber showing off "artificial curiosity" stuff a while back (e.g. http://people.idsia.ch/~juergen/interest.html ). In particular, ideas like "compression progress" have been influential on my own research about how to measure what's "interesting", and I've implemented a rudimentary version of PowerPlay (and a slight alternative) at http://chriswarbo.net/projects/powerplay (He's since applied these ideas to art, humour, etc. which I think is a nice thought, but not worth taking particularly seriously)

I followed the link to the OOPS source code from the comments: http://people.idsia.ch/~juergen/oopscode.c (originally from http://people.idsia.ch/~juergen/oops.html) Aaaand… “we're achieving levels of nerdery that shouldn't be possible.”

There's no reason to believe so. Advancements in CS, AI, biology, and general models of intelligence are slowly bringing us to the point where consciousness is no longer a mystery to the enlightened. One of my favorite researchers in general AI is Professor Schmidhuber, who has pioneered some of the (currently) more practical areas of machine learning, too: http://people.idsia.ch/~juergen/ Even when we create self-aware machines that can demonstrate generalized intelligence, I think most people will struggle to come to terms with what consciousness is (or more so, what it isn't). My spin on it: consciousness isn't real; it's a necessary perceived byproduct of 'everything else.' I believe Schmidhuber said it more elegantly, describing consciousness as a result of data compression.

For more details, refer to Schmidhuber's own website: http://people.idsia.ch/~juergen/ and his excellent review paper, "Deep Learning in Neural Networks: An Overview": https://arxiv.org/pdf/1404.7828.pdf

Jurgen Schmidhuber has an interesting perspective on the rewarding aspects of art and music. To paraphrase, "Artists (and observers of art) get rewarded for making (and observing) novel patterns: data that is neither arbitrary (like incompressible random white noise) nor regular in an already known way, but regular in way that is new with respect to the observer's current knowledge, yet learnable (that is, after learning fewer computational resources are needed to encode the data)". In other words, enjoyment of art is about learning (easy) patterns. Schmidhuber likens "fun" to the improvement of an observer's ability to compress a scene. The source is worth the read: http://people.idsia.ch/~juergen/creativity.html

For more practical persons I recommend skipping the historical parts of philosophy and go straight to Bayesianism. I recommend "Rationality" by E. Yudkowsky as an entry point, "Probabilistic Inference" by J. Pearl, "Artificial Intelligence" by Russel & Norvig for the theory and "Formal Theory of Creativity & Fun & Intrinsic Motivation" by J. Schmidhuber for inspiration. [1]: http://people.idsia.ch/~juergen/creativity.html

> This works great for things with patterns, but it doesn't work with language. Sure it does. http://karpathy.github.io/2015/05/21/rnn-effectiveness/ > Hawkins' mantra is sequence based memory. Note that Hawkins wrote “On intelligence” in 2005. Schmidhuber wrote “Learning complex extended sequences using the principle of history compression” in 1992, and “Long short-term memory”, with Hochreiter, in 1997. Today, a LSTM-based RNN can answer your Gmail. http://people.idsia.ch/~juergen/rnn.html http://googleresearch.blogspot.com/2015/11/computer-respond-to-this-em…

Ignoring the singularity talk because I think it's become meaningless, but Jürgen Schmidhuber would disagree with the conclusion that machines can't be creative, or at least that it's beyond algorithms. He has a formal theory of creativity [0] which claims to explain, among other things, music, humour, beauty [1] and fun. It centres around compression and Kolmogorov complexity. There's a great video in the first link. These are hard problems, but it's shortsighted to consider it impossible for us to build machines with approximate behaviour. Often with this class of criticism you'll have arguments along the lines of "sure the submarine moves through the water, but is it swimming?" Apologies to Dijkstra. [0] http://people.idsia.ch/~juergen/creativity.html [1] http://people.idsia.ch/~juergen/beauty.html

1991 > Each RNN tries to solve the pretext task of predicting its next input, sending only unexpected inputs to the next RNN above. This greatly facilitates downstream supervised deep learning such as sequence classification. By 1993, the approach solved problems of depth 1000 (requiring 1000 subsequent computational stages/layers—the more such stages, the deeper the learning). A variant collapses the hierarchy into a single deep net. It uses a so-called conscious chunker RNN which attends to unexpected events that surprise a lower-level so-called subconscious automatiser RNN. The chunker learns to understand the surprising events by predicting them. The automatiser uses my neural knowledge distillation procedure of 1991 [UN0-UN2] to compress and absorb the formerly conscious insights and behaviours of the chunker, thus making them subconscious. The systems of 1991 allowed for much deeper learning than previous methods. https://people.idsia.ch/~juergen/very-deep-learning-1991.html

Ah yes, Perceptron. Had a couple typos.. sorry, was on phone. The bio-inspiration was via Frank Rosenblatt, who is referred to in that article tho yeah, the history is over in his article: https://en.wikipedia.org/wiki/Frank_Rosenblatt#Perceptron "Rosenblatt was best known for the Perceptron, an electronic device which was constructed in accordance with biological principles and showed an ability to learn. He developed and extended this approach in numerous papers and a book called Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms, published by Spartan Books in 1962.[6] He received international recognition for the Perceptron. The Mark I Perceptron, which is generally recognized as a forerunner to artificial intelligence, currently resides in the Smithsonian Institution in Washington D.C." Your Juergen page is interesting, tho no direct comment on Rosenblatt there. He does cite the work on this page: https://people.idsia.ch/~juergen/deep-learning-overview.html (refs R58, R61) My reading is that a long-known idea, about multi-variate regression, was reinterpreted by Rosenblatt by 1958 via the bio-inspired Perceptron, and then that was criticized by Minksy and others and viable methods were achieved by 1965. When I was taught NNs by Mitchell at CMU in the 1990s (lectures similar to his book Machine Learning), this was the same basic story. Also reminds me of a moment in class one day when a Stats Prof who was surveying the course broke out…

Hinton didn’t invent back prop. > Explicit, efficient error backpropagation (BP) in arbitrary, discrete, possibly sparsely connected, NN-like networks apparently was first described in a 1970 master's thesis (Linnainmaa, 1970, 1976), albeit without reference to NNs. BP is also known as the reverse mode of automatic differentiation (e.g., Griewank, 2012), where the costs of forward activation spreading essentially equal the costs of backward derivative calculation. See early BP FORTRAN code (Linnainmaa, 1970) and closely related work (Ostrovskii et al., 1971). > BP was soon explicitly used to minimize cost functions by adapting control parameters (weights) (Dreyfus, 1973). This was followed by some preliminary, NN-specific discussion (Werbos, 1974, section 5.5.1), and a computer program for automatically deriving and implementing BP for any given differentiable system (Speelpenning, 1980). > To my knowledge, the first NN-specific application of efficient BP as above was described by Werbos (1982). Related work was published several years later (Parker, 1985; LeCun, 1985). When computers had become 10,000 times faster per Dollar and much more accessible than those of 1960-1970, a paper of 1986 significantly contributed to the popularisation of BP for NNs (Rumelhart et al., 1986), experimentally demonstrating the emergence of useful internal representations in hidden layers. https://people.idsia.ch/~juergen/who-invented-backpropagation-2014.htm… Hinton wasn’t the first to…

So the unconscious mind is (metaphorically) like a powerful massively parallel computer and the conscious mind is like a shell or GUI. Most of us are running "programs" that make us look, act, and feel like normal humans. (Anthropologists have pointed out that each culture, no matter how different from the others, manages to get its children to act like its adults.) If you learn to reprogram your unconscious patterns you become a meta-computer: a computer that can rewrite its own software to improve itself, a Gödel machine[1]. The main reason why you don't "formulate well-thought responses in real time" is that your ad hoc programming doesn't include that functionality, but it's there, intrinsic to your capabilities as a human being. Hypnosis is a way to put yourself into a sort of "debug mode" and communicate between your conscious and unconscious minds, and, if you like, reprogram your patterns. It's pretty easy to do. In a nutshell, learn to go into trance[2], then tell yourself a story in which you discover how to "formulate well-thought responses in real time" quickly and easily. When you come out of trance, you will. Feel free to ask questions, I'll do my best to answer. Most people aren't interested, so I appreciate the opportunity to pass along what I've learned. Cheers! [1] https://people.idsia.ch/~juergen/goedelmachine.html [2] I recommend "TRANCE-formations" by Bandler and Grinder.

Yes, Schmidhuber could present things in a better way and be more charitable. But do not overlook his work -- he has genuinely groundbreaking contributions to early theory of intelligence. I recommend having a stroll through his website, for anyone interested in AI (or even art, creativity, etc.): https://people.idsia.ch/~juergen/ Special Notes: Godel Machine : While clearly absolutely impractical, this is a nice theoretical construction on what we could mean by "self-improvement" in a real, definite way. Speed Priors Super Omegas (and various thoughts on algorithmic information) Always keep in mind a grain of salt when reading his results, but they are genuinely great. --- I think establishing causation and giving credit does have a role in research, namely giving additional opportunity to those researchers to do even better research, elucidating the history of science, and to an extent giving a monetary incentive and morale recognition for researchers w.r.t. prizes, grants, etc. For example isn't the story of Einstein, or von Neuman, or Feynman, etc. part of a narrative that makes science compelling? They had very interesting and lively lives and opinions that sometimes drove their research tastes. There is value in preserving individual history (as well as disregarding it when not particularly relevant). That said, a scientists (or a good citizen in general) should not be driven primarily by ego, and in science this becomes clearly bad, impeding open…

There is this quote by Jürgen Schmidhuber, director at the Swiss AI Lab IDSIA , creator of LSTM Neural Nets that power many of our current machine learning applications. " General purpose quantum computation won’t work (my prediction of 15 years ago is still standing). Related: The universe is deterministic, and the most efficient program that computes its entire history is short and fast, which means there is little room for true randomness, which is very expensive to compute. What looks random must be pseudorandom, like the decimal expansion of Pi, which is computable by a short program. Many physicists disagree, but Einstein was right: no dice. There is no physical evidence to the contrary randomness.html[http://people.idsia.ch/~juergen/randomness.html]. For example, Bell’s theorem does not contradict this. And any efficient search in program space for the solution to a sufficiently complex problem will create many deterministic universes like ours as a by-product. Think about this. More here computeruniverse.html [http://people.idsia.ch/~juergen/computeruniverse.html] and here. " I am out of depth with regards to quantum theory and quantum computation. If anyone with better knowledge with regards to quantum computation can clarify it would be great. Is there any rationale to support Schmidhuber's argument, and has any advance in making actual quantum computers disprove his theory?

what it is they have against repetition They find it boring. Would you come to HN every day if the linked articles never changed? I'm quite enamoured with Schmidthuber's theory of creativity [1, 2] Schmidhuber develops a theory of artificial curiosity and creativity for an autonomous agent. The agent is equipped with an adaptive predictor trying to predict future events from the history of previous events and actions. A reward-maximising, reinforcement learning, adaptive controller is steering the controller and gets curiosity reward for executing action sequences that improve the predictor. (I term this "Pattern Recognition as Pleasure".) This discourages the agent from executing actions leading to boring outcomes that are either predictable or totally unpredictable. Instead the controller is motivated to learn actions that help the predictor to learn new, previously unknown regularities in its environment, thus improving its model of the world, which in turn can greatly help to solve externally given tasks. Schmidhuber claims that his corresponding formal theory of creativity explains essential aspects of art, science, music, and humor. If we were to take Schmidhuber's approach seriously, then that that which we enjoy as great…

I'm having trouble finding the source of this quote, but here: >Many mathematicians have the opposite opinion; they do not or cannot distinguish the beauty or importance of a theorem from its proof. A theorem that is first published with a long and difficult proof is highly regarded. Someone who, preferably many years later, finds a short proof is "brilliant." But if the short proof had been obtained in the beginning, the theorem might have been disparaged as an "easy result." Erdős was a genius at finding brilliantly simple proofs of deep results, but, until recently, much of his work was ignored by the mathematical establishment. And here is an interesting study on what mathematicians consider beautiful: https://www.gwern.net/docs/math/1990-wells.pdf They found that mathematicians varied a great deal in what they considered beautiful. And that there isn't any consensus on what is considered beautiful. It also changes over time: >There was a notable number of low scores for the high rank theorems 9 Le Lionnais has one explanation [7]: "Euler's formula e^pi*i = - 1 establishes what appeared in its time to be a fantastic connection between the most important numbers in mathematics... It was generally considered 'the most beautiful formula of mathematics' . . . Today the intrinsic reason for this compatibility has become so obvious that the same formula now seems, if not insipid, at least entirely natural." I think the interestingness or surprisingness of a mathematical…

http://nuit-blanche.blogspot.com/2017/01/understanding-deep-learning-r… Here is an interesting paper that pinpoints the influence of regularization on learning with Neural networks. From the paper: > Our central finding can be summarized as: Deep neural networks easily fit random labels. and later: > While simple to state, this observation has profound implications from a statistical learning perspective: > 1. The effective capacity of neural networks is large enough for a brute-force memorization of the entire data set. > 2. Even optimization on random labels remains easy. In fact, training time increases only by a small constant factor compared with training on the true labels. > 3. Randomizing labels is solely a data transformation, leaving all other properties of the learning problem unchanged. I subscribe to Schmidhuber's notion that algorithms that rely upon randomness to select parameters have a very illusory "simplicity" that relies upon externalizing the complexity of the PRNG. http://people.idsia.ch/~juergen/newai/

withfries2: Bengio's 2009 survey and Schmidhuber's "conspiracy" blog post contain useful, accurate historical background from the perspective of two leading researchers, with lots of links to additional sources: http://www.iro.umontreal.ca/~bengioy/papers/ftml.pdf http://people.idsia.ch/~juergen/deep-learning-conspiracy.html Were I in your shoes, I would NOT have highlighted the Google YouTube experiment as "the" big breakthrough. It was just an interesting worthwhile experiment by one of many groups of talented AI researchers who have made slow progress over decades of hard work. Why single it out? -- PS. The YouTube experiment did not produce new theory, and from a practical standpoint, it would be unfair to say that it reignited interest in deep learning. Consider that the paper currently has only ~800 citations, according to Google Scholar.[1] For comparison, Krizhevsky et al's paper describing the deep net that won Imagenet (trained on one computer with one GPU) has over 5000 citations.[2] And neither of these experiments deserves to be called "the" big breakthrough. [1] https://scholar.google.com/citations?view_op=view_citation&hl=en&a… [2] https://scholar.google.com/citations?view_op=view_citation&hl=en&a…

Also see Schmidhuber's take on the Hinton + Hopfield Nobel prize: https://people.idsia.ch/~juergen/physics-nobel-2024-plagiarism.html

I spent some time in the academia. The person with whom an idea ends up associated often isn't the first person to have the idea. Most often is the person who explains why the idea is important, or find a killer application for the idea, or otherwise popularizes the idea. That said, you can open what Schmidhuber would say is the paper which invented residual NNs. Try and see if you notice anything about the paper that perhaps would hinder the adoption of its ideas [1]. [1] https://people.idsia.ch/~juergen/SeppHochreiter1991ThesisAdvisorSchmid…

His view aligns both with a leading neuroscience explanation of the brain (predictive coding [1]) and with Active Inference / the Free Energy principle [2] from optimal control theory. A similar theory of intelligence, called H-JEPA (hierarchical joint embedding predictive architecture) [3] is also put forward by Yann LeCun, a major AI pioneer. Another AI pioneer, Jürgen Schmidhuber, subsequently criticized LeCun's theory, but not for being faulty, but for rehashing ideas published in several papers from the 1990s onward. [4] 1: https://en.wikipedia.org/wiki/Predictive_coding 2: https://en.wikipedia.org/wiki/Free_energy_principle#Active_inference 3: https://openreview.net/forum?id=BZ5a1r-kVsf 4: https://people.idsia.ch/~juergen/lecun-rehash-1990-2022.html