Alex Irpan

Thanks a lot, and another great suggestion for improvement. I also found that the common advice is "tweak hyperparameters until you find the right combination". That can definitely help. But usually issues hide in different "corners", both of the problem space and its formulation, the algorithm itself (e.g., just different random seeds have big variance in performance), and more. As you mentioned, in real applications of DRL things tend to go wrong more often than right: "it doesn't work just yet" [1]. And my short tutorial definitely lacks in the area of troubleshooting, tuning, and "productionisation". If I carve time for expansion, this will likely make top of list. Thanks again. [1] https://www.alexirpan.com/2018/02/14/rl-hard.html

If you are not familiar with RL, I recommend first reading the two articles that the author links to: - https://www.alexirpan.com/2018/02/14/rl-hard.html - https://himanshusahni.github.io/2018/02/23/reinforcement-learning-neve… They are no so recent anymore, but still capture the problem well. Long story short: RL doesn't work yet. We're not sure it'll ever work. Some big companies are betting that it will. > My own hypothesis is that the reward function for learning organisms is really driven from maintaining homeostasis and minimizing surprise. Both directions are actively researched: maximizing surprise (to improve exploration), and minimizing surprise (to improve exploitation). See eg "Exploration by Random Network Distillation" for the first, "SURPRISE MINIMIZING RL IN DYNAMIC ENVIRONMENTS" for the second.

That's a great example. It would be great if you could write a blog post on OpenAI Five. There's a LOT of misinformation on this and could use a treatment like this: https://www.alexirpan.com/2018/02/14/rl-hard.html

At the beginning of this post the given definition of value doesn't seem correct to me, because I think it should be the expected value of the sum of all the rewards from t to infinity and not of R_t, but I could be wrong. Citing D. Silver: The agent's job is to maximise cumulative rewards (highlight cumulative). I would suggest David Silver course for reinforcement learning, http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html, another interesting post is https://jeremykun.com/tag/bandit-learning/ and the following (5 related post comparing implementations of the algorithms in python). I know I should not discourage people, but you should read the post Deep reinforcement learning doesn't work yet before drinking all the cool aid: https://www.alexirpan.com/2018/02/14/rl-hard.html Edited many times.

>I suggest reading Sutton's book on RL I've read part of the book, and followed David Silvers UCL lectures on my own, and have been focused on reading all the recent algorithm/approaches in the field last few weeks. >RL is extensively used in robotics ;) Is it? From Andrej Karpathy:s HN comment[1] and Alex Ipran's[2] comment in 2017 & 2018 I feel like it is just recently (around mid 2019) actual trial on robotics have started to show promise (SAC, E2E) in academia, and only a handful of start ups have approached this (Covariant, founded by one of the professor in the two papers above) But then again, I might be missing some insight here. Thank you for the extremely valuable feedback! :) The formulation of important research question is also something I'm trying to figure out by, getting myself updated in the field to the point of understanding what is missing (or under-studied)! As I mentioned in another comment I'm in the process of applying at a public research institute (as an research assistant) with potential for an industrial PhD position, so those feedback is just on-point valuable for me right now :) (Although the research is not RL focused but is trying to tackle the identical problem formulation in robotics using ML/AI algorithms) [1] https://news.ycombinator.com/item?id=13519044 [2] https://www.alexirpan.com/2018/02/14/rl-hard.html

I see what you mean- thank you for the clarification. Well, I'm not really the right person to discuss this issue since I can't claim to understand how the brain works. However, I do understand a few things about connectionist methods and it's my understanding that they are not very good models of the way the brain works at all (am I misrepresenting your turn of phrase, of "biologically plausible"?). In that sense, I doubt it's possible that a neural net would develop symbolic reasoning by dint of being in some way similar to a biological brain. In general, my experience with neural nets and gradient optimistation techniques is that unless a great deal of effort is spent directing their learning, they are prone to learn whatever is convenient, which is very often not what the human users want it to. For instance, see the following collection of anecdotes of evolutionary algorithms lerning whatever they please, rather than what the researchers were trying to teach them: https://arxiv.org/abs/1803.03453 Or the descriptions of the difficulties of training Deep Reinforcement Learning models in this article: https://www.alexirpan.com/2018/02/14/rl-hard.html Also, while I'm a staunch symbolicist myself, I'm not 100% convinced that symbolic reasoning is "natural". I think it rather took a lot of effort to develop such systems and most people still have a great deal of trouble using them with precision. If you meant to say that symbolic reasoning should arise spontaneously in…

> https://www.alexirpan.com/public/rl-hard/upsidedown_half_cheetah.mp4 You can argue, but it works. Goal accomplished. Nature did things that are stranger than that. Now you just have to add efficiency aspects into your reward functions and observe how it slowly finds a local minimum. Also something nature did a lot of times. Now hope that the remaining inefficiency is ok for you. Done. - You can start with a random neural net. It's not exactly empty, but it's ok. The randomness defines which local minimum you'll find this time around while you burn through your VC money desperately hoping that the AWS bills for all the GPU time will arrive after you've found the holy grail that makes your startup worth <insert bullshit valuation>. > "picking the right reward function" in RL is shockingly hard. Agree. x1000. And computational power and data quality. - on another post I've added some links (AI playing Mario): https://news.ycombinator.com/item?id=17489459

This is a very poor article. It manages to both be demeaning to people who want to learn something new while also being riddled with factual errors. Here are just a few factual errors: 1. The 11-line example is the simplest possible neural network with a very basic implementation of gradient descent in a high-level vectorization framework (numpy). It's exactly what you'd cover in the first week of a course on neural nets. But obviously there are many things to learn beyond that before you can solve real problems effectively with neural nets. In fact, the example code is 100% lifted from Hacker News user williamtrask's great article [1] which ironically closes with a list of 10 topics to investigate next to start learning the additional things you'd need to make your neural net useful for real problems. 2. This section: > '“Deep Learning” and n-Layers of depth is just a neural network that runs its output through itself. This is called Recursive Neural Networks (RNN), because you loop the loop.' ..is full of errors. RNNs are Recurrent, not Recursive. They have nothing to do with having more layers, but with having a memory of previous states. And in deep learning, having more layers is often a strategy to reduce the dimensionality of the input and learn higher-level representations of data by using a variety of tricks/layer types/connections. It's not just "looping the loop". 3. The whole section about not initializing the random state and thus getting different results…

With all due respect, this is a quite random reading list, and appears more a result of the 'if "deep learning" in post.title: post.upvote()' trend on Hacker News ... I will pick on two under the "Classics" section simply as I know the author and have used their work, so am not in any way saying the work isn't useful (it certainly can be in the right spot!), but it isn't "classic" or what I'd recommend for early readers at all. "Uncertainty in Deep Learning" and "Dropout as a Bayesian Approximation" were both published within the last year and a half and is a PhD thesis + paper on interpretations of neural networks in a Bayesian fashion. "Classic" for a paper + thesis less than two years old is quite a stretch even for the fast moving field of deep learning. The same holds true for many of the other papers in "Classics" such as batch norm which is (a) recent and (b) certainly not the only of such techniques (see layer norm, recurrent batch norm, ...) and (c) has complications in implementation[1]. As the simplest example, why is the original dropout paper[7] not under classics? It's an elegant paper, fundamentally important for current neural networks, and is more classic than "Dropout as a Bayesian Approximation" or "Dropout Rademacher Complexity of Deep Neural Networks" which are both listed. I'm also highly dubious of the noted neuroscience connection - most deep learning researchers use very little from neuroscience. Again, this list may be helpful to the creator of…

Here's an interesting blog post on the limitations of domain randomization for OpenAI's results: https://www.alexirpan.com/2019/10/29/openai-rubiks.html Basically the solve rate was much lower without the use of a Bluetooth sensor, and they did a bunch of other things that made the result less impressive. Still a long way to go here.

See also https://www.alexirpan.com/2024/07/08/tragedies-of-reality.html

Current state-of-the-art in reinforcement learning can barely make a physical robot walk. In theory, with transfer learning, we will probably see better success over time but I'm looking forward to seeing results in practice. A 2018 article about the challenges of reinforcement learning: https://www.alexirpan.com/2018/02/14/rl-hard.html

I think https://www.alexirpan.com/2019/10/29/openai-rubiks.html is a pretty good take. tldr: unfortunately, even tiny discrepancies between the simulation and reality (like the size of the cube's bezels or a few extra parts in the robotic hand) can significantly degrade your model's performance in the real world, even with tons of domain randomization during training. So, either we need to improve our simulations to randomize a lot more details, or we need to improve our neural nets to generalize more effectively.

Some more discussion on the difficulty of creating reward functions: https://www.alexirpan.com/2018/02/14/rl-hard.html#reinforcement-learni…

It's disputable whether any such limitations currently applied to AlphaStar are really working as intended. There has been some controversy over it. I saw the matches and it did seem that some of the actions taken by AlphaStar would not be possible to do by a human. Human APM is limited not just by the mechanical speed of one's hands, but also by the mental models that exist in one's brain. Muscle memory can't do everything for you. But in AlphaStar's case, it can perform every type of action as if it were done with the speed of muscle memory. The original commenter's remarks about TTFA seems to be equivalent to what I'm referring to. Relevant reading: https://www.alexirpan.com/2019/02/22/alphastar.html

Not meant as a critique to this book, but the fact that RL-based approaches rarely work for optimal control problems (in, for example, robotics) came as a surprise to me, given the hype and focus on RL [1]. It turns out that model-based methods for optimal control (e.g. linear quadratic control) invented quite a long time ago dramatically outperform RL-based approaches in most tasks and require multiple orders of magnitude less computational resources. Maybe there's some hope for RL method if they "course correct" for simpler control methods. At the moment, it seems like RL for robotics and control lies to the side of "research" and not "engineering." [1]: https://www.alexirpan.com/2018/02/14/rl-hard.html

From Alex Irpan's response: I was sour on the results themselves, because they smelled too much like PR, like a result that was shaped by PR, warped in a way that preferred flashy numbers too much and applicability too little Harsh! OpenReview's double blind system seems to work quite well in this regard as a peer review mechanism. Quick Opinions on Go-Explore https://www.alexirpan.com/2018/11/27/go-explore.html

https://www.alexirpan.com/2018/02/14/rl-hard.html Reinforcment learning for the average person is a big waste of time. Probably for anyone atm

Two months to advance the state of the art on a complex physics-based game with branching paths. Current approaches such as DQNs or god forbid DRL[1] barely reach the performance of my three year old cousin in atari game score maximization and are mostly non-transferable to new levels... Good luck. [1] https://www.alexirpan.com/2018/02/14/rl-hard.html

Both are tricky subject and lot of cooking-like thinking (there are popular recipes, general rules, but in all practical cases to master it there is a lot of non-transferable experience and knowledge). Some pointers: * http://rishy.github.io/ml/2017/01/05/how-to-train-your-dnn/ * http://www.alexirpan.com/2017/06/27/hyperparam-spectral.html At deepsense.io we are developing Neptune to facilitate that process (compare models, version-control them), here: https://go.neptune.deepsense.io (a public release in less that a month!).

https://www.alexirpan.com/2015/09/09/the-other-two-envelope-problem.ht… If your solution is the same as this article's, it's plain wrong. Even the natural number case is plain strong. It's very easy to demostrate as well: consider a trivia case where the distribution is just {P(1)=1/3, P(2)=1/3, P(3)=1/3} and you see 2 in the first envelope. There is no strategy to get a better chance than 50%. Therefore, any strategy that gives a better chance than 50% must implicitly make an assumption over the initial distribution (and therefore excludes a distribution like {P(1)=1/3, P(2)=1/3, P(3)=1/3}) Actually the article is even "wronger" than this, because "started A" and "switched" aren't independent and one can't simply use the product of their probability. The above example is a quick way to demonstrate it's not a general strategy without assumption to get >50% winning chance. Similarily, one can just use {P(1)=1/3, P(2)=1/3, P(3)=1/3} (this is a valid distribution over real numbers!) to demonstrate the real number strategy isn't general. Again, for both natural number and real number case, the discussion over strategies is only meaningful is we know something about the distribution. Interestingly, this article is wrong more or less in the same way as believing switching does give you more expected value in the original "twice money in another envelope" variation. Edit: For people who are interested in the switching strategy, check Randomized Switching in the Two-Envelope…

Edit 2: Reading through the "supplementary data" of the 2019 paper, it definitely appears that the AlphaStar which reached grandmaster was not limited in the same ways as the 2017 paper would suggest. x/y positions of units are not determined visually, but fed directly from the API. So AlphaStar absolutely can just run Attack(Position: carrier_of_interest->pos.x) and not mis-click. It's "map" / "vision" is really just a bounding box of an array of every entity/unit on the map and all the things that a human would have to spend APM to manually check (precise energy level, precise hit points remaining, exact location of invisible units, etc). See [7]. DeepMind showed they have some fixed time delays to emulate human experience, if they had a position fuzzer, they would have mentioned it. I'm reasonably convinced they gave AlphaStar huge advantages even in the 2019 version that was 'nerfed' from the 2018 version. The 2017 paper was a more ambitious project IMO that didn't quite get fully developed. Edit: 7 minutes after writing this I re-read the original paper[-1]. https://arxiv.org/pdf/1708.04782.pdf page 6 and 7 make it clear that DeepMind limited themselves to SpatialActions, so they cannot tell units "Attack Carrier" but have to say "Attack point x,y" (and x,y also has to be determined visually, not through carrier_of_interest->pos.x ). It's still not clear in the paper if any randomness is added to Attack(x,y). Additionally, I have some serious concerns about assuming…

In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6. “What are you doing?”, asked Minsky. “I am training a randomly wired neural net to play Tic-Tac-Toe” Sussman replied. “Why is the net wired randomly?”, asked Minsky. “I do not want it to have any preconceptions of how to play”, Sussman said. Minsky then shut his eyes. “Why do you close your eyes?”, Sussman asked his teacher. “So that the room will be empty.” At that moment, Sussman was enlightened. "picking the right reward function" in RL is shockingly hard. It actually works OK-ish when the problem space is strictly bounded, like with a game whose rules are known. After that, you start getting into sky-humping cheetah problems:https://www.alexirpan.com/public/rl-hard/upsidedown_half_cheetah.mp4 https://www.alexirpan.com/2018/02/14/rl-hard.html is a better article, perhaps, than this one.

Agh I typed up a response but it got deleted somehow. But basically, I view there as being 3 layers of understanding. For whatever level you're stuck at, there's different ways to get "unstuck". 1. Understand the task they're solving, how they did it, and their results. Anybody with a basic understanding of the domain should be able to do this by reading the abstract/intro and conclusion. If you find yourself having trouble here, you probably just need more background in the field. 2. Gain some intuition of why their method works. This is probably one of the hardest parts to figure out how to do, and is probably the part that most people stumble on. Really, this is basically the entirety of what you're trying to do when you learn math. There's also varying levels of intuition. There's "I get why this might work", "I get why this works", and "I get why it's impossible that this doesn't work", in order of difficulty. The more background you have, the easier this intuition is to grasp. Alternatively, you can bootstrap your intuition by reading other people's blog posts, talking to somebody who understands the paper, asking the authors, playing around with your own implementations, etc. I'm not sure anybody has any good answers for this stage. Personally, I really like good blog posts from bloggers I know are good, but unluckily, many papers do not have blog posts attached :( 3. Finally, there's the strict mathematical rigor part. These levels aren't really strict;…

AGI = Artificial General Intelligence, watch this for the main idea around the goal alignment problem: https://www.youtube.com/watch?v=EUjc1WuyPT8 They're explicitly not political, lesswrong is a website/community and rationality is about trying to think better by being aware of normal cognitive biases and correcting for them. Also trying to make better predictions and understand things better by applying Bayes' theorem when possible to account for new evidence: https://en.wikipedia.org/wiki/Bayes%27_theorem (and being willing to change your mind when the evidence changes). It's about trying to understand and accept what's true no matter what political tribe it could potentially align with. See: https://www.lesswrong.com/rationality For more reading about AGI: Books: - Superintelligence (I find his writing style somewhat tedious, but this is one of the original sources for a lot of the ideas): https://www.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostr… - Human Compatible: https://www.amazon.com/Human-Compatible-Artificial-Intelligence-Proble… - Life 3.0, A lot of the same ideas, but the other extreme of writing style from superintelligence makes it more accessible: https://www.amazon.com/Life-3-0-Being-Artificial-Intelligence/dp/11019… Blog Posts: - https://intelligence.org/2017/10/13/fire-alarm/ - https://www.lesswrong.com/tag/artificial-general-intelligence -…