Jason Wei

Hi all! Sharing some of our recent work around building RL envs and sims for agent training. There are a lot more technical details on building the benchmark in the post. If you are interested in more RL/Post-Training, I'd highly recommend reading this super in-depth blog from our partners at Yutori: https://yutori.com/blog/introducing-navigator Some more casual thoughts and lessons: 1) A high volume of quality RL environments / sims remain one of the largest blockers to training frontier agents, especially as labs/enterprises shift towards creating increasingly specialized AI coworkers that can do real work. 2) Building an RL env is VERY different from building a high quality dataset. While the primary input for dataset creation is specialized human annotators and clear rubrics, the inputs to building a great RL env involve humans, engineers, product, data, and an orchestration of everything together. There are a lot of green field problems when you move from building singular environments to SCALING 1-3 orders of magnitude. 3) There is a constant push/pull between building tasks that are easily verifiable and building tasks that are realistic. Its sort of like a 2x2 grid. The best (and most valuable) tasks are realistic and verifiable. There are constant tradeoffs being made, and we often find ourselves limited by the types of realistic tasks we can make if they lack a clear verifier. I'm reminded of Jason Wei's post here:…

Nice share. Increasingly I am thinking about ways to improve verification ("interestingness tests"), ever since reading https://www.jasonwei.net/blog/asymmetry-of-verification-and-verifiers-…

It's fundamentally because of verifier's law [0]. Current AI, and in particular RL-based, is already or will soon achieve super human performance on problems that can be - quickly - verified and measured. So maths, algorithms, etc and well defined bugs fall into that category. However architectural decision, design, long-term planning where there is little data, no model allowing synthetic data generation, and long iteration cycles are not so much amenable to it. [0] https://www.jasonwei.net/blog/asymmetry-of-verification-and-verifiers-…

This depends on the type of work being done. Sometimes the cost of verification is much lower than the cost of doing the work, sometimes it's about the same, and sometimes it's much more. Here's some recent discussion [0] [0] https://www.jasonwei.net/blog/asymmetry-of-verification-and-verifiers-…

We include a disclaimer later that researchers are debating whether it's possible to predict emergent capabilities. Wei has responded to that paper and others at https://www.jasonwei.net/blog/common-arguments-regarding-emergent-abil… at I don't think it's clear who is right

llm have emergent abilities [0] which could provide additional value to any output or label. [0] https://www.jasonwei.net/blog/emergence

Yet there are emergent behaviours from these LLMs that are both surprising and not immediately understood. [1][2][3] Everyone has theories, of course, but still pretty "magic" considering these behaviours weren't theorised in papers prior to observation. 1 - https://www.jasonwei.net/blog/emergence 2 - https://arxiv.org/pdf/2206.07682.pdf 3 - https://www.quantamagazine.org/the-unpredictable-abilities-emerging-fr…

Here is Jason Wei's response "Common arguments regarding emergent abilities": https://www.jasonwei.net/blog/common-arguments-regarding-emergent-abil… Wei was the lead author of the original "Emergent Abilities" paper: Wei, Jason, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, et al. “Emergent Abilities of Large Language Models.” arXiv, October 26, 2022. https://doi.org/10.48550/arXiv.2206.07682.

What about this "emergent" intelligence when size of the model is large enough? There are blogs about it being able to model problems internally or forming a mini NN of its own. As I've understood, even the researchers in the field have no idea why this happens. It's completely empirical and a "discovered phenomenon" at this point. [1]: https://www.jasonwei.net/blog/emergence [2]: https://jacobbrazeal.wordpress.com/2022/09/23/gpt-3-can-find-paths-up-…