Yao Fu

I saw this when 3.5 came out: https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emer… Haven't followed up on all the comments in it but speculates on why chain of thought improves when training on code.

Same here. If you believe the following research (which I do), the ability to perform complex reasoning is likely to be from training on code: https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emer… I think it's essential to increase the quantity of code tokens.

ChatGPT cannot correctly perform step by step operations within strict formal systems. On the other hand, it can provide a high level sketch of how tomathematically prove something[1]. I think this is amazing, because this is the opposite of how we normally think about computers. ChatGPT is highly intuitive instead of a formal reasoning system. No doubt some future AIs will combine this intuition with checks against formal verification engines to check correctness. [1] Section 6 of https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emer…

All 3 text-davinci models are available on openAI's api. including 3 (which is the GPT-3.5 gen). Code-davinci-002 is a code-tuned model, You can see a nice visual summary of the relationships between the openAI models at https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emer… Or the official source is https://platform.openai.com/docs/model-index-for-researchers

> It simplifies prompting and makes the LLM more steerable, more useful, more helpful. While this is true, there is also evidence that RLHF and supervised instruction tuning can hurt output quality and accuracy[1], which are instead better optimized through clever prompting[2]. [1] https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emer… [2] https://yaofu.notion.site/Towards-Complex-Reasoning-the-Polaris-of-Lar…

I saw this in my feed recently which was an interesting analysis on how code training was added as a fine tune (Codex) on a foundational model (GPT-3): https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emer… I do wonder if anyone is considering mixing in larger and larger percentages of The Stack https://huggingface.co/datasets/bigcode/the-stack with this or the Pile to get more code and see what happens. (Likely beyond mere mortals' budgets though.)

Basically agree -- feels off-putting, but not technically a wrong detail to add. An additional reason it rubs me the wrong way, however, is that I believe open-source software code is especially critical to ChatGPT family's capabilities. Not just for code-related queries, but for everything! (e.g. see this "lineage-tracing" blog post: https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emer… Thus, I honestly think firms operating generative AI should be walking on eggshells to avoid placing blame on "open-source". Rather, they really should going out of their way to channel as much positive energy towards it as possible. Still, agree the charitable interpretation is that this just purely descriptive.

I got the impression from this https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emer… > The initial GPT-3 is not trained on code, and it cannot do chain-of-thought Does this mean some extra steps are required to make gpt-3 perform CoT?

The InstructGPT paper only explains the RLHF part of how ChatGPT works. There's reason to believe that isn't enough to achieve ChatGPT's performance and behaviour (e.g. [1]). There are other components that make ChatGPT more powerful, and OpenAPI is not being open about them. [1] https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emer…

> There's a line of thought that if we're impressed with what we have, if it just gets bigger maybe eventually 'reasoning' will just emerge as a side effect. This is somewhat unclear and not really a strategy per se. A recent analysis revealed that training on code might be the reason GPT-3 acquired multi-step reasoning abilities. It doesn't do that without code. So it looks like reasoning is emerging as a side effect of code. (section 3, long article) https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emer…

> There's a line of thought that if we're impressed with what we have, if it just gets bigger maybe eventually 'reasoning' will just emerge as a side-effect. This is somewhat unclear and not really a strategy per se. It's kind of like saying Moore's Law will get us to quantum computers. It's not clear that what we want is a mere scale-up of what we have. Reasoning ability really does seem to emerge from scale: https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emer…

Similar question, tho by my read, chain-of-thought seems to be standalone and the primary breakthrough. I asked the author[1]. I've worked in AI&ML since the 90s and both of these do seem much more intuitively promising than simply making larger nets. This seems borne out by the discussion in his previous post, that in Oct '21 OpenAI ran GPT-3 on it, solving 35% and they estimated: "it appears likely that the 175 [billion link neural network] model would require at least two additional orders of magnitude of training data to reach an 80% solve rate.” But instead 85% was achieved within a year using chain-of-thought, not a larger model.[2] [1]https://twitter.com/PMayrgundter/status/1603224294124920832 [2]https://yaofu.notion.site/A-Closer-Look-at-Large-Language-Models-Emerg…

There are two camps, evident in this thread. one camp is 'its just a statistical model, it cant possibly know these things' The other camp (that I'm in) sees that we might be onto something. We humans are obviously just more than a statistical model, but nonetheless learning words and how they fit together is a big part of who we are. With LLMs we have our first glimpse of 'emergent' behaviour from simple systems scaled massively. Whats are we if not a simple system scaled massively. Check these links out: Evidence that LLMs form internal models of what they learn about: https://thegradient.pub/othello/ Evidence that training LLMs on code actually made them better at complex reasoning: https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emer… John Carmack: https://dallasinnovates.com/exclusive-qa-john-carmacks-different-path-… I think that, almost certainly, the tools that we’ve got from deep learning in this last decade—we’ll be able to ride those to artificial general intelligence. A lot of the argument comes down to semantics about knowing and thinking. "An LLM can't think and a submarine cant swim"

For all models smaller than 62B, direct prompting outperforms CoT. The first model where CoT outperforms direct prompting is Flan-cont-PaLM 62B on BBH. For 540B models, there are more settings where CoT outperforms direct prompting, but not all of them. Also, the number can be smaller than 540B. In Suzgun et. al. 2022, the authors show that the 175B InstructGPT and 175B Codex also have better CoT performance than direct prompting. Combining all the results, we get the two numbers 62B and 175B. So yes indeed, to enter the game of scale you do need a ticket to larger models than average. However, there are also other large models like OPT, BLOOM, and the first version of GPT-3. They all have 175B, yet their CoT performance is significantly worse, or even cannot do CoT. source: https://yaofu.notion.site/A-Closer-Look-at-Large-Language-Models-Emerg…