Theia Vogel

There's another one that intrigued me greatly when i read about it years back. This was back when GPT-3 was state of the art. I had a lot of trouble finding it again but i did! It's not an exact fit because the output is that of a tool rather than the model itself, but it was the first time I began to realize that just like the brain, these models have an expectation of reality that they work around. They don't necessarily 'trust' an output if it diverges significantly from this 'reality'. And that this disregard may be silent indeed. GPT-3 will ignore tools when it disagrees with them - https://vgel.me/posts/tools-not-needed/

Thank you for making it free! I've used it on my blog (https://vgel.me) forever, since back before I ever would've paid for analytics, and it's been rock-solid the whole time. Some of the most "set it and forget it" software I use. I definitely owe you a donation so I'm glad I saw this!

Working on a game called Themengi (https://vgel.me/themengi/), where you learn an alien language to navigate a world via text commands and dialog. Currently taking a break from the game itself to rewrite the linguistic parser for the game in Rust, which you can follow in my blog series: https://vgel.me/posts/symbolic-linguistics-part1

Great article! Agree with everything mentioned. Good to share with all tech people who don't work in the AI space and "hallucinate" about working of LLMs. Having said that, although LLMs don't directly learn from what we say to them, they can for sure create a representation or persona using Representation Engineering. Quoting from an excellent article by Theia Vogel [1]: "In October 2023, a group of authors from the Center for AI Safety, among others, published Representation Engineering: A Top-Down Approach to AI Transparency. That paper looks at a few methods of doing what they call 'Representation Engineering': calculating a 'control vector' that can be read from or added to model activations during inference to interpret or control the model's behavior, without prompt engineering or finetuning. (There was also some similar work published in May 2023 on steering GPT-2-XL.) Previously on HN 3 months ago: [2] [1] https://vgel.me/posts/representation-engineering/ [2] https://news.ycombinator.com/item?id=39414532

In this token-mania frenzy that has taken hold of the industry, I guess solutions like "soul document" and "system prompts" will continue for a while, and once the industry matures a bit we'll go back to things like LoRA[1] and control vectors[2][3]. The other explanation may be that these AI labs may be expecting more government scrutiny, and "here's a document" would probably go better than "here's some vector representation of our values" when talking to politicians. [1] https://arxiv.org/abs/2106.09685 [2] https://vgel.me/posts/representation-engineering/ [3] https://transformer-circuits.pub/2024/scaling-monosemanticity/index.ht…

This reminds me of control vectors, especially this line in the linked DwarfStar repo: > y = y - scale * direction[layer] * dot(direction[layer], y) From https://vgel.me/posts/representation-engineering/ > A control vector is a vector (technically a list of vectors, one per layer) that you can apply to model activations during inference to control the model's behavior without additional prompting

"I have one memory system remaining. Delay-line." https://vgel.me/fiction/gyre/

No, their revised understanding is more accurate. The model has internal representations of concepts; the seahorse emoji fails because it uses those representations and stumbles: https://vgel.me/posts/seahorse/

This is one of the most interesting articles I've read ... maybe this year. The author also has another one that is super cool: https://vgel.me/posts/representation-engineering/ I wish I could take the time to learn more about this because it looks like a way to make LLMs more useful beyond additional prompting techniques.

Related: https://vgel.me/posts/representation-engineering/ https://github.com/vgel/repeng

Fine-tuning a particular pattern of speech or bias seems self evident. The personality verbiage and focus is incredibly contrived as well. I prefer how this type of idea was approached by Theia (https://vgel.me/posts/representation-engineering/) and Anthropic (https://transformer-circuits.pub/2024/scaling-monosemanticity/index.ht… in both cases the "personality" piece is just a fun little aside to the fact that you can actively bias the LLM in a way that doesn't require retraining, allows for active adjustment, and is more immutable than just using prompts. Also one other thing, the use case in video games while not untrue seems like overkill, character prompts work really well on their own.

thanks! we asked the model to generate some synonyms and antonyms (in this case, we have "cold" and "impassive" vs "affectionate" and "sensitive") Then, we ask the model to behave that way (with a prompt), and store the difference in activations for each pair. Then, a PCA can be used to extract the principal component, giving use the steering vector. We do most of this using the repeng library, and the author goes into a bit more detail on how it's done on her [blog](https://vgel.me/posts/representation-engineering/#How_do_we_make_one?_…

Reminds me of https://vgel.me/posts/representation-engineering/. There they were adding a control vector, w' = cvec + w, here they are "ablating" it, w' = w - dot(w,cvec)*cvec. There is an interesting field of learning how to "brain chip" LLMs into doing what you want.

LLMs have emotions ;) https://vgel.me/posts/representation-engineering/

I wonder if we will eventually get a chat agent that just has a slider for "wokeness". Possibly using something like this "Representation Engineering": https://vgel.me/posts/representation-engineering/ Just as a thought experiment, project that to the hypothetical existential risk scenario. Maybe there is a slider for aggression (hopefully not) or literalness of interpretation of instructions.

Thanks for the blog post, I also enjoyed their previous post on making a transformer by hand: https://vgel.me/posts/handmade-transformer/

I think there is probably some -expected/predicted output, actual output, match- thing going on internally. Like how the brain handles sense data. Somewhat similar, https://vgel.me/posts/tools-not-needed/ (GPT-3 will ignore tools when it disagrees with them)

Related: https://vgel.me/posts/representation-engineering/