Hamel Husain

This is a technique called LLM-as-Judge. Well-studied at this point. Here’s a good intro: https://www.evidentlyai.com/llm-guide/llm-as-a-judge I recommend reading Hamel.dev posts. Here’s an example: https://hamel.dev/blog/posts/evals/

This is so hard! I don't yet have a great solution for this myself, but I've been collecting notes about this on my "evals" tag for a while: https://simonwillison.net/tags/evals/ The best writing I've seen about this is from Hamel Husain - https://hamel.dev/blog/posts/llm-judge/ and https://hamel.dev/blog/posts/evals-faq/ are both excellent.

If you want good up to date resources on the applied side I’d recommend checking out https://hamel.dev/notes/

This is the best comparison I've found that benchmarks the current OSS inference solutions: https://hamel.dev/notes/llm/inference/03_inference.html IME the streaming API in text-generation-inference works fine in production. (Though some of the other solutions may be better). I've used it with Starcoder (15B) and the time-to-first-token / tokens per second all seem quite reasonable out of the box.

About .ml The first demo Jeremy put out was called "Build Applications For LLMs in Python",¹ as part of the "Mastering LLMs" conference by Hamel Husain and Dan Becker.² (You can see a few PoC demos by the end of that video when Johno takes over, it looks a lot like what Gradio or Streamlit can do). So I think your .ml angle is definitely part of the original ethos of FastHTML (which isn't surprising coming from the founder of fast.ai & answer.ai, among other things). The FastHTML team explicitly recommends would-be contributors to consider making reusable components, the likes of Gradio's, to facilitate all the things notably relating to AI workflows. ---- About WordPress & CMS That part is admittedly much larger in scope. I'd expect it to rise in correlation with the success of FastHTML itself in the Python web ecosystem writ large (beyond data / AI) but no sooner—unless someone makes a killer case for a FastHTML-based Python CMS that becomes a driver of popularity, but that's admittedly a much taller and wider order than 'simply' becoming the go-to #1 Python/ML prototype-to-market-at-scale one-stop shop. I mean, just that is huge, and yet nowhere near WordPress. But tbh, I really like your idea, and I think it may eventually prove true, having used FastHTML first-hand for a few weeks now (and web dev being far from my turf). The fact is can ship with FastHTML, fast & well-behaved web apps, more than I ever could. If I ever get the time I'll play a bit to see what a…

> Can anyone direct me towards how to ... make one? https://hamel.dev/blog/posts/evals/ > What are "obvious" things that are important to get right - temperature set to 0? At least ~10 or 20 attempts at the same problem for each llm? LLMs are actually pretty deterministic, so there is no need to do more than one attempt with the exact same data. > Finally, any known/commonly used frameworks to do this, or any tooling that can call different LLMs would be enough? https://github.com/vercel/ai https://github.com/mattpocock/evalite

Here's another article by the same author with more background on AI Evals: https://hamel.dev/blog/posts/evals/ I've appreciated Hamel's thinking on this topic.

You need evals. I found this post extremely helpful in building out a set of evals for my AI product: https://hamel.dev/blog/posts/evals/

Lots of people can get impressive demos up and running, but if you want to run AI products in production, you're going to have to do system evals. System evals make sure your product is doing what it says on the box with unquantifiable qualities. We wrote a zine on system evals without jargon: https://forestfriends.tech Eugene Yan has written extensively on it https://eugeneyan.com/writing/evals/ Hamel has as well. https://hamel.dev/blog/posts/evals/

LLM evals = unit tests if your outputs are consumer facing, might want to red team too this is good for thinking about how and why of evals https://hamel.dev/blog/posts/evals for tooling i found promptfoo to be lightweight and easy to get started.

This is a good place to start: * https://hamel.dev/blog/posts/evals/#level-1-unit-tests And more broadly: * https://applied-llms.org/

here's an awesome post on the landscape https://hamel.dev/blog/posts/prompt/

Hamel Husain hasn't done testing vs llama.cpp, but this still might be of interest (includes mlc which is roughly in line w/ llama.cpp batch=1 perf): https://hamel.dev/notes/llm/inference/03_inference.html He has benchmarks on an A6000 which should be roughly in line w/ a 3090 if you want to compare to my numbers (I test mlc as well, although my 3090 results are slower since I'm testing a llama2-7b @ 4K context and mlc currently slows down significantly w/ longer context): https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYpb63e1ZR3ae…

That's a big topic :D First thing about building AI or LLM powered apps is that 90% of the time you don't need to train models from scratch. LLMs are models that have the world's knowledge, so you can "train them" by prompting them and giving them the right context. Where to start? I think the first thing is to play around with the OpenAI playground to get an idea of what's possible with prompts. You'll find tons of frameworks out there: LangChain, LlamaIndex, Agno, and the list goes on. But in my opinion, if you have a use case in mind, start very very simple. Almost a single prompt using the OpenAI SDK or compatible SDK like LiteLLM. The core challenge isn't technical complexity, it's the stochastic nature of LLMs. Your prompt works perfectly on one example, breaks on another. You fix it for the second case, and it stops working for the first. The whole trick of building an LLM powered application is finding how to deal with this in an iterative approach. You need to have a workflow that allows you to iterate quickly. For this, start very simple and then add complexity. Don't use frameworks at the beginning. I think Hamel's approach of error analysis (https://hamel.dev/notes/llm/officehours/erroranalysis.html) is a very good starting point. The idea is that you create a prompt, run it against a test set, then identify clusters of error types and systematically fix them. What tools do you need? In the beginning, just a playground and a Jupyter Notebook. At some point…

- open-source vs closed fork of vscode - transparent assistant panel vs opaque composer. you control your own prompts (cf. [0]) - in Zed the assistant panel is "just another editor", which means you can inline-assist when writing prompts. super underrated feature imo - Zed's assistant is pretty hackable as well, you can add slash commands via native Zed extensions [1] or non-native, language-agnostic Context Servers [2] - Zed's /workflow is analogous to Cursor's composer. to be honest it's not quite as good yet, however it's only ~1 week old. we'll catch up in no time :) - native rust vs electron slop. Zed itself is one of the larger Rust projects out there [3], can be hard to work with in VS Code/Cursor, but speedy in Zed itself :) [0]: https://hamel.dev/blog/posts/prompt/ [1]: https://zed.dev/docs/extensions/slash-commands [2]: https://zed.dev/docs/assistant/context-servers [3]: https://blog.rust-lang.org/inside-rust/2024/08/15/this-development-cyc…