Philipp Schmid

Pass^k and not Pass@k (see https://www.philschmid.de/agents-pass-at-k-pass-power-k). Would be a great twofer to see the code used to run the benchmarks as examples.

I've only just skimmed this blog post but if I'm reading correctly FunctionGemma can work just like what's intended here, a "contextless" tool router. Going one level up you as a developer have a choice how much context you want to provide to the model. Philipp Schmid wrote a good blog post about this, titling this "context engineering". I like his idea because instead of just blindly throwing stuff into a model's context window and hoping to get good performance, it encourages folks to think more about how what's going into the context in each turn. https://www.philschmid.de/context-engineering Similarly I think the blog post you linked has a similar sentiment. There's nuanced approaches that can yield better results if an engineering mindset is applied.

I'm working on Conductor https://github.com/skanga/Conductor Conductor is a LLM agnostic framework for building sophisticated AI applications using a subagent architecture. It provides a robust platform for orchestrating multiple specialized AI agents to accomplish complex tasks, with features like LLM-based planning, memory persistence, and dynamic tool use. It provides a robust and flexible platform for orchestrating multiple specialized AI agents to accomplish complex tasks. This project is inspired by the concepts outlined in "The Rise of Subagents" by Phil Schmid at https://www.philschmid.de/the-rise-of-subagents and it aims to provide a practical implementation of this powerful architectural pattern.

Gist is 1. Gather training data 2. Format it into JSONL or Hugging Face Dataset format 3. Use Axolotl or Hugging Face peft to fine-tune 4. Export model to GGUF or HF format 5. Serve via Ollama https://adithyask.medium.com/axolotl-is-all-you-need-331d5de1b7c0 https://www.philschmid.de/fine-tune-llms-in-2025 https://blog.devgenius.io/complete-guide-to-model-fine-tuning-from-hug…

If you are looking for something that actually explains most concepts behind it, Karpathy’s series will teach you what you want to know. (Mentioned elsewhere) If you are looking for command line tools to fine tune and evaluate models on known datasets, this article is a good take! https://www.philschmid.de/fine-tune-llms-in-2025

it can be almost anything like your HN comments or some corporate wiki, then get colab pro 10$ month or some juicy gaming machine and fine-tune that using eg this tutorial https://www.philschmid.de/instruction-tune-llama-2 but https://www.reddit.com/r/LocalLLaMA/ is full of different fine tuned models.

What's surprising about this is how sparsely defined the rewards are. Even if the model learns the formatting reward, if it never chances upon a solution, there isn't any feedback/reward to push it to learn to solve the game more often. So what are the chances of randomly guessing a solution? The toy Countdown dataset here has 3 to 4 numbers, which are combined with 4 symbols (+, -, x, ÷). With 3 numbers there are 3! * 4^3 = 384 possible symbol combinations, with 4 there are 6144. By the tensorboard log [0], even after just 10 learning steps, the model already has a success rate just below 10%. If we make the simplifying assumption that the model hasn't learned anything in 10 steps, then the probability of 1 (or more) success in 80 chances (8 generations are used per step), guessing randomly for a success rate of 1/384 on 3-number problems, is 1.9%. One interpretation is to take this as a p-value, and reject that the model's base success rate is completely random guessing - the base model already has slightly above chance success rate at solving the 3-number CountDown game. This aligns with my intuition - I suspect that with proper prompting, LLMs should be able to solve CountDown decently OK without any training. Though maybe not a 3B model? The model likely "parlays" its successes on 3 numbers to start to learn to solve 4 numbers. Or has it? The final learned ~50% success rate matches the frequency of 4-number problems in Jiayi Pan's CountDown dataset [1]. Phil does…

Use SageMaker: https://www.philschmid.de/sagemaker-llama-llm