Gorilla Blog

Humans of AI Podcast #7 An amazing conversation with Shishir Patil the creator of the Gorilla LLM, a large language model specifically trained to use APIs! Shishir is currently a 5th year PhD student at the University of California, Berkeley whose work broadly covers ML-Systems, LLMs, Edge-ML, and Sky computing. Definitely give the episode a listen to hear Shishir's story. And to read more about #GorillaLLM, check out the project page! https://gorilla.cs.berkeley.edu

> This is a very strange article considering that Llama, the mother of all open-weight models, has led to anything but success for Meta. I see this as a fault with Meta's models, not a fault with the concept of open weights. The Llama family just aren't very useful. They make flowery prose but they're terrible at tool calling [1][2][3] so there just isn't much I can actually accomplish with them. [1] https://gorilla.cs.berkeley.edu/leaderboard.html [2] https://benchlm.ai/best/tool-use [3] https://benchlm.ai/llm-agent-benchmarks

> GLM-4.6 is great value but still not solid enough for tool calls, not that fast, etc. so if you can afford something more reliable I'd go for that, but encouraging. Funny you should say that, because while it is a large model the GLM 4.5 is at the top of Berkley's Function Calling Leaderboard [0] and has one of the lowest costs. Can't comment on speed compared to those smaller models, but the Air version of 4.5 is similarly highly-ranked. [0]https://gorilla.cs.berkeley.edu/leaderboard.html

The best ones so far are fine-tunes. But I agree those numbers aren't great and we haven't figured out tool-calling yet https://gorilla.cs.berkeley.edu/leaderboard.html

Hey folks, I'm on the Gemma team, we released new model(s) just recently, and I saw many questions here about function calling. We just published the docs to detail this more. In short Gemma3's prompted instruction following is quite good for the larger models and that's how you use the feature. You don't need to take our word for it! We were waiting for an external and independent validation from the Berkeley team, and they just published their results. You can use their metrics to get a rough sense of performance, and of course try it out yourself in AIstudio or locally with your own prompts. https://gorilla.cs.berkeley.edu/leaderboard.html Hope you all enjoy the models!

I've been using LMSYS as the best available high level "these are models to pay the most attention to" benchmark - I've not seen anything else that's nearly as useful on that front. The other one I consult occasionally is https://gorilla.cs.berkeley.edu/leaderboard.html - the Berkeley Function-Calling Leaderboard - because function calling is a really useful characteristic of smaller models. I believe the best ways to evaluate these models is with your own set of task-specific evals for your own project, but these are REALLY difficult to construct. I still don't have a good solution there for my own work.

Do you leverage Gorilla (https://gorilla.cs.berkeley.edu/) at all? If not, perhaps consider if it would solve some pain for you. https://news.ycombinator.com/item?id=36073241

- "Are You Human" https://arxiv.org/pdf/2410.09569 is designed to be directly on target, i.e. cross cutting set of questions that are easy for humans, but challenging for LLMs, Instead of one type of visual puzzle. Much better than ARC for the purpose you're looking for. - SimpleBench https://simple-bench.com/ (similar to above; great landing page w/scores that show human / ai gap) - PIQA (physical question answering, i.e. "how do i get a yolk out of a water bottle", common favorite of local llm enthusiasts in /r/localllama https://paperswithcode.com/dataset/piqa - Berkeley Function-Calling (I prefer https://gorilla.cs.berkeley.edu/leaderboard.html) AI search googled "llm benchmarks challenging for ai easy for humans", and "language model benchmarks that humans excel at but ai struggles with", and "tasks that are easy for humans but difficult for natural language ai". It also mentioned Moravec's Paradox is a known framing of this concept, started going down that rabbit hole because the resources were fascinating, but, had to hold back and submit this reply first. :)

The Berkeley Function Calling Leaderboard [1] might be of interest to you. As of now, it looks like Hammer2.1-3b is the strongest model under 7 billion parameters. Its overall score is ~82% of GPT-4o's. There is also Hammer2.1-1.5b at 1.5 billion parameters that is ~76% of GPT-4o. [1] https://gorilla.cs.berkeley.edu/leaderboard.html

this is one example https://gorilla.cs.berkeley.edu/leaderboard.html

The Berkeley Function-Calling Leaderboard tracks function calling/structured data performance from multiple models: https://gorilla.cs.berkeley.edu/leaderboard.html Llama isn't on there but a few finetunes of it (Hermes) are OSS.

Author of Gorilla Execution Engine (GoEx) here! The key idea behind GoEx: 1. "Post-facto validation" for assessing LLM actions after execution. 2. "Undo" button and “Damage confinement” abstractions to manage unintended actions & risks! GoEx (code below) inherently supports Gmail/Slack/Dropbox/Spotify/Github, try it out to build your own LLM agents! Blog: https://gorilla.cs.berkeley.edu/blogs/10_gorilla_exec_engine.html Code: https://github.com/ShishirPatil/gorilla/tree/main/goex

From [1]: > We demonstrate that our RAG approach trains the model to perform better RAG on the set of documents it is trained on i.e., in-domain. By removing the oracle documents in some instances of the training data, we are compelling the model to memorize domain-knowledge. What if you wanted to train it to say that it didn't find the answer? [1] https://gorilla.cs.berkeley.edu/blogs/9_raft.html

It's "Retrieval Augmented Fine Tuning". The related blog post is interesting: https://gorilla.cs.berkeley.edu/blogs/9_raft.html

"Gorilla: Large Language Model Connected with Massive APIs" (2023) https://gorilla.cs.berkeley.edu/ : > Gorilla enables LLMs to use tools by invoking APIs. Given a natural language query, Gorilla comes up with the semantically- and syntactically- correct API to invoke. With Gorilla, we are the first to demonstrate how to use LLMs to invoke 1,600+ (and growing) API calls accurately while reducing hallucination. We also release APIBench, the largest collection of APIs, curated and easy to be trained on! Join us, as we try to expand the largest API store and teach LLMs how to write them! eval/: https://github.com/ShishirPatil/gorilla/tree/main/eval - "Gorilla: Large Language Model connected with massive APIs" (2023-05) https://news.ycombinator.com/item?id=36073241 - "Gorilla: Large Language Model Connected with APIs" (2023-06) https://news.ycombinator.com/item?id=36333290 - "Gorilla-CLI: LLMs for CLI including K8s/AWS/GCP/Azure/sed and 1500 APIs (github.com/gorilla-llm)" (2023-06) https://news.ycombinator.com/item?id=36524078

https://www.swebench.com https://swe-rebench.com https://livebench.ai/#/ https://eqbench.com/# https://contextarena.ai/?needles=8 https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long… https://artificialanalysis.ai/leaderboards/models https://gorilla.cs.berkeley.edu/leaderboard.html https://github.com/lechmazur/confabulations https://dubesor.de/benchtable https://help.kagi.com/kagi/ai/llm-benchmark.html https://huggingface.co/spaces/DontPlanToEnd/UGI-Leaderboard