Fin AI Research

The product was Freedom[1] and I believe they were using Fin[2] for their email support. - [^1]: https://freedom.to - [^2]: https://fin.ai

I think the most direct answer is that at scale, inference can be batched, so that processing many queries together in a parallel batch is more efficient than interactively dedicating a single GPU per user (like your home setup). If you want a survey of intermediate level engineering tricks, this post we wrote on the Fin AI blog might be interesting. (There's probably a level of proprietary techniques OpenAI etc have again beyond these): https://fin.ai/research/think-fast-reasoning-at-3ms-a-token/

I hear you -- you and other teams are capable of building internal versions that just work. I'm equally excited -- I've spent much of my career building janky internal versions of popular SaaS out of necessity since we didn't have the budget to buy. To be able to do a better job with less effort is enticing. But this is: a) a step-change that hasn't had a full year to bake; we should all anticipate the pain associated in the medium term after a few iterations, inevitable feature add-ons, etc. b) beside the point. Yes, many teams can also build internally, but it doesn't change the fact that others find value in outsourcing. Just because it's much easier, or rather __because__ it's easier to stand stuff up, it's imperative we prioritize what gets built. If [Anthropic](https://fin.ai/customers/anthropic) themselves are willing to vouch for the value-add, I think it's silly to suggest that teams with budget and higher priorities should trade the time and focus to roll their own.