Together AI Blog

o1 is better but deepseek r1 on https://together.xyz is also an option for something the Together ads say is 9x cheaper. Why does this relate? For ai, I would suggest picking the best model and I can run 9x the amount of work for the same price for about the same performance. Being able to see the reasoning is life changing for things.

In HuggingChat (https://huggingface.co/chat) you can test open models for free and even test specific providers. From there I collected the following US providers currently serving GLM 5.2: - Together (https://www.together.ai/models) - Fireworks (https://fireworks.ai/models) - Featherless (https://featherless.ai/models)

"Together with architectural refinements, our Mamba-3 model achieves significant gains across retrieval, state-tracking, and downstream language modeling tasks. At the 1.5B scale, Mamba-3 improves average downstream accuracy by 0.6 percentage points compared to the next best model (Gated DeltaNet), with Mamba-3's MIMO variant further improving accuracy by another 1.2 points for a total 1.8 point gain. Across state-size experiments, Mamba-3 achieves comparable perplexity to Mamba-2 despite using half of its predecessor's state size. Our evaluations demonstrate Mamba-3's ability to advance the performance-efficiency Pareto frontier." It's apparently optimised for inference efficiency Additionally: https://www.together.ai/blog/mamba-3 "Mamba-3 SISO beats Mamba-2, Gated DeltaNet, and even Llama-3.2-1B (Transformer) on prefill+decode latency across all sequence lengths at the 1.5B scale."

Asked Gemini about that: "are there efforts to train big LLM in a distributed fashion à la seti@home ? " answer was really interesing: - https://github.com/PrimeIntellect-ai/prime - https://www.together.ai/

Here's one I've used: * https://www.together.ai/ Here are all the models: * https://docs.together.ai/docs/chat-models * https://docs.together.ai/docs/language-and-code-models

Thanks, I need to spend some time digging into OpenRouter. The main requirement would be reliable function calling and JSON, since Plandex relies heavily on that. I'm also expecting to need some model-specific prompts, considering how much prompt iteration was needed to get things behaving how I wanted on OpenAI. I've also looked at Together (https://www.together.ai/) for this purpose. Can anyone speak to the differences between OpenRouter and Together?

I think we'll start seeing a lot more services like https://www.together.ai soon. Having open-weight models better than gpt-3.5 will drive a lot of competition on the LLM infra.

[Author] TL;DR OS LLM models are coming. Dolly's not that great -- I've hit lots of issues using it to be honest . MosaicML has a nice commercially usable model here: https://www.mosaicml.com/blog/mpt-7b I think they're one of the leading ones (bias: they're kinda competitors to my employer Anyscale, but you gotta say something's good when it is). Red Pajama are leading an effort to build a fully open source model similar to LLaMa. https://www.together.xyz/blog/redpajama

Looks awesome! Been doing a lot of OCR recently, and love the addition to the space. The reigning champion in the PDF -> Markdown space (AFAIK) is Facebook's Nougat[1], and I'm excited to hook this up to DSPy and see which works better for philosophy books. This repo links the Zerox[2] project by some startup, which also looks awesome, and certainly more smoothly advertised than Nougat. Would love corrections/advice from any actual experts passing by this comment section :) That said, I have a few questions if OP/anyone knows the answers: 1. What is Together.ai, and is this model OSS? Their website sells them as a hosting service, and the "Custom Models" page[3] seems to be about custom finetuning, not, like, training new proprietary models in-house. They might have a HuggingFace profile but it's hard to tell if it's them https://huggingface.co/TogetherAI 2. The GitHub says "hosted demo", but the hosting part is just the tiny (clean!) WebGUI, yes? It's implied that this functionality is and will always be available only through API calls? P.S. The header links are broken on my desktop browser -- no onClick triggered [1] https://facebookresearch.github.io/nougat/ [2] https://github.com/getomni-ai/zerox [3] https://www.together.ai/products#custom-models

Their models are much smaller: 1T vs 5T for the frontier models. 1T is Sonnet/Google Flash size, not Opus size. The $0.87/M tokens price for Mimo Pro is probably subsidized. Mimo models aren't widely available on western providers, but Kimi and Deepseek are similar sizes and cost about the same to run. They are priced $3-$4/M tokens (which is right were Google's very confused range of Flash models are priced at: between $0.40/M tokens and $9/M tokens depending on exactly which model - and you don't want the $9 one!). Anthropic overprices Sonnet (probably because of their capacity issues). GPT 5.4 mini is $4.50/M tokens. https://docs.fireworks.ai/serverless/pricing https://www.together.ai/pricing

The Usefulness Gap in Proof-of-Useful-Work: An Empirical Study of Pearl’s cuPOW Protocol https://arxiv.org/html/2606.04819v1 All devs accounts anonymous https://github.com/pearl-research-labs/pearl https://www.together.ai/blog/together-ai-partners-with-pearl-research-… https://huggingface.co/pearl-ai/Gemma-4-31B-it-pearl

Orders of magnitude seems a bit ambitious. The implementation from the DeepMind paper achieved a 2-2.5X https://arxiv.org/pdf/2302.01318 and most of the tests I've seen [1][2] have been similar, but there are different variations (Medusa, Ouroboros, etc) that can do better/be combined. Recently Together.ai published SpecExec, a SD variant which did claim to get a 10-18X speedups: https://www.together.ai/blog/specexec [1] https://www.reddit.com/r/LocalLLaMA/comments/17h4rqz/speculative_decod… [2] https://arxiv.org/pdf/2402.01528v3

You mean like this? https://www.together.ai/blog/together-moa

i havent seen anyone mention this yet so i'll be the first - what is the comparison vs StripedHyena? https://www.together.ai/blog/stripedhyena-7b

Piece with less detail than the source linked from the article: https://www.together.ai/blog/stripedhyena-7b

Memory is part of compute. I have no idea why you've separated them. >How exactly are you going to find enough data to feed these monstrous petabyte scale models? You won't! Your models will stay small and become cheap over time! People have this idea that we're stretching the limit of tokens to feed to models. Not even close. This is a 30T token dataset https://www.together.ai/blog/redpajama-data-v2 It's just web scrapped which means all the writing largely inaccessible on the web (swaths of fiction, textbooks, papers etc) and in books are not a part of it. It doesn't contain content in some of the most written languages on earth (e.g Chinese. You can easily get trilions of tokens of Chinese alone)

> What about alternatives? Custom models? Apart from the Big 3 (in no particular order): - https://together.ai/ - https://replicate.com/ - https://anyscale.com/ - https://baseten.co/ - https://modal.com/ - https://banana.dev/ - https://runpod.io/ - https://bentoml.com/ - https://brev.dev/ - https://octo.ai/ - https://cerebrium.ai/ ...

This is targeted towards GPU rental services like RunPod as well as API providers such as together AI. Together.ai is charging $0.90/1M tokens at 70B parameters. https://www.together.ai/pricing

the price is pretty low: https://www.together.ai/pricing 4B MODEL, PRICE 1K TOKENS: $0.0001 register with an email, test account has 25$ credit, python API as well, the model is smaller, but good to have some fun with the API integrated with other systems, haha...

There are actually a few ways to do this; and we have four: - `rps_doc_ml_wikiref_score`: a classifier that classifiers random webpage with Wiki references (used in Llama-1) - `ccnet_perplexity`: perplexity of an LM trained on Wikipedia (used in CCNet) - `rps_doc_ml_wikipedia_score`: classifier prediction for the document being a Wikipedia article - `rps_doc_wikipedia_importance`: Used in https://arxiv.org/abs/2302.03169 You can see the full table here: https://together.ai/blog/redpajama-data-v2

The same author Tri Dao, released FlashAttention 2 in July. https://together.ai/blog/tri-dao-flash-attention

Have you looked at using the long context (32K) version of the Llama v2 7B released by Together AI? https://together.ai/blog/llama-2-7b-32k

These foundational models have already been created https://www.together.xyz/blog/redpajama . Government backing has already been used to create a fully open source model. The effort has been multi-national. Government licenses aren't currently required to run one. "An award of computer time was provided by the INCITE program. This research also used resources of the Oak Ridge Leadership Computing Facility (OLCF), which is a DOE Office of Science User Facility supported under Contract DE-AC05-00OR22725."

Big Tech is built on open-source handouts. And open-source being built on Big Tech's research and models doesn't mean it relies on them. See: the many papers published by Standford and co. which rival OpenAI's research, RedPajama (https://www.together.xyz/blog/redpajama) which rivals LLaMa. And these are built with open alternatives, there'd be more motivation without. Big Tech has the advantage of compute and resources, but it's probable that open-source has the advantage of research. Since companies may have a few very talented individuals, but open-source has way more people and includes academia.

Many of the currently ongoing reproductions of LLaMa for starters. (see: Red Pajama[0]). Or any of the OpenAssistant affiliated projects. [0] - https://www.together.xyz/blog/redpajama

Because it's fun :-) And effort to bring up a could-be-commercial version is on going. https://www.together.xyz/blog/redpajama

Yeah, the synchronous updates are a big deal. To get an idea of how much bandwidth this typically takes, Oracle has 1600gbps per node of interconnect with latency very low, guessing in the tens of microseconds. A really good home connection might have 1gbps of interconnect with latency in the tens of milliseconds. The big question is whether we really need all this interconnect -- GPT-JT[1] is a very promising step in this direction. The idea is that we just drop most of the gradient updates and it still works well. Unclear whether this will take off generally -- if it does it would be a huge deal, because 1600gbps of interconnect is very expensive. [1]: https://www.together.xyz/blog/releasing-v1-of-gpt-jt-powered-by-open-s…

Its selling points are the privacy model of its hosting, and that ot is using an open source engine. The engine is OpenChatKit from Together: https://www.together.xyz/blog/openchatkit

OpenChatKit released by Together is a first open-source ChatGPT alternative. OpenChatKit is a 20B chat-GPT model! The model is an instruction-tuned large language model, fine-tuned for chat from EleutherAI GPT-NeoX-20B with over 43 million instructions under an Apache-2.0 license! You can try it for free on Hugging Face. Demo: https://huggingface.co/spaces/togethercomputer/OpenChatKit Model: https://huggingface.co/togethercomputer/GPT-NeoXT-Chat-Base-20B Announcement: https://www.together.xyz/blog/openchatkit #opensourcesoftware #machinelearning #ai #chatgpt

K3 is highly competitive. They're essentially tied here: > https://www.together.ai/blog/kimi-k3-vs-claude-fable-5-on-deepswe-cost… And they're very very close here: > https://artificialanalysis.ai/models And that's if you don't take cost and guardrails into consideration. If you do, K3 beats Fable hands down. That aside, I'm not an insider, and I can't really say when K3 completed training. What we do know is that it released to the public in mid-July, about a month ago. Fable 5's initial botched release was in mid-June, and it was re-released in early July. > No one is going to release highly powerful near-future models for the same reason no one releases how to make physically compact but megaton yield fusion devices So there's essentially an open Mythos-class model right now. Near-Mythos-class if you really want to quibble. Maybe past some threshold such models won't release to the public at all, or will release only in closed form, but we cannot predict when that threshold will be crossed. Fortunately for the world, it remains politically advantageous for China to support open model development. It's a winning strategy, especially in the long-term. Besides, I don't think a Uyghur Underground would be able to acquire the hardware to run a K3-class model, to say nothing of a Skynet-class AI.