MosaicML Blog

I would monitor and research each of these top models to determine which best fits your use case. https://lmsys.org/blog/2023-05-25-leaderboard/ https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://assets-global.website-files.com/61fd4eb76a8d78bc0676b47d/64547… https://www.mosaicml.com/blog/mpt-7b Also keep up to date with r/LocalLLaMA where new best open models are posted all the time.

[Author] TL;DR OS LLM models are coming. Dolly's not that great -- I've hit lots of issues using it to be honest . MosaicML has a nice commercially usable model here: https://www.mosaicml.com/blog/mpt-7b I think they're one of the leading ones (bias: they're kinda competitors to my employer Anyscale, but you gotta say something's good when it is). Red Pajama are leading an effort to build a fully open source model similar to LLaMa. https://www.together.xyz/blog/redpajama

>Why isn't there a framework that does all this automatically for you? Check MosaicML if it might help in your case. I haven’t tried myself but they’ve most customizations and speed up optimizations I came across in the recent times https://www.mosaicml.com/blog/supercharge-training-composer Also worth checking out their “training from scratch” blog posts. Training StableDiffusion: https://www.mosaicml.com/blog/training-stable-diffusion-from-scratch-c… Training GPT-3: https://www.mosaicml.com/blog/billion-parameter-gpt-training-made-easy

MosiacML, the latest and greatest training framework to hit the scene, is taking the machine learning world by storm. The code is top-notch, and the multiple optimizations that work right out of the box with HuggingFace models are a game-changer. It's no wonder that so many businesses are hopping on the MosiacML bandwagon - this framework is the real deal. Personally, I've been loving the MosiacML trainer for training our Bert Models. With the composer library, scaling training is a breeze - no more worrying about OOMs or streaming data from your cloud buckets. For more information, be sure to check out https://www.mosaicml.com/blog/composer-0-12-release. In the ever-evolving field of machine learning, staying relevant is key. Believe me, I've wasted hours debugging problems with other outdated frameworks. So if you want to stay ahead of the curve and produce top-notch models, MosiacML is the way to go.

Traditionally, training runs can "explode" and fail, but there are methods to incrementally back them up and resume when that happens, see https://www.mosaicml.com/blog/mpt-7b

https://pytorch.org/blog/experience-power-pytorch-2.0/ https://www.mosaicml.com/blog/amd-mi250

is this similar to the mosaicml amd MI250 vs nvidia A100 results but with consumer grade hardware? https://www.mosaicml.com/blog/amd-mi250 might be interesting to team up

Eh well, it is very close to just working. From "Training LLMs with AMD MI250 GPUs and MosaicML": > It all just works. No code changes were needed. https://www.mosaicml.com/blog/amd-mi250

SuperHOT: https://kaiokendev.github.io/til LoRA: https://news.ycombinator.com/item?id=35288015 The Salesforce Model (XGen): https://news.ycombinator.com/item?id=36514936 MPT (Mosaic): https://www.mosaicml.com/blog/mpt-30b

"MPT-30B...that is significantly more powerful than MPT-7B and outperforms the original GPT-3." Source: 1. Original announcement: https://www.mosaicml.com/blog/mpt-30b

AIUI it uses the Llama architecture, but not Facebook's Llama weights. It uses MPT-7B, which was trained from scratch: https://www.mosaicml.com/blog/mpt-7b

https://www.mosaicml.com/blog/mosaicbert I think an useful article for people who want to train BERT from scratch with 20$ (in this case by renting GPUs). This model has also an actual good GLUE score.

Open source LLMs are getting pretty good. The one mosaicml[0] just released is impressive and I expect a dozen more like it by the end of the year. 0 - https://www.mosaicml.com/blog/mpt-7b out Mosiac

>an individual or nonprofit can't do that. https://laion.ai/ https://www.mosaicml.com/blog/mpt-7b There's dozens if not hundreds more individuals and nonprofits doing these things.

> ~$1 million: Cost to train a 13 billion parameter model on 1.4 trillion tokens MosaicML claims they trained a 7 billion parameter on 1 trillion tokens with a budget of $200k. https://www.mosaicml.com/blog/mpt-7b Does training cost scale linearly with model size and token count? If so, that suggests a lower bound of $600k to train the 13 billion params model. (Still roughly the same magnitude)

An introduction and development guide for open-source LLM — MPT-7B Edit: See also https://www.mosaicml.com/blog/mpt-7b

Mosaic wrote about their new model here. https://www.mosaicml.com/blog/mpt-7b It was trained on 65k inputs and has decent performance working with 80k+ tokens.

https://www.mosaicml.com/blog/mpt-7b 65k+ context window, open source, open weights

There is plenty of less attention-grabbing work being done on "domain specific LLMs" like BioMedLM[0], Med-PaLM[1], BloombergGPT[2], etc. That reminds me - I saw a somewhat-clever acronym variant for LLM that communicated this the other day but it escapes me ATM... [0] - https://www.mosaicml.com/blog/introducing-pubmed-gpt [1] - https://cloud.google.com/blog/topics/healthcare-life-sciences/sharing-… [2] - https://dev.to/reaminated/thoughts-on-bloomberggpt-and-domain-specific…

Disclaimer: I work for MosaicML (MosaicML is the creator of the training platform used by Replit). Training these models from scratch on your domain specific data is not as expensive as one might think. We have provided some cost estimates in our blogs. https://www.mosaicml.com/blog/mosaicbert https://www.mosaicml.com/blog/training-stable-diffusion-from-scratch-c… https://www.mosaicml.com/blog/gpt-3-quality-for-500k

Author here. Ask me anything--happy to answer questions. Also, if you like this kind of work, you might like what I've been building for the past year: Composer [1]. It speeds up neural net training by a lot (e.g., 7x faster for ResNet-50) [2] and, in contrast to Bolt/MADDNESS, is polished, documented code you can get working in <5min. [1] https://github.com/mosaicml/composer [2] https://www.mosaicml.com/blog/mosaic-resnet

15 million GPU hours is interesting, llama 2 had about 1.7 million hours for the 70B (3.3M for all). Their GPU hours are on LUMI which is AMD MI250 GPU, which Mosaic reports is about 75% of the A100 (https://www.mosaicml.com/blog/amd-mi250) but AFAIK hasn't been tested at this kind of scale. so, validating if the MI250 in a cluster can effectively train a large scale LLM will be useful. not that it really matters to any other group, you won't get these in the cloud anytime soon. either way - say that they have about 10M A100 hours. if they use half their budget for the biggest model as FB did with Llama (rest on testing / smaller models as they scale), then they'll budget 5M A100 hours for the biggest model, or 3x llama-2's 1.7M A100 hours (2k A100 for 21 days). However, why wouldn't Llama-3 also be 3x bigger than Llama-2? FB's H100's are coming online; they apparently have 4k H100 in a cluster (https://www.nextplatform.com/2023/09/26/meta-platforms-is-determined-t… and each H100 is about 3x the speed of an A100 (https://www.mosaicml.com/blog/coreweave-nvidia-h100-part-1). So if they provided the same 'calendar time' of 21 days over a single cluster of 4096 H100, that would be about 6.2M A100 hours for a 70B model. there is an entirely separate question of, LLM's are functions of compute _and_ data, will they also find enough high quality _and_ kosher tokens in their low resource languages?

In the data center, I think AMD is a lot more viable than most people think. MosaicML recently did a test and were able to swap MI250s with A100s basically seamlessly, within a single training run even, and ran into no issues: https://www.mosaicml.com/blog/amd-mi250 If you have an officially supported card https://rocm.docs.amd.com/en/latest/release/gpu_os_support.html#suppor… and are using PyTorch, then you're pretty much good to go. Also, HIPify works pretty well these days. I think where most people have been getting into trouble is with trying to run with unsupported cards (eg, *ALL* of AMD's consumer cards), or wanting to run on Windows. This is obviously a huge fail on AMD's part since anyone who's tried to do anything with any of those consumer cards will just assume the data center cards are the same, but they're quite different. It doesn't help that I've never seen any CDNA2 card on sale/available in retail. How does AMD ever expect to get any adoption when no developers have hardware they can write code to? It's completely mental.

Pretty sure it's not "real" (model) context width. Another wide context model is MosaicML's MPT-7B-StoryWriter-65k+ which they are describing as having a context width of 65k, but then give a bit more detail to say they are using ALiBi - a type of positional encoding that allows longer contexts at inference time than training (i.e beyond the real context width of the model). For these types of "extended context" models to actually reason over inputs longer than the native context width of the model, I assume that there is indeed some sort of vector DB trickery - maybe paging thru the input to generate vector DB content, then using some type of Retrieval Augmented Generation (RAG) to process that using the extended contexts ? Maybe someone from Anthropic or MosaicMl could throw us a bone and give a bit more detail of how these are working ! https://www.mosaicml.com/blog/mpt-7b https://arxiv.org/abs/2005.11401

MosaicML (https://www.mosaicml.com/careers) | Full Time | San Francisco / Palo Alto / San Diego / New York | Onsite Role: Software Engineer - Cloud Platform (https://boards.greenhouse.io/mosaicml/jobs/4327398004) In this role, you will... - Design and build our SaaS products for efficient ML training - Ensure reliability, security, and scalability of our SaaS platform as we build out new features and onboard customers - Deploy technologies on top of Kubernetes, build workload-aware orchestration - Learn about the machine learning software necessary to optimize training performance across multiple scales of hardware MosaicML is making ML training more efficient by improving efficiency of neural network training with algorithmic methods that deliver speed, boost quality and reduce cost. We're well funded by Lux Capital, DCVC, and Playground Global to name a few. We're growing quick, and if you're interested in learning more please reach out to: shawn [at] mosaicml.com