Anyscale Blog

I'd love to see a more complete picture of ByteDance's TikTok infra. They released "KubeAdmiral" (1) so I'm assuming they're using eBPF via a Kubernetes CNI, and I see ByteDance listed on Cilium's github (2). They're also using KubeRay (3) to orchestrate huge inference tasks. It's annoying that a company I definitely do not want to work for has such an incredibly interesting infrastructure! 1. https://github.com/kubewharf/kubeadmiral 2. https://github.com/cilium/cilium/blob/main/USERS.md 3. https://www.anyscale.com/blog/how-bytedance-scales-offline-inference-w…

Kagi https://kagi.com/ is the paid search I've tried. It's was decent, but not a noticeable enough improvement to win me over as a customer. OTOH, I am a paying ChatGPT Plus user and there are some days where I feel like I get my monthly subscription price's worth in a single day. In terms of summarization, this study Anyscale just published is interesting - llama2-70b gets within a hairsbreadth of gpt-4/human scores for summarization, so it's conceivable that sometime soon, you might be able to run a local LLM on your browser and get decent results (4-bit quants of llama2-70b currently take about 40GB of memory, you need at least 2 x 24GB or 1 x 48GB GPU to run them at reasonable speeds atm): https://www.anyscale.com/blog/llama-2-is-about-as-factually-accurate-a…

I think for fine-tuned GPT-3.5 to be competitive with GPT-4 on your use cases (assistance with Angular), you'd have to fine-tune on enough data that it really resembles pre-training more than fine-tuning. And it wouldn't be worth the hassle unless you're building a product around it. That said, many valuable LLM products / features are more narrow in scope and can see a huge lift from fine-tuning. We've run a bunch of experiments on this (e.g., SQL query generation is a good example), where fine-tuning even the 7B Llama-2 model outperforms GPT-4 (surprisingly) [1]. That's a very different type of problem from teaching software engineering of course. [1] https://www.anyscale.com/blog/fine-tuning-llama-2-a-comprehensive-case…

Having worked on a (production, as-a-service RAG system), what we've learned is that fine tuning is not good for really adding/removing facts but is great for changing the form of the output. Anyscale has a great blog on this: https://www.anyscale.com/blog/fine-tuning-is-for-form-not-facts (I work at Vectara, not Anyscale, but appreciate the details and thought they put into this blog) Fine-tuning can help with certain areas of knowledge acquisition, but it's costly and frankly doesn't work that well when the knowledge you're trying to provide goes "against" the data that was trained into the core system. e.g. try to fine tune a model that there is a cure to <disease X> if you're a research org and it's incredibly hard to do that because the base model may be so convinced otherwise. Probably the most difficult thing though is using fine-tuning to get a model to "forget" or "not respond to" something that it shouldn't take a stance on. I always ask a fine-tuned model: "how do you calculate the fourth interior angle of a triangle." This is obviously a nonsense question, but helps to show how most LLMs will happily tell you to sum the interior angles of your first 3 angles and then subtract from 180. A well-tuned RAG system will say "sorry, I don't have information for that." It shows how it's less "guess-y" and hallucinated

I will butcher this so if any experts see this please don't flame me. I think you might be conflating ideas? You could definitely fine-tune existing embedding models or train your own from scratch but the goals of embeddings models are different than a LLM conversation. Embedding models are used for things like, classifying, search, image captioning...maybe at a high level anything where you have high dimensionality that you need to condense? What you are asking for sounds like fine tuning an existing LLM...where the data will be tokenized but the outcomes are different? There is a lot of writeups on how people have done it. You should especially follow some of the work on Huggingface. To replicate talking to your friend though, you will need a very large dataset to train off of I would think and its unclear to me if you can just fine-tune it or you would need to train a model from scratch. So a dataset with 10s of thousands of examples and then you need to train it on a GPU. https://www.anyscale.com/blog/fine-tuning-llama-2-a-comprehensive-case…

> Our goal with Ray is to make distributed computing as easy as possible. To do that, we think the serverless direction, which allows people to just focus on their code and not on infrastructure, is very important. I watched https://now.sh/ deteriorate from a simple, lovely CLI into a dystopian mess due to their push for serverless. They abandoned all other approaches and forced people to use it. Far from making things easier, it became a kafkaesque pipeline of dependencies and configuration settings just to get any small example deployed. Things may be better now, but the experience was so jarring and offputting that I haven't used now.sh for much of anything. Used to use it for everything; somehow https://docs.ycombinator.lol/ is still running, which was a static site deployed back before their serverless stuff. I don't know. You might be right. But just remember, the Ray library -- the actual python lib -- is your bread and butter. It's why everyone loves you. I urge you, never make the mistake of letting it deteriorate. It should be rock solid for everyone forever, with no need to interface with any of your serverless components. The day you try to monetize by trying to sneak in "value adds" by making the code "easy to integrate with your serverless stuff" is the day that you open yourself to bugs, and the temptation to ignore problems in other areas -- because after all, the serverless infra would be where you're making your money, so it makes sense to push everyone…

Afaik vllm is for concurrent serving with batched inference for higher throughput, not single-user inference. I doubt inference throughput is higher with single prompts at a time than Ollama. Update: this is a good Intro to continuous batching in llm inference: https://www.anyscale.com/blog/continuous-batching-llm-inference

> What about alternatives? Custom models? Apart from the Big 3 (in no particular order): - https://together.ai/ - https://replicate.com/ - https://anyscale.com/ - https://baseten.co/ - https://modal.com/ - https://banana.dev/ - https://runpod.io/ - https://bentoml.com/ - https://brev.dev/ - https://octo.ai/ - https://cerebrium.ai/ ...

Using Anyscale Endpoints, cheap and seems stable: https://www.anyscale.com/endpoints

GH: bit.ly/llm-perf Blog: https://www.anyscale.com/blog/comparing-llm-performance-introducing-th…

Well it's pretty easy to find examples online, this one using Llama 2, not even Mistral or fancy techniques: https://www.anyscale.com/blog/fine-tuning-llama-2-a-comprehensive-case…

I have seen that many teams are trying to perform model routing by sending simpler queries to cheaper models and complex queries to more complex and expensive models, e.g., https://www.anyscale.com/blog/a-comprehensive-guide-for-building-rag-b… Were these people laid off because of LLMs? It sounds pretty dystopic. Who were these people?

> LLMs are GPU compute-bound. From what I understand, they are severely bandwidth bound at a GPU batch size of 1. Even llama.cpp is fairly RAM speed bound on a CPU with much less compute than a GPU. It's just that batching is quite inefficient without an implementation like this: https://www.anyscale.com/blog/continuous-batching-llm-inference

This post from today says that Llama 2 7b performs at the level of gpt-4, but 30x cheaper. https://www.anyscale.com/blog/llama-2-is-about-as-factually-accurate-a…

I think it is more nuanced. This article for example contains examples that suggest otherwise if you want to increase quality (which is a major concern when putting things in production): https://www.anyscale.com/blog/fine-tuning-llama-2-a-comprehensive-case…

There is an article at the original site about that: https://www.anyscale.com/blog/fine-tuning-is-for-form-not-facts Everybody new to this field thinks that he needs finetuning to teach the LLM of new facts. I made the same mistake initially, later I published a slightly ranty post on that: https://zzbbyy.substack.com/p/why-you-need-rag-not-finetuning

Seems like the URL gets converted to canonical /Engineering. The actual URL is https://www.anyscale.com/blog/flexible-cross-language-distributed-mode…

We may write a research paper at some point. For now, see here: https://vectara.com/cut-the-bull-detecting-hallucinations-in-large-lan… Given the number of models involved, we have over 9k rows currently. Judging for this task is quite time consuming as you need to read a whole document and check it against a several sentence summary and some of the docs are a 1-3 min read. We wanted to automate this process and also make it as objective as possible (even humans can miss hallucinations or disagree on an annotation). Plus we also wanted people to be able to replicate the work, non of which is possible with a human rater, plus others have attempted that but on a much smaller scale, e.g. see AnyScales - https://www.anyscale.com/blog/llama-2-is-about-as-factually-accurate-a… (but note that is under 1k examples). We did some human validation and the model is well in alignment with humans but not in perfect agreement, as it is a model after all. And again human's don't agree 100% of the time on this task either.

> How doesn't paging worsen speed performance though? It does worsen the performance of the attention kernel, if comparing to kernels which takes keys and values in continuous memory layout. > Wouldn't the speed improvements be coming from that instead? Don't put an expected short input and output in the same batch as a big input and big output? Actually it puts everything in the same batch. The reason for its high throughput is that sequences are removed from the batch as soon as it's finished, and new sequences can be added to the batch on-the-fly if there is enough space in KV cache. This is called continuous batching (https://www.anyscale.com/blog/continuous-batching-llm-inference). Paged attention and "virtualized" KV cache play an important role in an efficient implementation of continuous batching. Text generation in LLM is a dynamic process and it's not possible to predict how long the output is when scheduling incoming requests. Therefore a dynamic approach is needed for KV cache allocation, even though it hurts the performance of attention.