Scale AI Blog

I read through all the comments and it is amazing that nobody is mentioning the fact that the reason why this 2020 post is coming up now is because of this: Scale recently laid off 200 full-time employees and terminated 500 contractor positions. The CEO, author of the post, went to work for Meta. https://scale.com/blog/scale-ai-announces-next-phase-of-company-evolut…

In 2021, Alex Wang of Scale (the author of the linked article) uncritically quoted the following from Eric Schmidt: "Eric states that the global growth of AI technology is a national security threat for the United States. In March, we believed we were ahead of China’s technological development in the AI sector, but they quickly showed us that we were wrong. By June, they’d demonstrated a model capable of producing human-like text that was comparable to OpenAI’s GPT-3 model in quality." https://scale.com/blog/Eric-Schmidt-Geopolitics-TransformX-Scale-AI As such, it's important to note that the author views the entire Chinese AI industry as a threat to the United States, and may believe that the growth of AI anywhere in the rest of the world is a threat to the United States.

Scale AI (https://scale.com/) | San Francisco, CA, MX, AR, or Remote, US | Full-Time | Software Engineer eCommerce Scale helps you deliver better value from your AI models with better data, better labels, and most recently, better AI infrastructure. Full details here: https://grnh.se/99333f9b5us or DM me @exploreml to learn more!

Scale AI | Product Security Engineer | TypeScript, Node, Python, AWS | Full-time | Hybrid in San Francisco, CA or NYC or Remote This is for a team that I'm working closely with and helping grow. We're hiring for somebody with a hybrid SWE and Security background to help scale up the team. It's currently one engineer who's got a ton on his plate, so the ideal person for this role is somebody who's interested in learning, digging in deep to fix issues (especially shipping PRs), and helping shape the future roadmap of the team. It is not an analyst role. The role is primarily targeted at mid-career folks with a few years of experience (2+ years minimum). Those that are Senior/Staff level folks, especially those with a strong SWE background that are curious about security, are encouraged to apply if this role resonates strongly. (I personally transitioned from SWE -> ProdSec, years ago, and several other folks on the broader Security team have non-Security backgrounds too.) Here is the role: https://scale.com/careers/4602047005 Feel free to apply there, email me with you resume (on my profile), or add me on LinkedIn[0]. I'll try my best to answer questions and reply to everybody, but sometimes there is so much inbound that it isn't possible. Thanks! 0: https://www.linkedin.com/in/freewortley/

Thanks for the pointer. We do agree that not all agentic systems should be multi-agent. Having said that, from our experience we see that for complex workflows e.g. customer support for enterprises, both quality and maintainability stands to gain when the system is decomposed into smaller scoped agents. We see a parallel of this in humans as well. For instance, when we call into customer support we get routed to different human agents based on our query. OpenAI says something similar in their recent guide on building agents [0]: "For many complex workflows, splitting up prompts and tools across multiple agents allows for improved performance and scalability. When your agents fail to follow complicated instructions or consistently select incorrect tools, you may need to further divide your system and introduce more distinct agents." A relevant benchmark here might be the Instruction Following benchmark: https://scale.com/leaderboard/multichallenge. Even the best reasoning models top out at ~60% accuracy on this. The options to improve accuracy then, are (a) either fine-tune a model on this task specific dataset, (b) or decompose the problem into smaller sub-problems (divide-and-conquer) - this is more practical and maintainable. [0] https://cdn.openai.com/business-guides-and-resources/a-practical-guide…

Scale | Backend/Full Stack and Frontend (Full-Time) | SF We label data for your favorite computer vision teams. Our mission is to accelerate the development of AI applications - we believe building a high quality labelled dataset is the biggest bottleneck to deploying supervised deep learning systems, so that's what we're tackling first. We’ve had phenomenal past few years, raised an $1B series C, and are looking to grow our team of 130 (35 engineers). We’re looking for engineers to work on projects ranging from making labelling more efficient via front-end work/ML work to launching completely new product lines. Frontend: https://jobs.lever.co/scaleai/0381f41e-7779-463e-a6c7-926fa21e4f90?lev… Backend: https://jobs.lever.co/scaleai/c1443865-f64e-4467-bfdc-8980533a2ad3?lev… Fullstack: https://jobs.lever.co/scaleai/868065d0-7139-42cf-999f-3f50d476d38e?lev… Learn more about us at: https://scale.com/blog

it has been pretty much a benchmark for memorization for a while. there is a paper on the subject somewhere. swe bench pro public is newer, but its not live, so it will get slowly memorized as well. the private dataset is more interesting, as are the results there: https://scale.com/leaderboard/swe_bench_pro_private

Let’s get a reality check. Current agent performances are no where near that kind of threat, and it’s uncertain when we get there. I think people continue bullshitting in this domain because they’re worried they have no moat, so they have to discourage via sophistry or bold claims. Don’t take my word for it, here are the stats https://scale.com/leaderboard For reference, 25% means getting 1 in every 4 question right. Secondly, if we ever do get to that point, people have lots of social mechanisms they’ve developed over millions of years of evolution which will kick into place. I am not sure anyone dreaming of this “utopia” is then going to be around to see it.

Looks like the associated article is: https://scale.com/research/swe_bench_pro (link in the repo is wrong)

https://scale.com/leaderboard/swe_bench_pro_commercial I definitely trust the totally private dataset more.

can't wait to see enigma eval (the hardest benchmark) https://scale.com/leaderboard/enigma_eval

- https://scale.com/blog/synthetic-data-fine-tuning-llms - https://eugeneyan.com/writing/recsys-llm/ - https://cookbook.openai.com/examples/sdg1

Scale AI is a provider of human data labeling services https://scale.com/rlhf

People could try but it would not help with the withheld datasets so much, and it would be possible to add more to it. If the withheld data was closed, and only available to an assessment system, gaming that would be pretty difficult. Scale.com's SEAL Leaderboards take a similar approach. The ARC Prize still exists too, and it's waiting for winners. See https://scale.com/leaderboard

Not sure what you mean by this, it is used by the military, and not in secret: https://scale.com/donovan Scroll down on the page, OpenAI is listed as a model provider, with logo and everything. Or do you mean some kind of more 'direct' deal with military?

I was able to find the first 3 jobs in your jobs section elsewhere. https://jobs.lever.co/tesorio/a702e44a-7329-43df-bff6-0a05f49c7747 https://transak.com/careers https://scale.com/careers/4328006005

Both Palantir and Donovan are looking to use LLMs in the military: https://www.palantir.com/platforms/aip/, https://scale.com/donovan This might be technically correct, in the sense that I think these companies have their own LLMs they're pushing? They're not literally using OpenAI's GPT model. But all LLMs are vulnerable to this, so it doesn't practically matter if they're using specifically GPT vs something in-house, the threat model is the same.

I’m pretty sure Anthropic’s Claude is doing that. https://scale.com/blog/chatgpt-vs-claude

It seems to be a theme with AI. Scale.ai (no affiliation, not a customer) has a product called Spellbook https://scale.com/spellbook

Anthropic is building Claude, a chatGPT alternative. Review here [1]. The move is interesting as it potentially signals doubt in their in own current AI capabilities (Deepmind/Lambda/Sparrow). Declinging to comment about it and not announcing this everywhere like Microsoft did with OpenAI is also odd. [1] https://scale.com/blog/chatgpt-vs-claude

https://scale.com/spellbook Build, compare, and deploy large language model apps with Scale Spellbook.

and the Scaliens. "I have made the difficult decision to reduce the size of our team by 20%, which means saying goodbye to many talented Scaliens." [1] [1] https://scale.com/blog/company-update

If it's done in a reCAPTCHA like way, it can be done fairly efficiently and for cheap. In fact Scale AI does just this, they do manual labor operations such as captioning images, as an API. Here's their product for image labeling: https://scale.com/rapid. Unstable Diffusion is also doing their captioning like how I mentioned, with groups of volunteers as well as hired individuals.

Because I'm talking about AI, not AGI. And because I'm observing the frontier of AI: https://twitter.com/AdeptAILabs/status/1570144499187453952 https://twitter.com/runwayml/status/1568220303808991232 https://scale.com/blog/text-universal-interface

Scale AI | Strategic Product Manager | REMOTE or SF | Full Time | https://scale.com/careers Scale (YC 16) is one of the fastest growing AI startups in the world. Our mission is to accelerate the development of AI and our first product involves providing high quality labeled data to help our customers train ML models. We are looking for a PM to lead strategic projects. This person will be in charge of delivering results for our most important customers or developing new products and processes in order to move key company metrics. The ideal candidate would have an entrepreneurial mindset (past founders would be ideal) and strong technical skills. Full job description: https://scale.com/careers/935d4882-8002-4816-9d87-c4b8b045d640 Feel free to email me directly at ruben[at]scale.com

> Scale makes software to help people train autonomous driving technology and other computer vision systems. True; I would also point out Scale is also involved in various fields independent of computer vision, eg NLP[1], document processing[2], and recently, ML dataset management[3]. [1] https://scale.com/text [2] https://scale.com/document [3] https://scale.com/nucleus (disclaimer: I am an employee at Scale.)

It gets worse. They just updated the Terms, however, they copied them directly from this site here: https://scale.com/legal/terms They even forgot to change the links, so all sebenz.ai links still go to the scale site...

At Scale (scale.com), we strongly believe that the “open-source” alternative to this is pretty critical. We’ve built this index for autonomous driving datasets (https://scale.com/open-datasets) and are building that out for other domains right now. Open source data has been a pillar to progress in ML (starting with ImageNet). It should continue to be the case that data that enables researches is sufficiently democratized.

Alexander Wang of scale. "was recognized as Forbes 30 under 30" https://scale.com/press

I work at Scale - I've met a bunch of people that work on our platform, and seeing the impact that it's had on their lives is actually a huge source of inspiration to me. There's a writeup highlighting some of their stories at https://scale.com/blog/positive-externalities - based on my personal experience, I can say that it's not bullshit.