Databricks Blog

Databricks -- San Francisco -- https://databricks.com * Software Engineer (ONSITE) * Software Engineer Intern (ONSITE) * Product Manager (ONSITE) Databricks was founded in 2013 by the team that started Apache Spark, meaning you might not only use Spark but also get to work on it :). We provide a cloud data processing platform based on Apache Spark used by customers including top 5 banks, healthcare and media companies. We've also done some really cool technical stuff, such as setting the 2014 GraySort record (http://www.wired.com/2014/10/startup-crunches-100-terabytes-data-recor… We are hiring engineers in the following areas: * Backend (JVM, AWS, database engine) * Frontend (React, D3) * Machine learning List of available positions: https://databricks.com/company/careers

https://www.databricks.com is doing this already with data as well as multiple other companies And I have first hand knowledge of well-known companies building their own tooling because the SaaS offerings have a bad price/feature ratio.

This was a simplification. Of course you need some extra VRAM I/O based on your KV cache size. But assuming your KV cache size is << model size, that simplification is pretty accurate. See, e.g. https://www.databricks.com/blog/llm-inference-performance-engineering-… You can just scroll to the first chart they have that explains the idea.

Remember: inference is memory bound. https://www.databricks.com/blog/llm-inference-performance-engineering-…

Here is Databricks page on that: https://www.databricks.com/glossary/data-lakehouse That wasn't very enlightening (at least not to me), so here is Google's page: https://cloud.google.com/discover/what-is-a-data-lakehouse That page wasn't great either. As best as I understand it, they are trying to add more structure to the data lake, and optionally ACID compliance, so that datawarehouse tools can talk to the lake directly.

Disclaimer: original blog author > as opposed to solving an actual problem This was literally the point of the post. No one really knows what the future of LLMs will look like, so DSPy just iteratively changes in the best way it can for your metric (your problem). > someone actually using for something other than a toy example DSPy, among the problems I listed in the post, has some scalability problems, too, but I am not going to take away from that. There are at least early signs of enterprise adoption from posts like this blog: https://www.databricks.com/blog/optimizing-databricks-llm-pipelines-ds…

Make your website look like this: https://curl.se/ And not like this: https://www.databricks.com/

It's similar to llama2. > If, on the DBRX version release date, the monthly active users of the products > or services made available by or for Licensee, or Licensee’s affiliates, is > greater than 700 million monthly active users in the preceding calendar > month, you must request a license from Databricks, which we may grant to you > in our sole discretion, and you are not authorized to exercise any of the > rights under this Agreement unless or until Databricks otherwise expressly > grants you such rights. https://www.databricks.com/legal/open-model-license

Gaudi2s started coming out in 2022 (https://huggingface.co/blog/habana-gaudi-2-benchmark) but didn't hit mass scale. I think Gaudi3 will & others have seen similar performance for Gaudi2 eg Databricks: https://www.databricks.com/blog/llm-training-and-inference-intel-gaudi… mlperf etc We are about to drop stable diffusion 3 which is the best image model out there (https://x.com/EMostaque/status/1764941367682256950?s=20) with similar architecture to Sora by OpenAI that can be used for any modality. We have hundreds of millions of downloads of our models so are looking for big scale as we move to every pixel being generated & this stuff goes from research to mass deployment.

Those limits seem to be decreasing every year though. Lot of competition in the analytical SQL space. Also worth noting Delta Live Tables, another very popular streaming SQL implementation. https://www.databricks.com/product/delta-live-tables

Quite a few do - here is databricks https://www.databricks.com/

the GPT-J-6B one is Dolly 1.0, previously released Dolly 2.0 is Pythia-12B fine-tuned on this new dataset on their hugging face page [1] they admit the performance may not be much or any better than the original model (I am guessing this may be a weakness of Pythia-12B, which was intended for model-training research rather than best results) the main point of Dolly 2.0 is the new dataset is unencumbered legally [2] whereas Alpaca et al were trained on ChatGPT transcripts, so commercialising those models would contradict OpenAI licensing terms [1] https://huggingface.co/databricks/dolly-v2-12b [2] https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercial…

Databricks | Senior Staff Software Engineer | Onsite (SF Bay area and Seattle areas only) | Full Time | https://www.databricks.com We are building the core of the Trust and Safety org at Databricks (Abuse, Security and Privacy/Compliance). We plan to build/evolve our infrastructure to be correct by design to scale the system for more complex products, larger teams and higher qps. If this is something you've seen/done before and/or are willing to take a stab at it, join us! Security: https://databricks.com/company/careers/open-positions/job?gh_jid=58924… Compliance: https://databricks.com/company/careers/open-positions/job?gh_jid=58924… If you want to directly reach out: [email protected]

On the following link you can learn how Zalando implemented this concept in the real world: https://databricks.com/session_na20/data-mesh-in-practice-how-europes-…

You can do real-time feature engineering in your application as well. But we recommend to do it in a feature pipeline that also syncs the features to the feature store. We have a talk about it coming out at the spark/ai summit: https://databricks.com/session_eu20/real-time-feature-engineering-with…

Thanks for the wishes! Indeed it's rarely worth it to build an automated tuning tool: - Unless you operate at a massive scale (eg Dr Elephant + TuneIn projects, originally developed at LinkedIn) - Or you operate a big data platform yourself. If you're curious about our ML approach, we gave a tech talk about it at last year's Spark Summit: https://databricks.com/session_eu19/how-to-automate-performance-tuning…

There are approaches for using Spark to distribute hyperparameter tuning and cross validation: https://databricks.com/blog/2016/02/08/auto-scaling-scikit-learn-with-… However, for the example in this post, I would recommended using the logistic regression provided by MLlib to scale up.

So much this. Another perfect example: https://databricks.com They just raised a $140m round of financing so apparently they have some good stuff going on. If you look at the website though: The Unified Analytics Platform. Accelerate innovation by unifying data science, engineering, and business. Sorry, what? Click on "learn more about the platform": DATABRICKS IS A TRULY UNIFIED APPROACH TO DATA ANALYTICS AT SCALE. Founded by the team who created Apache Spark, Databricks provides a Unified Analytics Platform that accelerates innovation by unifying data science, engineering, and business. I still have no clue what exactly TF the product is but I sure got my weekly dose of BS buzzwords.

It maybe worth investigating sparklyr[1][2] from RStudio. [1] https://github.com/rstudio/sparklyr [2] https://databricks.com/blog/2017/05/25/using-sparklyr-databricks.html

Would Databricks solve this problem? https://databricks.com/ They are essentially Apache Spark-as-a-service and have an API that allows you to submit a job on a cluster that you can configure to autoscale: https://community.cloud.databricks.com/doc/api/#jobs.JobsService.creat… https://community.cloud.databricks.com/doc/api/#jobs.ClusterSpec.NewCl…

Databricks - https://databricks.com/ - San Francisco, CA - Frontend, backend, site reliability engineers Databricks is building a cloud platform that makes data science simple. We are founded by the team that created Apache Spark, and are dedicated to the open source development of Spark. Frontend / Full stack engineers: focused on providing a feature rich notebook that enhances the user's experience with Spark. Working with React, Backbone, d3, new and creative visualizations, and improving the way people work with big data is what we do. We are also hiring for other software engineering positions. Experience with scala and working knowledge of distributed systems is a major plus! https://databricks.com/company/careers If you have any questions feel free to email me directly at [email protected]

There are some interesting platforms that popped up over the last few years: http://zillabyte.com/ http://databricks.com/ https://yhathq.com/products/scienceops The thing I'm most excited about with these platforms is seeing distributed computing become accessible to more developers. Lowering the barrier to entry (not needing to know the Hadoop ecosystem or have a PhD to do data science) can help developers answer questions their companies have and be more productive. Just my two cents.

Are you using an API, or running locally? If so, are you running with a quant, or other 'optimisations'? I've been using it via openrouter pretty heavily as my daily driver for the past week and loving it, have never experienced incoherent rubbish even at 500k+ contexts (that's usually way higher than I'd typically compact at), and tool calling reliability is better than Opus 5 in the Claude Code harness. Modern Anthropic models frequently get tool calls wrong, invent non-existent references or SQL tables, or have gibberish CJK characters in the output, like out of nowhere. Of course, they're great at self-recovery after an incorrect tool call, but so is Deepseek v4 flash. If you're running a quant, and esp with a quant'd KV cache, then yeah, not surprised if you're getting incoherent results; but you're not running the real/full model. Also, which harness? Try something like Pi or OMP. Models perform better in these harnesses than Claude Code: https://www.databricks.com/blog/benchmarking-coding-agents-databricks-… The main reason to use Cladue Code is a subsidised Anthropic subscription. If you're on API rates, you should not use Claude Code; you pay more for worse results. Claude Code is sadly quite bloated these days, and comes with a lot of proprietary context window garage like claude design skills, claude.ai artifacts, etc that you probably don't use, and if you do, well, you can add it.

Hey! Thanks for the feedback! I work on many of these things at Databricks, so figured I'd chime in on this. Firstly, while routing is important, simple things like observability into the token costs of various features, which can drive optimization of better default parameters, are low hanging fruit everyone should do. 1. As for the cache busting, we're very aware of this. Had said this in a thread above too, so copying it here. We're going to do a followup blog detailing our routing approach, but we're designing it to be cache aware. The router takes in the task description and infers what models and harnesses are available and makes a recommendation up-front. So essentially the routing decision is made when the harness + model is kicked off and it's only changed halfway through if there's a major delta in complexity from the initial judgment. Therefore, most of the time the cache is maintained just as it would be before. This does have disadvantages too, but the cache is the dominating cost reduction force. 2. We do love Pi (we published this too: https://www.databricks.com/blog/benchmarking-coding-agents-databricks-… but we didn't mention it here because we're still in the process of making it available to devs internally. There's a lot that goes into this more than just cost (e.g. feature gaps vs. other common harnesses). 3. As for existing utilization dashboards, it's good that each tool has one, but we found that as we gave developers…

It's really easy to get lost in the technical jargon that the vendors who are selling products throw around, but this article has missed the important part, and spent all the time talking about the relatively unimportant part (data formats). You need to step back and look from a broader perspective to understand this domain. Talking about arrow/parquet/iceberg is like talking about InnoDB vs MyISAM when you're talking about databases; yes, those are technically storage engines for mysql/mariadb, but no, you probably do not care about them until you need them, and you most certainly do not care about them when you want to understand what a relational DB vs. an no-SQL db are. They are technical details. ... So, if you step back, what you need to read about is STAR SCHEMAS. Here are some links (1), (2). This is what people used to be before data lakes. So the tldr: you have a big database which contains condensed and annotated versions of your data, which is easy to query, and structured in a way that is suitable for visualization tools such as PowerBI, Tableau, MicroStrategy (ugh, but people do use it), etc. to use. This means you can generate reports and insights from your data. Great. ...the problem is that generating this structured data from absolutely massive amounts of unstructured data involves a truly colossal amount of engineering work; and it's never realtime. That's because the process of turning raw data into a star schema was traditionally done via ETL…

MosaicML/Databricks | San Francisco Bay Area or New York | Machine Learning Engineer - Performance Optimization | Full-time Founded in late 2020 by a small group of machine learning engineers and researchers, MosaicML enables companies to securely fine-tune, train and deploy custom AI models on their own data, for maximum security and control. Compatible with all major cloud providers, the MosaicML platform provides maximum flexibility for AI development. Introduced in 2023, MosaicML’s pretrained transformer models have established a new standard for open source, commercially usable LLMs and have been downloaded over 3 million times. MosaicML is committed to the belief that a company’s AI models are just as valuable as any other core IP, and that high-quality AI models should be available to all. Now part of Databricks since July 2023, we are passionate about enabling our customers to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI platform so our customers can use deep data insights to improve their business. We leap at every opportunity to solve technical challenges, striving to empower our customers with the best data and AI capabilities. Apply here or reach out at daya@[company name…

Take a look at Databricks' getting a hand from internal team to create the dolly 15k dataset. ( https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercial… ) For training AGI (artificial general intelligence) maybe only a select few mega companies with massive datasets will be able to come up with training data. There are so many other use cases that OSS projects can enable otherwise. Individuals or smaller companies have unique data that can be used to augment existing open source models. Many use cases are area specific, and without the need for general intelligence. Palantir just did a talk at the AIPCon ( https://youtu.be/o2b0DwNg6Ko ), where they recommended the use of many LLMs, open and closed. ( the example had Llama 2 70B, GPT4, Palm coding, claude, + fine-tuned models) feeding into their synthesizer. While I want open source to win, especially as an open source maintainer on Ollama (if you haven't seen it yet, it's one of the easiest ways to run LLMs locally - https://github.com/jmorganca/ollama ), I think the work so far in this space has been a positive sum one - open or closed.

ClickHouse at least supports local joins quite well. Perhaps another way to put it is that BigQuery, Redshift, and Snowflake are not optimized for real-time response on large, wide tables. ClickHouse has features that allow it to pack multiple entities in a single table, then pull data out in a single scan. This includes tricks like simulating joins using aggregation. [0] It's a great design for feeding tenant dashboards with fixed latency (say 2s or less) and predictable cost. This use case is shared by many SaaS offerings. Over time I think the differences will become less as current database engines converge on commonly required features. ClickHouse join types have expanded over the last year and features like join reordering are in the 2023 roadmap. [1] Conversely incumbent cloud databases are adding features to support real-time analytics. I work on ClickHouse at Altinity and can't speak for Druid and Pinot. Perhaps someone else with detailed knowledge can chip in. [0] https://www.databricks.com/dataaisummit/session/opening-floodgates-ena… [1] https://github.com/ClickHouse/ClickHouse/issues/44767

> While performing the benchmarks, we noticed that the Snowflake pre-baked TPC-DS dataset had been recreated two days after our benchmark results were announced. An important part of the official benchmark is to verify the creation of the dataset. So, instead of using Snowflake’s pre-baked dataset, we uploaded an official TPC-DS dataset and used identical schema as Snowflake uses on its pre-baked dataset (including the same clustering column sets), on identical cluster size (4XL). We then ran and timed the POWER test three times. The first cold run took 10,085 secs, and the fastest of the three runs took 7,276 seconds. *Just to recap, we loaded the official TPC-DS dataset into Snowflake, timed how long it takes to run the power test, and it took 1.9x longer (best of 3) than what Snowflake reported in their blog.* https://databricks.com/blog/2021/11/15/snowflake-claims-similar-price-…

We make heavy use of Jsonnet at work (https://databricks.com/blog/2017/06/26/declarative-infrastructure-json… It's worked great. Having a hermetic, pure templating system whose only output is a set of JSON/YAML files means that you can refactor fearlessly: as long as the materialized JSON/YAML doesn't change, you are 100% sure your refactor is safe. Bazel's StarLark dialect of Python (https://github.com/bazelbuild/starlark) has similar benefits. The language is simple and remarkably well specified, enough that we implemented our own intellij plugin (https://plugins.jetbrains.com/plugin/10852-jsonnet) and even our own faster compiler (https://github.com/databricks/sjsonnet) without much effort at all. There are odd corners in the language, but not something that most people will end up bumping into in typical usage. The templates certainly get messy in large configurations, but no more messy than any other code, and the hermeticity/purity greatly helps in managing the messiness. It's certainly less messy/odd than the copy-paste configs or be-spoke JSON/YAML templating systems that inevitably appear in messy deployment environments! The last thing of note is the lack of static types: this definitely affects usability to some extent, and especially hinders IDE support from being as useful as it is in e.g. Java. But having a useful/ergonomic type system that fits this specific problem space is probably still an unsolved research question.