Dremio Blog

Dremio | Full Stack and System Engineers | Santa Clara, CA | Onsite (once shelter-in-place ends) | https://www.dremio.com/ We are looking for Full Stack and System engineers. Our stack is Java and React and open source our core technologies (such as Apache Arrow). Dremio is the Data Lake Engine company. Our mission is to reshape the world of analytics to deliver on the promise of data with a fundamentally new architecture, purpose-built for the exploding trend towards cloud data lake storage such as AWS S3 and Microsoft ADLS. We dramatically reduce and even eliminate the need for the complex and expensive workarounds that have been in use for decades, such as data warehouses (whether on-premise or cloud-native), structural data prep, ETL, cubes, and extracts. We do this by enabling lightning-fast queries directly against data lake storage, combined with full self-service for data users and full governance and control for IT. The results for enterprises are extremely compelling: 100X faster time to insight; 10X greater efficiency; zero data copies; and game-changing simplicity. And equally compelling is the market opportunity for Dremio, as we are well on our way to disrupting a $25BN+ market. Find out more at https://www.dremio.com/careers/

Dremio has a Model Context Protocol (MCP) server implementation: https://www.dremio.com/dremio-mcp-server/ imho, your best bet is to align your company on using the enterprise offerings of one of the major providers (OpenAI, for example). Once you make that purchase, you can integrate your Dremio lakehouse with their tooling using the MCP approach: https://platform.openai.com/docs/guides/tools-remote-mcp

Common opensource options (other than Spark and Flink): 1. Dremio: https://www.dremio.com/ 2. Trino: https://trino.io/ 3. Iceberg Java API: https://iceberg.apache.org/docs/1.6.1/api/

Dremio is a really nice product built using arrow. https://www.dremio.com

Really nice! Something similar (with a GUI) is https://www.dremio.com/ that I think has gained some traction.

https://dremio.com tries to solve this exact problem. It’s open source too.

Interesting project! I only got to know about Apache Calcite a few months/weeks ago, but it seems like it's becoming a de facto way of supporting SQL on analytics products. I can think of Dremio [1] as an example. From an implementation perspective, I wonder how AthenaX will compare to the Apache Beam SQL DSL [2] (I haven't tried it out, still learning DoFn, ParDo, etc.). Beam is also leveraging Calcite, with the partial difference that it runs* on Flink amongst others. I think it's safe to say that the streaming analytics ecosystem remains interesting and exciting! Hope I'll be able to try out AthenaX, congratulations on the release. [1] https://www.dremio.com [2] https://github.com/apache/beam/tree/DSL_SQL

We considered starting a company in SF a couple years ago. I'm glad that we decided to instead start it in Mountain View (http://www.dremio.com). We're able to attract very talented people from San Jose, Milpitas, Redwood City, etc. where rents are a bit more reasonable.

Neat! Matt Topol from Arrow gave a cool talk in 2022, showing a service that did Data Federation via GQL but using gRPC and Apache Arrow for performance https://www.dremio.com/wp-content/uploads/2022/05/Powering-a-Data-Mesh…

There is a bunch of other projects that grew out of arrow which are also contributing a lot to data engineering: https://www.dremio.com/blog/apache-arrows-rapid-growth-over-the-years/

This article might help, it explains Arrow's performance benefits: https://www.dremio.com/apache-arrow-explained/

Also, for more on Arrow and Flight: https://www.dremio.com/understanding-apache-arrow-flight/

Dremio is JVM based but with compilation down to LLVM and they contributed Gandiva to the Apache Arrow project. Dremio seems pretty impressive but I haven't personally used it. https://www.dremio.com/announcing-gandiva-initiative-for-apache-arrow/

More on the Gandiva Initiative here: https://www.dremio.com/announcing-gandiva-initiative-for-apache-arrow/

Nice article, thanks! It would be great if DuckDB handled this itself, but it seems like to be competitive with Athena on really massive datasets, you need to have a metadata layer that is used to figure out which parquet files in S3 DuckDB actually needs to query and then potentially run those in parallel. This seems to be the architecture of Puffin (which I haven't personally tried using yet). [1] https://www.boilingdata.com/ [2] https://boilingdata.medium.com/lightning-fast-aggregations-by-distribu… [3] https://github.com/sutoiku/puffin One possible thing to look into would be whether this dataset is partitioned too much. My understanding is that the recommended file size for individual parquet files is 512MB to 1GB, whereas here they are 50MB. It would be interesting to see the impact of the partitioning strategy on these benchmarks. [4] https://parquet.apache.org/docs/file-format/configurations/ [5] https://www.dremio.com/blog/tuning-parquet/

I think the reason this is discussed now is because yesterday I tried to re-submit the Apache Arrow article. Here's what I wrote: https://en.wikipedia.org/w/index.php?title=Draft:Apache_Arrow&oldi… It was rejected / reverted 10 minutes later by a Wikipedia editor. The blog post from Justin was in July 2019 (https://www.dremio.com/why-apache-arrow-wikipedia/) There's many interesting and good points in the discussion here, thank you! To add my 2 cents: - Apache Arrow is notable, deserves a Wikipedia page. It might not have been when someone first tried to create a Wikipedia page for it in 2017 (see https://en.wikipedia.org/w/index.php?title=Draft:Apache_Arrow&acti… but in the three years since it has become a major project, see e.g. https://blogs.apache.org/foundation/entry/the-apache-software-foundati… Notability is clearly subjective, depends on what the author and reviewer find interesting. In the variant I submitted yesterday I tried to make it clear why it's notable - Apache arrow is a standard format that connects different languages, runtimes, data systems, communities, e.g. the Python and Java data communities. See e.g. https://wesmckinney.com/blog/apache-arrow-pandas-internals/ - Apache Arrow is to my knowledge partly the brainchild of Wes McKinney, creator of pandas, it's his attempt (looking strongly like success) to resolve a major issue in data science. - I think it's a good point Justin made at…