Trino Blog

You might find https://trino.io/ interesting. It allows you to bolt on a MPP SQL execution engine on top of any data source including pre-built connectors for Druid and Kafka. It's all ANSI SQL and the best part is you can combine data from heterogenous sources. e.g. You can join data between a topic in Kafka and a table in Druid or even between Kafka, S3 and your RDBMS. Disclaimer: I'm a maintainer of the project.

There could be a lot of reasons that are highly engine dependent, for this specific case. A general answer, perhaps... not specific to the case you've specified. Query optimization is a science with multiple dimensions [1]. I'd wager every problem in computer science plays a role somehow in query optimization. Query performance is based on a combination of actions you take to optimize the design of your system to get the best performance (e.g. data modelling, index design, hardware, query style, and more), and the patterns the system can recognize based on your inputs and the data itself, with the resources it has available, to optimize your queries. There are known patterns for optimization that are discovered over the years, many hard learned from practical experience. This is why older "popular" engines sometimes are more mature and more performant - they have optimizations built for the common use cases over long periods of time. That is not to say older engines are always better, just that they have often had more exposure to the variety of problems that occur. The reason why the engine "can't figure it out" is that most engines, even the best ones, are quite complex - combinations of known rules as well as more fuzzy logic, where the engine uses a combination of information and heuristics to essentially explore a possible solution space, to try to find the optimal execution plan. Making the right decision, well, can be difficult and given the nature of these…

Common opensource options (other than Spark and Flink): 1. Dremio: https://www.dremio.com/ 2. Trino: https://trino.io/ 3. Iceberg Java API: https://iceberg.apache.org/docs/1.6.1/api/

Trino is pretty good (open source presto). https://trino.io/

I would not compare it to lambda. It is backed by Trino https://trino.io/ and can do map/reduce kind of jobs that lambdas can not.

What if my SQL engine is Presto, Trino [1], or a similar query engine? If it's federating multiple source databases we peel the SQL back and get... SQL? Or you peel the SQL back and get... S3 + Mongo + Hadoop? Junior analysts would work at 1/10th the speed if they had to use those raw. [1] https://trino.io/

Trino can be fault tolerant but you have to explicitly enable fault tolerant execution. It might be worth running your benchmarks against Trino with fault tolerant execution mode enabled. Check the documentation here: https://trino.io/docs/current/admin/fault-tolerant-execution.html Adding fault tolerant to execution to Trino was a big and complicated project for anyone interested in more details check here: https://trino.io/blog/2022/05/05/tardigrade-launch.html

btw, if you want to know the backstory on why Presto is now called Trino, here's the article: https://trino.io/blog/2022/08/02/leaving-facebook-meta-best-for-trino.…

Just a quick follow up .. here is some of the innovation that happened in the project in the last decade - https://trino.io/blog/2022/08/04/decade-innovation.html

Big shout out to Brian Olsen from the Trino community (and Starburst) for helping the Trino community be successful - https://github.com/bitsondatadev - https://www.linkedin.com/in/bitsondatadev/ I recommend the Trino Slack for people not already in it: https://trino.io/slack.html

Trino is the community-open-source fork of Presto (they were recently required to change the name from PrestoSQL, which they had been using to make the Presto lineage clear). (not affiliated in any way). https://trino.io/

I want to second that plug for Athena for ad-hoc analysis. (If you're hosting your own stuff and at the scale where it'd be useful, there's Presto/Hive, which Athena is based on, and/or at Trino, the Presto fork maintained by some of its initial developers.) It was useful for me when tweaking spam/bot detection rules a while ago; if I could roughly describe a rule in a query, I could back-test it on old traffic and follow up on questionable-looking results (e.g. what other requests did this IP make around the time of the suspicious ones?). We also used Athena on a project looking into performance, and on network flow logs. The lack of recurring charges for an always-on cluster makes it great for occasional use like that. You can use what the docs call "partition projection" to efficiently limit the date range of logs to look at (https://docs.aws.amazon.com/athena/latest/ug/partition-projection.html… so it was free-ish to experiment with a query on the last couple days of data before looking further back. More generally, Athena/Presto/Hive support various data sources and formats (including applying regexps to text). Compressed plain-text formats like ALB logs can already be surprisingly cheap to store/scan. If you're producing/exporting data, it's worth looking into how these tools "like" to receive it--you may be able to use a more compact columnar format (Parquet or ORC) or take advantage of partitioning/bucketing…

DuckDB has support for ODBC/ADBC and even JDBC (so Trino should be easy, https://trino.io/docs/current/connector/duckdb.html).

Some more interesting use cases in production: Running python UDFs in Trino: https://trino.io/docs/current/udf/python.html Running the Ruby parser in Jruby: https://blog.enebo.com/2024/02/23/jruby-prism-parser.html

Having specific DDL would be fine for most purposes. As I said in a sibling response, I don't mind the differences in how tables are defined or in the speed at which they run. I get that you need runtime statistics, likely on your live tables, to get good performance guarantees. What drives me bonkers is the silly differences like ISNULL v COALESCE. That one, at least, has a mostly easy mechanical change to get between the two. Actually, what really drives me bonkers is when there are no local options. The aggregate functions such as https://trino.io/docs/current/functions/aggregate.html#approx_percenti… are super convenient for reporting purposes. And I can't think of any reason I can't run some of those queries against trivial data locally to show/confirm what a report is supposed to be doing.

Trino, not Presto. Presto, created by FB, was required to let any FB engineer merge without OWNERS (because Facebook doesn't have OWNERS files unless it would create a SEV1). Subsequently, original creators of Presto forked it to PrestoSQL. So Facebook trademarked the name Presto. So creators renamed it Trino. https://trino.io/blog/2020/12/27/announcing-trino.html

According to https://trino.io/blog/2020/12/27/announcing-trino.html , because of trademark registration and enforcement by Facebook. (IMHO, PrestoSQL also looked maybe a little too much like PostgreSQL in this space.) I'm imagining an unusually efficient brandstorming session. "OK, folks, idea hats on, there are no bad ideas... we've got Presto..." "Uh... new Presto... New-o..." "Neutrino..." "Trino?" "Trino!" "Searching it now!"