Scattered Thoughts

The person behind Zest runs an interesting blog/work journal, I recommend subscribing [1]. I could be wrong, but my understanding is that they don't have a full-time job. Instead they do some amount of consulting, and then spend most of their time and intellectual energy working on their projects. Which is pretty cool I think. [1] https://www.scattered-thoughts.net

I love these ideas! I've been thinking about the "fully relational" language ever since I worked with some product folks and marketers at my start up 15 years ago who "couldn't code" but were wizards at cooking up SQL queries to answer questions about what was going on with our users and product. There was a language written in rust, Tablam[0] that I followed for a while, which seemed to espouse those ideas, but it seems like it's not being owrked on anymore. And Jamie from Scattered Thoughts[1] has posted some interesting articles in that direction as well. He used to work on the old YC-company/product LightTable or Eve or something, which was in the same space. I've also always thought Joe Armstrong's (RIP) thought of "why do we need modules" is really interesting, too. There's a language I've seen posted on HN here a couple times that seems to go in that approach, with functions named by their normalized hash contents, and referred to anywhere by that, but I can't seem to remember what it's called right now. Something like "Universe" I think? [0] https://github.com/Tablam/TablaM [1] https://www.scattered-thoughts.net [2] https://erlang.org/pipermail/erlang-questions/2011-May/058768.html

I use Postgres JSON functions to return nested results. The database itself contains no JSON; just a well-normalised data model. However, the queries return nested JSON in the format required by the application, e.g. for rendering an HTML template or returning JSON to the client in a single round trip. Check out the old dogs can sort of learn new tricks in this great article: https://www.scattered-thoughts.net/writing/sql-needed-structure/

HYTRADBOI 2022 had 571 attendees (https://www.scattered-thoughts.net/writing/hytradboi-2022-postmortem/)… At that point even one matrix chat room per track became hard to follow. I'm looking forward to doing it on zulip instead. Having unread counts per thread should make it much easier for speakers to keep track of which questions they haven't answered yet.

This is a tech-focused selection from my personal list, with a few more business-y sources thrown in: - https://erikbern.com - https://lethain.com - https://commoncog.com - https://scattered-thoughts.net - https://danluu.com - https://ideolalia.com - https://staysaasy.com - https://collabfund.com - https://blog.nelhage.com - https://jacobian.org - https://equalventures.substack.com - https://defmacro.org - https://spakhm.com - https://apenwarr.ca - https://scarletink.com - https://benkuhn.net - https://cpojer.net - https://brandur.org - https://zeptonaut.com Let me know if you want some reading recommendations for any of these!

https://www.scattered-thoughts.net/writing/against-sql/ is another great takedown of SQL.

SQL is indeed pretty terrible IMO. Here's my favorite source to trot out when people inevitably disagree: https://www.scattered-thoughts.net/writing/against-sql/

(just jumping in here) You may want to take a look at https://www.scattered-thoughts.net/writing/an-opinionated-map-of-incre… which describes things on the spreadsheety and not-very-spreadsheety ends of the spectrum (moreso the latter). If you follow some of the links there you should find some good descriptions of various implementation strategies, e.g. Naiad style timely dataflow or incremental-style pseudotime.

Quoting from https://www.scattered-thoughts.net/writing/against-sql/: --- There are many cases where a small change to a computation requires totally changing the structure of the query, but subqueries are my favourite because they're the most obvious way to express many queries and yet also provide so many cliffs to fall off. -- for each manager, find their employee with the highest salary > select > manager.name, > (select employee.name > from employee > where employee.manager = manager.name > order by employee.salary desc > limit 1) > from manager; name | name -------+------ alice | bob --- If the inner query has more than one row, the query will raise an error. That's difficult to know from the SQL alone.

Yeah. There's this quote I love: > If you have a mountain of shit to move, how much time should you spend looking for a bigger shovel? There's no obviously correct answer - it must depend on the size of the mountain, the availability of large shovels, how quickly you have to move it etc. But the answer absolutely cannot be 100% of your time. At some point you have to shovel some shit. From https://www.scattered-thoughts.net/writing/things-unlearned/

> Datalog has great compositionality, which is one of SQL's big weaknesses There's a good write-up on this aspect (amongst others) here: https://www.scattered-thoughts.net/writing/against-sql/ > recursive queries are completely natural in Datalog, whereas in SQL they are bolted on with recursive common table expressions And similarly: https://github.com/frankmcsherry/blog/blob/master/posts/2022-12-25.md

Nim's very GC-oriented which just isn't great/usable for the kernel. Yes, you can turn it off and allocate/deallocate manually, but then you barely gain anything on top of C apart from an oop system. Zig isn't improving correctness too much either. It provides some niceties, but no guarantees around correctness if I understand correctly. It's not trivial - here's a whole post about it https://www.scattered-thoughts.net/writing/how-safe-is-zig/

Why would anyone use a new systems programming language that is not safe? Zig might be better than C in some aspects, but is it worth it to switch to Zig when you realize that it's not much safer than C? [0] [0]: https://www.scattered-thoughts.net/writing/how-safe-is-zig/

This article praises everything Zig does and lists no negatives of its design. I'd recommend a different article for a balanced take, particularly regarding comparisons with Rust: https://www.scattered-thoughts.net/writing/assorted-thoughts-on-zig-an…

Worth clarifying that Bun isn't a from-scratch JS runtime, it's a wrapper around JavaScriptCore (WebKit's JS engine), like Node is around V8 As you say, there's still plenty of room for vulnerabilities in the parts it does implement, and Zig isn't strictly memory-safe. However, Zig has lots of modern features like optional-types that help quite a bit with avoiding memory errors: https://www.scattered-thoughts.net/writing/how-safe-is-zig/ Which is to say your question is definitely valid, but there are also reasons to think this isn't as huge a concern as it might seem

But that it does not bind well with the memory model of a certain wayland implementation does not mean that it cannot bind well with C generally. It also does not mean you cannot use it for Wayland since there is a native Wayland implementation. [0] And for the systems level programming there is stuff like the current work to make use of Rust and the guarantees it gives in large C based systems like the Linux kernel. That said that Zig is not a worthy endeavour, it does just not give as strong guarantees about memory safety as Rust works to give. (at least if this article is up to date [1]) [0]: https://github.com/Smithay/smithay [1]: https://www.scattered-thoughts.net/writing/how-safe-is-zig/

This is tangential, but the author is an incredible technical writer: https://scattered-thoughts.net/

This is a great in-depth comparison of Zig and Rust from the POV of someone who has used both extensively for non-toy projects. https://scattered-thoughts.net/writing/assorted-thoughts-on-zig-and-ru…

Learn how to prove theorems. "How to Prove It" by Velleman is the best book for this (imho). (Amazon link http://www.amazon.com/How-Prove-It-Structured-Approach/dp/0521675995) More good advice at http://scattered-thoughts.net/blog/2014/11/15/humans-should-learn-math…

SQL is kind of a monster language to parse properly[1], and has some unfortunate and confusing edge cases[2]. It would have cost a ton of time and energy to implement properly, and all it would have ended up with is "SQL but worse." SQL is also really awkward around APIs, since API norms and schemas don't map cleanly to SQL tables and JOINs. For example, many APIs take arguments, and while the most natural way to express an API call in SQL is as a SELECT or JOIN, requiring user-provided arguments there is really clunky. Worse, if the API that takes an argument sets a default value if that argument is not provided, then you end up in a situation where a `SELECT * FROM API` can return fewer results than a `SELECT * FROM API WHERE <some predicate>`. Using a new language meant that I could design it to best highlight Trustfall's capabilities with minimal work. Trustfall's core is language-agnostic, so it should be possible to put a new language parser in front of it in the future -- including SQL. So I haven't ruled it out forever, I just didn't want the first thing I built to be "SQL but worse" because it's in a domain poorly suited for it. [1]: https://twitter.com/sc13ts/status/1413729761247916036 [2]: https://www.scattered-thoughts.net/writing/select-wat-from-sql/

I suggest using the fact foreign keys are constraints with unique names, and using these names to explicitly specify what column(s) to join between the two foreign key tables. In PostgreSQL [2], foreign key contraint names only need to be unique per table, which allows using the foreign table "as is" as the constraint name, which allows for nice short names. In other databases, the names will just need to be a little longer. Given this schema: CREATE TABLE baz ( id integer NOT NULL, PRIMARY KEY (id) ); CREATE TABLE bar ( id integer NOT NULL, baz_id integer, PRIMARY KEY (id), CONSTRAINT baz FOREIGN KEY (baz_id) REFERENCES baz ); CREATE TABLE foo ( id integer NOT NULL, bar_id integer, PRIMARY KEY (id), CONSTRAINT bar FOREIGN KEY (bar_id) REFERENCES bar ); We could write a normal SQL query like this: SELECT bar.id AS bar_id, baz.id AS baz_id FROM foo JOIN bar ON bar.id = foo.bar_id LEFT JOIN baz ON baz.id = bar.baz_id WHERE foo.id = 123 I suggest adding a new binary operator, allowed anywhere where a table name is expected, taking the table alias to join from as left operand, and the name of the foreign kery contraint to follow as the right operand. Perhaps "->" could be used for this purpose, since it's currently not used by the SQL spec in the FROM clause. This would allow rewriting the above query into this: SELECT bar.id AS bar_id, baz.id AS baz_id FROM foo JOIN foo->bar LEFT JOIN bar->baz …

> Yep. I meant it was easy to do it the inefficient way where you just refresh the entire query when any table mentioned in the query changes. You would just have to also check if something was a view and recursively parse the SQL that is used in the view. Just use Postgres `LISTEN` and triggers or the WAL for change monitoring. Unfortunately, it turns out that recursively refreshing views still leads to surprising behavior. I think post summarizes the problem quite nicely: https://scattered-thoughts.net/writing/internal-consistency-in-streami… If you cannot refresh all of the views, at a single point in time, then there will be internal inconsistencies in your dataset. When looking at automatic refreshing, simple triggers and `LISTEN/NOTIFY` don't scale, as was mentioned in the comment regarding Hasura's multiplexing. I think, in the absence of incrementally maintained views, their multiplexing strategy is a good compromise for databases like postgres. However, it should be noted that continuous query / subscription of views is the exact scenario under which incremental computation will provide both lower latency and greater resource efficiency.

Thinking of programming as a dialogue with the computer is a really strong metaphor that I gladly recognize as something that inspired you [1][2]. Short: You provide the goals and the constraints, then through a back and forth dialogue with the computer you describe and clarify the problem (and also your understanding). You keep on negotiating and providing more constraints until it simply finds a possible solution (the problem domain is now computable) or the best fitting solution now satisfies your needs (in time). The hardest parts are probably: 1. Collecting, developing the constraint solvers. Should we imagine that collection more as platform of user contributed (programmed/trained) packages? 2. Developing the dialouge that is not just textual but visual (in both ways) (notepad vs IDE vs Theater(?!) don't forget macros/acting out!)). Showing is always faster then writing and gives one better cues on the closures/framework of the actual discussion. [1] http://scattered-thoughts.net/blog/2014/07/21/imperative-thinking-and-… [2] https://gist.github.com/jamii/8957881c8eaa4035f4ae

https://www.scattered-thoughts.net/writing/an-opinionated-map-of-incre… Can help map out the landscape.

Materialized views in Postgres don't update incrementally as the data in the relevant tables updates.^1 In order to keep it up to date, the developer has to tell postgres to refresh the data and postgres will do all the work from scratch. Incremental Materialized views are _hard_. This^2 article goes through how Materialize does it. MSSQL does it really well from what I understand. They only have a few restrictions, though I've never used a MSSQL materialized view in production.^3 [1]: https://www.postgresql.org/docs/current/rules-materializedviews.html [2]: https://www.scattered-thoughts.net/writing/materialize-decorrelation [3]: https://learn.microsoft.com/en-us/sql/t-sql/statements/create-material…

Flink SQL is quite limited compared to Feldera/DBSP or Frank’s Materialize.com, and has some correctness limitations: it’s “eventually consistent” but until you stop the data it’s unlikely to ever be actually correct when working with streaming joins. https://www.scattered-thoughts.net/writing/internal-consistency-in-str…

The author outlines a fair bit of the design thinking in this post [0]. What Datalog and co give you is the ability to do interesting program transformations. Mostly that's not because they are "declarative", if that means anything, but because they avoid all the little bits of non-determinism that imperative languages include. If you avoid those you're pretty much set. Coming from another angle, JAX [1] shows how you can do program transformations in Python with a relatively small set of semantic restrictions. [0]: https://www.scattered-thoughts.net/writing/unexplanations-sql-declarat… [1]: https://github.com/jax-ml/jax

Yes. Here's a neat essay that helps capture why I believe that. https://www.scattered-thoughts.net/writing/speed-matters/

prior art: https://research.google/pubs/sql-has-problems-we-can-fix-them-pipe-syn… https://www.scattered-thoughts.net/writing/against-sql/ - talks about compositionality https://www.edgedb.com/blog/we-can-do-better-than-sql (2019) (List taken from https://lobste.rs/s/vmyk7d/composable_sql#c_layb8n)

There have been proposals to have global state in programming languages which function like databases with the advantages of monitoring/persistence/naming etc, but also retain the modularity of local state. https://www.scattered-thoughts.net/writing/local-state-is-harmful/ https://awelonblue.wordpress.com/2012/10/21/local-state-is-poison/