Pat Shaughnessy

:). Thank you, this is definitely helping my motivation to write more of these posts! If you haven't seen it before, I think you'll also like this post (not by me): http://patshaughnessy.net/2014/10/13/following-a-select-statement-thro…

This article was also discussed before: https://news.ycombinator.com/item?id=8449329 Pat Shaughnessy has 3 other similar articles about Postgres internals. He does a great job of combining high level explanations, visual aids, and gritty details. Discovering the Computer Science Behind Postgres Indexes http://patshaughnessy.net/2014/11/11/discovering-the-computer-science-… A Look at How Postgres Executes a Tiny Join http://patshaughnessy.net/2015/11/24/a-look-at-how-postgres-executes-a… Is Your Postgres Query Starved for Memory? http://patshaughnessy.net/2016/1/22/is-your-postgres-query-starved-for…

I've been hoping to come across something like this ever since Pat Shaughnessy's Ruby Under a Microscope (http://patshaughnessy.net/ruby-under-a-microscope). This looks fantastic!

As mentioned in the opening paragraph, this is a repost from Pat Shaughnessy's blog[1] from 2013. Might be worth linking to the original and adding the year to the title... [1]: http://patshaughnessy.net/2013/10/24/visualizing-garbage-collection-in…

This was already blogged elsewhere last year. Part 1) http://patshaughnessy.net/2014/10/13/following-a-select-statement-thro… and Part 2) http://patshaughnessy.net/2014/11/11/discovering-the-computer-science-…

Pat Shaughnessy's Ruby Under a Microscope[1] is a fantastic resource for looking at Ruby's internals, especially concerning parsing, Ruby's class structure, and the YARV virtual machine. It's probably easiest to begin with a Lisp dialect, since a basic one needs only two data types (singly-linked lists and symbols) and its eval function is so simple. Peter Norvig has a great tutorial[2] on implementing a Lisp in Python. Beyond that, it can be helpful to look at the source code for other languages similar to what you're looking to create, to get a sense of how they do things. That's more or less what I found helpful, in any case. Good luck! [1] http://patshaughnessy.net/ruby-under-a-microscope [2] http://norvig.com/lispy.html

To anyone who loves this I highly recommend reading "Ruby under a microscope" by Pat Shaughnessy. http://patshaughnessy.net/ruby-under-a-microscope

I can see the advantages of using the Arel predicates, but to me this: User.order("id desc") reads better than: User.order(User.arel_table[User.primary_key].desc) The latter seems a lot less expressive. Also, Pat's book is great if you're curious about Ruby internals: http://patshaughnessy.net/ruby-under-a-microscope

This is a bit old now, but this blog post has a great description of the GC changes that came with MRI 2.0: http://patshaughnessy.net/2012/3/23/why-you-should-be-excited-about-ga…

I discuss circular references and how CPython handles them a bit in my followup post: http://patshaughnessy.net/2013/10/30/generational-gc-in-python-and-rub…

The author's second article also contains information on how Python deals with circular references (a tracking garbage collector very similar to mark-and-sweep). http://patshaughnessy.net/2013/10/30/generational-gc-in-python-and-rub… It's also worth noting that you don't always need to deal with circular references in the case of short running programs. Additionally you can eliminate the possibility of ever creating a circular reference by using an immutable language.

Excellent point. Ruby's take on this has been interesting. http://patshaughnessy.net/2012/3/23/why-you-should-be-excited-about-ga… I'm not a ruby expert but by understanding is that basically they've moved the refcount field out of the struct and out of the memory page. It would be nice if python did something like this. [edit] My summary of what ruby does is totally wrong and while this is interesting and applies to memory managment, doesn't necessarily apply to refcounting.

Hi, thanks for the comment and encouragement! If you're looking for books on the internals of Rails, this one could be right up your street: http://patshaughnessy.net/ruby-under-a-microscope If you're looking for more general but advanced books there are loads of fantastic ones - some of my favourites are Eloquent Ruby, Design Patterns in Ruby, Metaprogramming Ruby and there's Crafting Rails Applications for Rails (though I haven't started this myself yet - but I hear it's very good).

Great question. I'm a huge fan of separating the business specifications from talking about the execution. ------- Some runtimes will do this under the hood. RDBMS execution plans are a good example. Plan costs are calculated using data-dependent statistics. And cache-friendliness is taken into account: in postgres, hash joins are made to fit L1 (tuples per bucket = 10) and L3 caches (work_mem default to 4MB, after which batches are introduced) [1] Some JITs will also monitor performance and behavior, and try to speculatively prune unused paths. But this is all letting the runtime deal with it. ------- Execution should be treated as an orthogonal but tweakable concern. And be exposed in some way to the user. Halide [2] is one very interesting approach, is letting user choose: * between materialization and compute-again * parallelism levels * chunk sizes * scheduling, etc. However the storage is implied by the execution chosen. Which to me is a constraint that is automatically enforced, and not exposed. ------- All knobs, and their consequences, should be readily available to both users and compilers. To me, typed languages could, should contain so much more information about performance (or anything!). The perfect abstraction should be the seamless integration of 4 languages: * Logical business requirements (what you want done). Should be user friendly * Business properties requirements (constraints like liveness, performance envelope, security, or…

Beyond Go and channels, this article/post touches on a subject that is personally a real pain to me. > It’s so cool to find yourself writing tutorial for beginners! And it's a bit painful when you are trying to implement big and sophisticated system(s). Channels are primitives. They are low-level building blocks and I highly doubt you want to work with them on daily basis. Posts/talks that relate to beginners. You know the drill, you only have a limited time to talk about something in a conference and you only have limited time to write a blog post. It's easy and tempting to cover the basics without diving in too much into the details. What you are left with as a professional is very thin data on how to accomplish complex stuff. If you look at Docker and all the talks/posts around docker, they cover the absolute basic. Where's deployment to production? where's alerting/monitoring? One of the few people I see solve this problem is: http://patshaughnessy.net/. I've been trying myself to do more in depth posts that cover more than just the basics, it's really hard work.

I'm working on a little static site generator now for my own site. The PHP/SQL wordpress kind of option just annoys me and have held me back from publishing altogether (don't even get me started on publishing services and social media sites. Well, stuff like hugo, nanoc, ikiwiki... inspirational! My static site generator is using rake for file rules to do stuff with git hooks. It's going to depend on Ruby, but won't be a "Ruby framework". Rake might as well run php, perl, a bash script or a binary. Rake is just a wonderful tool for the job, and Ruby is about as ubiquitous as Perl these days. Here's some inspiration: http://patshaughnessy.net/2015/1/8/using-rake-to-generate-a-blog Be sure to check out the "power rake" talk he mentions if you haven't already. Edit: one more thought. On the idea of teaching casual users "git", I think that's an amazing idea. It touches on tip of an iceberg: computer/scientific/technological literacy gaps. If our technologies continue down the path of "black magic," with the ever extreme dichotomy of priests who know the secrets vs. ordinary/"dumb" people who are locked out of the inner knowledge, it's just not going to be conducive to a bright future. Remember, regular people are dependent on technology, but have begun to develop superstitions about how it works. It's magic to people! This is such a deep issue.

Thanks, nitrogen. Following your info led to a couple of other neat discoveries, and a bunch more useful material [1]. As a result here are some new ways I learned to get 5 from 2 + 2: require 'fiddle' a = 2 b = 2 Fiddle::Pointer.new(a.object_id << 1).to_value.ceil + b => 5 Note: This happens because calling `to_value` on the `Fiddle::Pointer` object returns a Float (2.0000000000000004) instead of Fixnum (which was a surprise to me). And here's another using `Object#tap`: a = 2 (a + a).tap{|x| a = x + 1}.send(:eval,'a') => 5 [1] These links were helpful for me. The last one is an MIT licensed book called, "Kestrels, Quirky Birds, and Hopeless Egocentricity." http://stackoverflow.com/questions/1872110/is-ruby-pass-by-reference-o… http://patshaughnessy.net/2014/1/9/how-big-is-a-bignum http://stackoverflow.com/questions/2818602/in-ruby-why-does-inspect-pr… http://combinators.info/#kestrels Edit: line formatting

It's interesting to compare this to how python does it (reference counting with generations): http://patshaughnessy.net/2013/10/30/generational-gc-in-python-and-rub… "At first glance, Ruby and Python seem to implement garbage collection very differently. Ruby uses John McCarthy’s original mark and sweep algorithm, while Python uses reference counting. But when we look more closely, we see that Python uses bits of the mark and sweep idea to handle cyclic references, and that both Ruby and Python use generational garbage collection in similar ways. Python uses three separate generations, while Ruby 2.1 uses two. This similarity should not be a surprise. Both languages are using computer science research that was done decades ago – before either Ruby or Python were even invented. I find it fascinating that when you look “under the hood” at different programming languages, you often find the same fundamental ideas and algorithms are used by all of them. Modern programming languages owe a great deal to the ground breaking computer science research that John McCarthy and his contemporaries did back in the 1960s and 1970s."

Ruby Under a Microscope - https://patshaughnessy.net/ruby-under-a-microscope

I feel that a discussion of ltree is incomplete without discussion of manipulating the tree. e.g. splicing, cutting, moving, deleting branches [1] etc. Also, how ltree uses Gist indexes [2]. Reference: [1] http://patshaughnessy.net/2017/12/14/manipulating-trees-using-sql-and-… [2] http://patshaughnessy.net/2017/12/15/looking-inside-postgres-at-a-gist…

> you almost inevitably need to learn to read the Ruby codebase (written in C) That's how I ran across this reference, but from the other direction — I've been reading the MRI codebase with the goal of writing faster/better Ruby :) I've been working on my own version of a file-type identification library (even before the recent `mimemagic` drama) and have found it very necessary to understand the way allocation works since the code in question sits in my app's startup path and gets referenced more frequently than anything else. This post series by Pat Shaughnessy was also very useful for me: http://patshaughnessy.net/2013/2/8/ruby-mri-source-code-idioms-3-embed…

PostgreSQL has Ltree http://patshaughnessy.net/2017/12/13/saving-a-tree-in-postgres-using-l…

I'm actually tempted to start a series of blog posts in this vein for Ruby. The book Ruby Under a Microscope [0] covers these topics (and more), at least from a cursory glance of the blog series, but it would be nice to refresh that material in my own head for Ruby 3. [0]: http://patshaughnessy.net/ruby-under-a-microscope

http://patshaughnessy.net/2017/12/13/saving-a-tree-in-postgres-using-l… Look into ltree in postgres or similar

In Postgres you can use tree data with LTREE. http://patshaughnessy.net/2017/12/11/trying-to-represent-a-tree-struct…

If this sort of stuff is relevant to you, http://patshaughnessy.net/ruby-under-a-microscope is an essential book to read.

This cheerfully reminded me of the thing in MRI Ruby of (relatively recent) yesteryear where strings longer than 23 characters were twice as slow, qv. http://patshaughnessy.net/2012/1/4/never-create-ruby-strings-longer-th…

This doesn't really answer your question but you might be interested in this book http://patshaughnessy.net/ruby-under-a-microscope http://www.amazon.com/Ruby-Under-Microscope-Illustrated-Internals/dp/1…

In fact, such a book has been written before already: http://patshaughnessy.net/ruby-under-a-microscope

MRI has a global interpreter lock, so that does not make much sense. In fact, the diagram is simply wrong. This was rectified by the author in an article two weeks later: http://patshaughnessy.net/2012/1/18/seeing-double-how-ruby-shares-stri…