Edward Z. Yang

author here. honest answer: there isn't much that covers the whole thing end to end, that's partly why I started writing this. but the ones I learned from: - Edward Yang's "pytorch internals" post: http://blog.ezyang.com/2019/05/pytorch-internals/ it's the classic, goes deep on the c++ side. - the pytorch developer podcast, same author. short episodes, one internal topic each - the pytorch repo's own CONTRIBUTING.md, the folder layout explained by the maintainers - and if you want to build autograd instead of reading about it, Karpathy's micrograd / zero to hero mine is planned as 12 parts, the list is at the end of the post. part 1 (chapter Tensor: storage, strides, views) is in progress now.

That said, Spivak's work on framing databases in terms of categories [2] is pretty fascinating, and might be worth poking at. [2] http://blog.ezyang.com/2010/06/databases-are-categories/

I see a lot of comments in here about the v2+ rule’s reason for existing being to allow importing multiple major versions of a module. That’s not it at all. As the blog post (which was easily Googled by me and explains things quite well IMO) states: “If an old package and a new package have the same import path, the new package must be backwards compatible with the old package.” They’re trying not to break compatibility until a downstream developer explicitly opts in to that breakage. The simplest way to do that is to give the module a new name. And the simplest way to do that is append the major version to the name. Here is the post: https://blog.golang.org/v2-go-modules For more background on this principle, I recommend Rich Hickey’s Clojure/conj keynote “Spec-ulation” from 2016: http://blog.ezyang.com/2016/12/thoughts-about-spec-ulation-rich-hickey…

I’ve found this blog post to be the best of the bunch http://blog.ezyang.com/2010/11/dp-zoo-tour/

I think it simply reflects that the distinctions each is making are, to a great degree, subjective and up for debate. See, for example, http://blog.ezyang.com/2012/08/is-haskell-liberal-or-conservative/, which argues that Haskell of all languages is, in a sense, liberal.

My favorite article on this topic is Edward Z. Yang's "Tomatoes are a subtype of vegetables", which illustrates function contravariance with very charming drawings. http://blog.ezyang.com/2014/11/tomatoes-are-a-subtype-of-vegetables/

For the syntax, this article might help you: http://blog.ezyang.com/2011/11/how-to-read-haskell/ I personally continue to find many Haskell library code hard to read. But if you make some effort you own code can be quite readable. Unfortunately I don't know if there is any guideline to make your Haskell code more readable. For example, I'll be glad to get some advices regarding the following questions: - When to use type synonyms/data/records/tuples/lists for containers? - How to name your variables, type variables? - When to split your code between a long one-line or many small named functions? - When to split your code in many different modules? - etc...

I think this[1] blogpost has one of the best introductions to Haskell syntax that I have ever seen, and it doesn't get lost in other stuff. Perhaps you might find it useful. After some experience, I think the biggest pain point in Haskell's syntax is that it is very light on keywords so many things that would show up as syntax errors in other languages might show up as type errors in Haskell. For example, forgetting a comma in a list will result in a function application between the two list elements instead of a syntax error. [1] http://blog.ezyang.com/2011/11/how-to-read-haskell/

List comprehensions actually don't tend to get used very frequently, so I would not point to that as a good example of what Haskell code looks like. It's hard to point to one example, as different parts of the code look very different. That is, pure functions look very different from monadic functions, which both look very different from datatype declarations and typeclass instances. I would recommend poking around a whole codebase, one example of which can be found here: https://github.com/xmonad/xmonad This post may also be useful: http://blog.ezyang.com/2011/11/how-to-read-haskell/

I've been running maildir-label-syncing Sup for a while. It works great, but you need the master Sup client to be running for new email to get sorted. http://blog.ezyang.com/2012/12/maildir-synchronizing-sup/

The only semi-regularly updated ones I currently have in my RSS reader are: - James Hague's "Programming in the 21st Century": http://prog21.dadgum.com/ - Edward Z. Yang's blog: http://blog.ezyang.com/ Rest of my daily blog hits I get via Hacker News and reddit/r/haskell

For what is worth, Edward Yang is writing an excellent series on space leaks in Haskell: http://blog.ezyang.com/2011/05/space-leak-zoo/ The series require Haskell knowledge but this summary is fairly readable if you have a good grounding on programming language theory and implementation.

I love his diagrams! Friendly and accessible. Found a post on how he makes them: http://blog.ezyang.com/2010/04/diagramming-in-xournal-and-gimp/ though if you have a tablet and a good sense of design the software is probably not that important.

I wrote a distributed sat-solver in Erlang (good old fashioned search problem). It was my intro project to get to know the language. Has been great fun and now Erlang is in charge of computers talking to each other while Java takes care of the heavy lifting. Also a similar problem solved with Erlang (nothing to do with me), and featured here on hacker news... http://blog.ezyang.com/2010/08/tour-of-preach-distributed-erlang/ Great stuff. Erlang may make you smile (again).

Here are a bunch of blogs from my RSS reader that I think are worth reading. Some of them haven't updated in a while, but the archives may be worth a look anyway. * Blogs with a math focus: http://blog.sigfpe.com http://bit-player.org http://mathr.co.uk/blog/ https://www.johndcook.com/blog/ * Blogs with a programming languages focus: https://existentialtype.wordpress.com https://bluishcoder.co.nz http://blog.ezyang.com http://akkartik.name http://lambda-the-ultimate.org http://mrale.ph * Blogs with a UI design focus: http://www.brandonwalkin.com/blog/ http://ignorethecode.net/blog/ http://interuserface.net * Blogs with a graphics / 3D rendering focus: http://c0de517e.blogspot.com http://diaryofagraphicsprogrammer.blogspot.com https://directtovideo.wordpress.com http://graphicrants.blogspot.com http://www.reedbeta.com http://www.realtimerendering.com/blog/ https://blog.demofox.org http://eelpi.gotdns.org/blog.wiki.html * Other interesting / more general programming blogs (many of these have a "systems" focus): http://danluu.com http://ewontfix.com http://chneukirchen.org/blog/ http://neverworkintheory.org http://apenwarr.ca/log/ https://www.snellman.net/blog/ https://www.pvk.ca http://psy-lob-saw.blogspot.com https://randomascii.wordpress.com http://fastcompression.blogspot.com https://fgiesen.wordpress.com http://blog.plover.com http://chneukirchen.org/trivium/ http://rachelbythebay.com/w/

You might want to start with: http://blog.ezyang.com/2010/10/ocaml-for-haskellers/ and explore the languages for what they are yourself. I haven't tried Haskell apart from the very basic stuff, but if you have tried your hand at it for too long, and aren't able to make any headway, then you might as well invest time in learning OCaml/F# or Erlang or Idris or Rust or Elm instead. I hear Rust and Elm are pretty good too. OCaml doesn't do mutli-core, I think [0], whilst Haskell has great support parallelism, concurrency [1]. Erlang, of course, is absolutely amazing at that too. I primarily learn languages to expand my understanding of various programming constructs. It feels liberating to know that things are much easier in certain languages due to presence of certain constructs absent in other languages. For instance, OCaml is insanely good for compiler design and/or static analysis primarily due to 'variants' and the type system; Go (channels)/Erlang (actor-based programming, hot-plug code)/JVM-based languages (Clojure/Scala et al running on vastly optimized vm) for building massively concurrent systems; Ruby for being expressive, powerful, and simply awesome; JavaScript for... well... getting a taste of event-driven programming. The challenges faced when using some of these languages are fascinating and at some point, it all comes together, as you try searching for a language in another language (like trying to mimic functional/reactive paradigms in Java 7 or below when…

The point is a little more subtle than this, or how the article made it out to be. Global coherence and confluence (which GHC attempts to enforce unless told otherwise) is great for refactoring and equational reasoning and (yes) mathematics and programming at large, as Ed Kmett's fantastic "Typeclasses v. the World" talk covers: https://www.youtube.com/watch?v=hIZxTQP1ifo Global uniqueness of instances (which GHC does NOT enforce) is bad for modular programming, or programming at large, which is a point some of the commenters here are making. http://blog.ezyang.com/2014/07/type-classes-confluence-coherence-globa… The submitted HN link here has a terrible title. What the author is proposing is enforcing typeclass laws, which is a great idea (but difficult in practice to implement). Practice and time has shown that lawless classes are the ones that people hate, the ones that don't add any benefit and in fact make it harder to reason about large Haskell programs. I hope people don't take away from this link that classes in Haskell are bad. Haskell is three things: pure, functional, and typeclasses. They're part of the identity of Haskell; they're the reason we set off on this wonderful journey into the uncharted oceans of functional programming back in 1990. Typeclasses are AWESOME. And they can be improved. Postscript: Kmett also has a package that persuades the compiler to let you have dynamically generates instances. It has the same user experience as…

I must respectfully disagree that Haskell's memory footprint is simply 'low'. This is because the memory footprint of a given Haskell program is not at all transparent, and Haskell is notorious for leaking memory in a maddeningly opaque fashion [1, 2, 3, 4]. Space leaks might be relatively straightforward to diagnose and fix for a true domain expert, but I would not want to have to rely on someone having such abstruse knowledge in a production application. It goes without saying that a space leak in a production app is a really, really bad thing. Although, I suppose one's choice of Haskell is a function of one's own risk/reward profile. Haskell and its failure modes are hard to understand. That induces extra risk that some people (myself included) might be uncomfortable with. That said, I am now enthusiastically following you guys and hope to see the proverbial averages get sorely beaten. [1] http://neilmitchell.blogspot.com/2013/02/chasing-space-leak-in-shake.h… [2] http://blog.ezyang.com/2011/05/calling-all-space-leaks/ [3] http://blog.ezyang.com/2011/05/space-leak-zoo/ [4] http://blog.ezyang.com/2011/05/anatomy-of-a-thunk-leak/

Cabal has had nix-style local builds since 2016: https://blog.ezyang.com/2016/05/announcing-cabal-new-build-nix-style-l… https://cabal.readthedocs.io/en/2.0/nix-local-build-overview.html Also cabal isn't positioned to be a system level package manager. Haskell programmers are the type to want both their application builds and system dependencies to be reproducible and predictable. > Stop being lazy, go back to engineering first principles and it makes little sense to stay with Nix. Guix or any rewrite as a library in a well-developed language* makes more sense. Getting Guix packages to be as complete as nixpkgs isn't a matter of laziness though. One person wouldn't be able to do it no matter how disciplined they were.

PyTorch internals: http://blog.ezyang.com/2019/05/pytorch-internals/ A good basic thing to know is a tensor is a single dimensional array under the hood. At least as far as I know in my mental model - I have not yet looked at the code :-) Therefore "converting" a 2 * 5 * 10 tensor into a 2 * 50 one is an immediate operation, it doesn't have to scan. Just change some metadata.

Yes it's quite an unfortunate situation. Edward Z Yang has a canonical post on this: http://blog.ezyang.com/2016/07/what-template-haskell-gets-wrong-and-ra…

An even more fine-grained binary search technique for isolating compiler bugs is optimization fuel: http://blog.ezyang.com/2011/06/debugging-compilers-with-optimization-f…

Perhaps the proper programming term might be co-ETL or co-data-lake? (Where we use 'co' in the same sense as eg http://blog.ezyang.com/2012/10/duality-for-haskellers/ )

Nice! I was wondering if implicit parameters were equivalent to dynamic scoping. It seems to be the case. more on the topic: http://blog.ezyang.com/2020/08/dynamic-scoping-is-an-effect-implicit-p…

We also have "optimization fuel" in Rust, https://camlorn.net/posts/April%202017/rust-struct-field-reordering/ Which comes from http://blog.ezyang.com/2011/06/debugging-compilers-with-optimization-f… which cites a paper from 93.

Not quite, there were a bunch of other persistent data structures. Finger trees, PATRICIA tries, etc. And the perf was competitive. See http://blog.ezyang.com/2010/03/the-case-of-the-hash-array-mapped-trie/ for example. In there Haskell's IntMap (powered by a PATRICIA trie) just beats out Clojure's HAMT without transients. In practice they're probably more or less equal, because in order to make IntMap usable for arbitrary keys and not just ints, you need to have a hashing step first that generates an int from an arbitrary key, so you have the overhead of a single hash. Ah you're totally right RE transients. I don't know why I had the notion they were tied to HAMTs... Nonetheless I view it as one of the big improvements that Clojure brought to the scene.

Interesting, I learned this technique of having a third, synchronization intermediary with offlineimap and this helpful blog post: http://blog.ezyang.com/2012/08/how-offlineimap-works/

I had a horrible problem with space leaks when running weeks in production, didn't help that I used a pure Haskell db at the time. :) Here are some more info http://blog.ezyang.com/2011/05/space-leak-zoo/ It's been almost a decade since though. Maybe the compiler catches more nowadays.

For people who know the basics this article describes Pytorch in more detail: http://blog.ezyang.com/2019/05/pytorch-internals/

I was expecting this to be about fractional cascading which is a really clever trick: http://blog.ezyang.com/2012/03/you-could-have-invented-fractional-casc…