Distill

Distill was a new take at publishing research/ideas in deep learning in a visual way: https://distill.pub/ I love their articles and while it was hard to sustain, the quality of the ones in their are pretty good. They provide some tips and templates on how to develop such visual storytelling articles.

Anything from Setosa [0] is really good. It contains interactive, animated illustrations of several Machine Learning ideas. I _loved_ reading papers from Distill Pub [1] as they contained interactive diagrams. My most favorite one so far is the thread on Differentiable Self-organizing Systems [2]. I liked the lizard example very much as it is interactive, and lizards grow lost organs back. I think this is funny. [0]: https://setosa.io [1]: https://distill.pub [2]: https://distill.pub/2020/selforg/

As another recommendation, Distill is higher level, and has less topic coverage, but their article quality is fantastic: https://distill.pub/

Thanks for the link! I haven't used TikZ too much, maybe only a dozen or so times, but definitely am hoping to have something great for interactive diagrams. Will certainly keep this in mind as I add more features. I'd be interested in having a look at what are the most important and widespread types of diagrams and data visualizations in academic work and research, as I'd like to make that sort of thing better. I'm particularly inspired by the interactive visualizations on https://distill.pub , however I think those are painstakingly hand-coded in SVG+JS+CSS+HTML.

Great visualization! Is it based on https://distill.pub? See, e.g., https://distill.pub/2017/momentum/

GitXiv very much worth taking a look at if you're into this kind of thing: http://www.gitxiv.com/page/about edit: as is Chris Olah's Distill project: https://distill.pub/

Here's a ML one (linked to by the above site): http://www.r2d3.us/visual-intro-to-machine-learning-part-1/ Distill has some great ML articles that are kind of similar: https://distill.pub/

I'd suggest searching for and reading blog posts of people who are studying deep learning. Academic papers are very pretentious, they are the tools for scholars to declare victories. They are not the best materials to understand. http://www.wildml.com/ https://medium.com/mlreview https://distill.pub/

I'm also starting to follow people and communities that work with deep learning in new ways. Here are some of my favorites: [1] http://colah.github.io/ [2] https://iamtrask.github.io/ [3] https://distill.pub [4] https://experiments.withgoogle.com/ai

I see that this is the author of the excellent "Quantum Country" so perhaps the case being made here is obviously biased, for a good reason. I get a sense that the author seems to argue that most book authors are not conscious of the working memory of the reader. I don't think that is true. Many "good" non-fiction books I've read actually rely on space-repetitions throughout the text quite heavily. The author, to their credit, does note this for textbooks. Often concepts in long texts are referenced in a non-linear manner, either in expectation of ideas to come or in retrospect to draw parallels & new connections. Unfortunately, the onus here is on the reader to realize and extract the maximum juice out of this non-linear flow. The key idea of this post is to shift this burden to the "medium" and not the reader. Books can't handle this responsibility in their current form. I certainly agree with this. Distill [1] is pioneering this from the perspective of medium-reader engagement. Unsurprisingly, Michael Nielsen, a collaborator of the author of the post, is a member of the steering committee at Distill. :-) I love these new directions in dissemination of ideas! [1]: https://distill.pub

https://distill.pub is such a platform for articles about machine learning https://explorabl.es is a more general hub. > especially when you can tell that the 'teacher' is also the UI designer and developer in one and there's a very high level of quality and polish. There's nothing quite like it for learning a complex subject. Agreed. Unfortunately I think this is the inherent limitation to the explosion of such material (Bret's seminal essay [0] was almost a decade ago) - it takes an insane amount of skill, in mostly orthogonal disciplines, as well as a lot of time and effort to make it. Some tools attempt to make them a little straightforward to program, but due to the nature of it you can't really have a one size fits all solution that isn't just a general programming language + rendering engine. And there is no "explorables industry" to fund the production of those (unlike say, textbooks), so it ends up being mostly side projects for otherwise gainfully employed developers. It would be amazing to have eg. attempts at building an entire K-12 curriculum around explorable explanations. It's also the kind of work that probably loses in clarity and value when done by a team/committee rather than a single person dedicated to their vision. [0] http://worrydream.com/ExplorableExplanations/

I pretty much agree with you. I have a bachelor's degree in pure mathematics from a top 15 university -- I even published research as an undergraduate. Deep learning strikes me as an experimental science and even a subset of ordinary programming more than it does a subset of mathematics or statistics or statistical learning. I've read the entirety of the Deep Learning book by Goodfellow et. al. It's a fantastic book, but I was shocked at the lack of robust theory and the relative simplicity of the math. Even an introductory analysis textbook like Rudin is considerably harder, and Rudin is regularly tackled by undergraduates at any highly ranked university. The only reason to fret about this in my opinion is if you're a PhD machine learning engineer who doesn't want the field to open up to non-PhDs. I think data scientists are used to being able to say, "hey, if you don't have a PhD, you really can't do or understand what I do" -- deep learning represents potentially a huge culture shock to that attitude. But even the Google Brain research team has some non-PhDs now. I do think deep learning practitioners should learn the math. I just don't think there's actually that much math to learn. Certainly, if you read through the TensorFlow MNIST tutorial and you have no idea what cross-entropy is and you don't understand what the softmax layer is for, you need to go back to the basics. But these are concepts that anyone with any reasonable engineering degree can pick up relatively…

Fantastic article. For readers interested in this, let me point to the somewhat similar "Activation Atlas" interactive paper, published in the sadly now-defunct scientific journal Distill. https://distill.pub/2019/activation-atlas/

For a split second there I believed there is a new Distill publication! Their articles were the most inspirational and eye-opening resource on my beginnings of ML journey, the quality of visualizations definitely made lasting impact on my mental models of _what is going on_. My favourite goes definitely to The Building Blocks of Interpretability (https://distill.pub/2018/building-blocks/), those images landed in a lot of my university presentations and the dog made everyone immediately interested ;-)

Regardless of whether this means AGI has been achieved or not, I think this is really exciting since we could theoretically have agents look through papers and work on finding simpler solutions. The complexity of math is dizzying, so I think anything that can be done to simplify it would be amazing (I think of this essay[1]), especially if it frees up mathematicians' time to focus even more on the state of the art. [1] https://distill.pub/2017/research-debt/

The author of the Airfoil article has made several similar tutorials on various topics: https://ciechanow.ski/archives/ For machine learning, Distill.pub has some excellent hands-on tutorials. For example, here's one on momentum: https://distill.pub/2017/momentum/

Very nice post - thank you. This is how maths and stats should be taught. Reminds me a bit about https://distill.pub/

Loosely related, but also a great read: https://distill.pub/2020/circuits/zoom-in/

If you include deep learning in CS then https://distill.pub/ has a lot to offer in this category.

Not sure if this counts as a blog, but I really liked Distill: https://distill.pub/ The articles explain machine learning concepts in a very clear and sometimes interactive way.

Really cool seeing this & excited to really start playing with it. 2023 is going to be funky! It's pretty wild to see how quickly the space of Generative AI Media is coming along. I started a newsletter on the topic, called The Art of Intelligence (GPT-3 came up with the name) with the first post going out last Friday on the topic of how far are we from AI generated videos, and simulated worlds like the Holodeck given the rapid progress of these visual A.I. Thought y'all might find it interesting: https://artofintelligence.substack.com/p/dall-e-stable-diffusion-and-t… This type of progress also reminds me of a really lovely publication from 2017 in Distill.pub, on the topic of these A.I. enabled creation tools - I think y'all would enjoy seeing what folks were thinking even then: https://distill.pub/2017/aia/

This is cool. I’m wondering what the responsive part is — the example seems like it’s just latex2html output basically? I think it would be super cool to have more https://distill.pub style papers, with like interactive plots instead of static image files and stuff. Edit: I think the jargon I was looking for is “reactive”, not “responsive”

I hadn't heard about Works in Progress before, but looks like incredibly well researched content. Pretty stark difference between the writing and what you see on something like sfgate.com or cnn.com. I find it pretty awesome that the Stripe is successful enough to start branching out from core product into founder hobby areas. The cynical perspective is that this is not core to the product, but I find it kind of beautiful that the founders can do a bit of what is personally interesting to them with their company. This reminds me of another incredible resource for deep learning articles: https://distill.pub/, which unfortunately looks like is going silent.

I’m looking for collaborators for re-implementing “modern” machine learning and deep learning models/papers. Modern is in quotes as I’d actually like to focus less on the super recent, and more on those around ~5 years old, as the compute required is usually more feasible. As well as the implementation (which will be open sourced, well written and documented), I’d also like https://distill.pub/ style articles to go along with the implementations. I’d also like to get into algorithmic trading, but this is something I’m at the very early stages of researching into. If any of that sounds interesting, contact details are in my profile.

http://distill.pub/about/ It’s been done and it is amazing. This is the best journal in the world imho

We can understand how and why neural networks work. The issue is that obtaining this understanding requires a lot of effort. OpenAI invested in trying to understand how a simple CNN works. You can read their results here: https://distill.pub/2020/circuits/ > In the original narrative of deep learning, each neuron builds progressively more abstract, meaningful features by composing features in the preceding layer. In recent years, there’s been some skepticism of this view, but what happens if you take it really seriously? > InceptionV1 is a classic vision model with around 10,000 unique neurons — a large number, but still on a scale that a group effort could attack. What if you simply go through the model, neuron by neuron, trying to understand each one and the connections between them? The circuits collaboration aims to find out.

Yes, and follow the latest work of Alexander Mordvintsev, e.g.: https://distill.pub/2020/growing-ca/

This already exists, with a focus on machine learning: https://distill.pub/

For a gold standard of publishing see http://distill.pub/ Open access and not stuck in the dark ages with dissemination.

I feel like the papers published at https://distill.pub/ does a decent job of moving toward a more modern solution. There's no doubt that doing it this way often requires a lot more work, and maybe even a new set of skills to even make or visualize though. And it doesn't really incorporate the diff concept in any way atm, but I imagine it's up their alley at least.