Mike Innes

Julia is great for numerical computing, but it's not as good as it can be, even in that area. Widening the community's base of expertise will help. I'll just list a few things that come to mind. Many Base functions are currently untyped, which means it's hard to abstract out the key properties that enable reuse, compiler optimizations, and parallelism. See [0] for a language expert's take on numerical computing, which involves defining traits like associativity that enable automatic parallelism by default. A stronger culture of functional programming could make your code faster and easier to understand. It's often (though not always) easier for tooling to optimize pure functions over immutable data structures (and arrays). Arrays are currently mutable, and there is too much emphasis on mutating functions. The iteration protocol can be made more memory-efficient for large collections and simpler, following Rust's implementation [1]. Macros are useful for high-performance computing [4], and Julia's macros can be made more composable [2]. In general, systems languages don't make language decisions lightly. They have committees, discuss how other languages do things, make proposals. This allows more perspectives on each decision. That would be an improvement over the more ad-hoc style of Julia development, as long as Julia can avoid adding every possible feature, which is a risk of expanding the decision-making body [3]. [0] https://www.youtube.com/watch?v=EZD3Scuv02g [1]…

Indeed, the latter problem is the same as implicit traits. For the former, you can absolutely do it without having object-orientation or anything like that. Simply lower the iterator protocol to ``` itr = iterator(x) while (i = next(itr)) !== nothing # stuff end ``` Then `itr` can be mutable, even when `x` isn't. This is what most modern languages does, and it solves all the problems I mentioned, since the state is stored in the mutable iterator, while at the same time, `x` can be unaffected. Read more here: https://mikeinnes.github.io/2020/06/04/iterate.html

What’s the difference of NuMBA CUDA and Pytorch or similar? If you’re doing custom kernels you should take a look at the Julia library CuArray [1] and generic kernels [2]. I really like that I don’t have to dig into C++ and deal with all of the memory and kernel management. 1: https://github.com/JuliaGPU/CuArrays.jl 2: http://mikeinnes.github.io/2017/08/24/cudanative.html

Coincidentally enough I'm neck deep in this area for the last month (I haven't read this paper but I've read others from the group). There are a bunch of things around this that are related and look vaguely the same: abstract interpretation, staged execution, symbolic execution and partial evaluation. I'll give a use case (the use case I'm interested in) that might help to illustrate the point: dynamic neural nets. How do you optimize neural nets that have control flow and dynamic shapes (variable size tensors)? Most (all?) hardware fast paths (GPUs, accelerators, SIMD registers) have hard (parden the pun) requirements on memory layout; e.g, if you want to use vector/SIMD units on CPUs effectively you have to align and chunk your data to fit into the fixed width registers. So for nets that have dynamics it's pretty hard generate machine code for using these things - most people pad to some "good enough" width and pay the price on throughput. All of these techniques speak to this problem (among others obviously) by inferring as much as possible at compile time and kicking the can to runtime for the rest. Partial evaluation, the one I'm most familiar, propagates, usually on an IR, as much statically known information as it can infer (depending on the quality of the implementation), and simplifies thereby (think things like substituting lengths of lists to simplify comparisons). After the partial eval loop hits a fixed point (no more changes discovered) you have a…

Building a relational lang (https://github.com/Tablam/TablaM) I allow myself to get derailed in how provide a nicer "linq/iterator" protocol that work inside rust and outside in the lang (so, how I write them in rust is close to what the user could write in the lang). The regular iterator protocol, as today in rust, make hard to do stuff like JOINS, GROUP BY and other fancy stuff (because you need to decompose the computation in a partial state machine. This is hard even for a developer, impossible to ask for a regular data user). Also, you need to duplicate all that for async (with streams) and other abstractions... I'm now trying to understand transducers (https://www.reddit.com/r/rust/comments/gqiyej/potentials_advantages_of… and stumble upon effects: http://mikeinnes.github.io/2020/06/12/transducers.html that look to my the clean way I wish to use. But what look easy in python/f#/etc in rust is HARD to do. So I'm in a kind of limbo :)

Just complementing the other reply, there is a small article about how you can implement a source to source AD in Julia [1]. Basically Julia has a special type of macro called generated function [2] which instead of executing during the AST lowering phase (when the compiler still didn't evaluate the symbols) it executes during the final step of compilation (when type inference already ran and you have all the exact types), and in that function you can return either the AST or Julia's SSA IR directly (which is good for AD since it closely resembles the execution graph since it avoids mutability). And you can also inspect the IR of any function call and manipulate it within the language [3], so you can recursively create the tape entirely at compile time. [1] http://blog.rogerluo.me/2019/07/27/yassad/ [2] https://docs.julialang.org/en/v1/manual/metaprogramming/index.html#Gen… [3] https://mikeinnes.github.io/IRTools.jl/latest/#Evaluating-IR-1

> With your linked spiral code, I can't even really tell where the algorithm starts and where the setup code ends - which of course might easily be attributed to my unfamiliarity with Spiral ;) Heh, I've noticed that the setup code tends to come out longer than the actual kernels. It is inside `kernel = cuda`. Spiral is indentation sensitive like Python and F#. The stuff inside {} is just module creation, think of it like tuples with named fields. > Do you have any benchmarks for the softmax kernel? If that kernel has optimal performance, it would be quite interesting. If it's sup par, it looks much longer than a simple version. No, I've yet to actually benchmark it. It really depends on how good of a job NVCC does with the generic sequential reduce kernel. I'll do an in depth analysis when I am done with all the neural network work that I am doing currently which might take a while. https://github.com/mrakgr/The-Spiral-Language/blob/7ecd30bdf36e8f59dd8… You can see how it is implement here for the forward and backward parts. https://github.com/mrakgr/The-Spiral-Language/blob/7ecd30bdf36e8f59dd8… In the actual cost function, it is a bit different since I fuse the forward and the backward parts. It is a bit ugly because of all the type checking for whether the argument is a dual, but that is all done at compile time. >…

>GPU programming without macros looks like then take a peek at this: I'm not sure if you missed my examples, or if there is some other missunderstanding going on ;) GPU code in Julia works 100% without macros, but you are free to use them, so people do ;) Also, you can pretty much write the gpu kernels like pseydo code, not sure how much simpler it can get. With your linked spiral code, I can't even really tell where the algorithm starts and where the setup code ends - which of course might easily be attributed to my unfamiliarity with Spiral ;) I wrote an article some time ago about generic programming with Julia's gpu infrastructure: https://techburst.io/writing-extendable-and-hardware-agnostic-gpu-libr… If you can make concrete examples of how things can be simpler than this, I'd be delighted to hear them :) Cuda only (for no reason actually) and with lots of macros, here is another article about gpu programing in Julia: https://mikeinnes.github.io/2017/08/24/cudanative.html Do you have any benchmarks for the softmax kernel? If that kernel has optimal performance, it would be quite interesting. If it's sup par, it looks much longer than a simple version.