Michael Malis

If you want to follow along, I've been writing about it on my blog (https://malisper.me/) an you can follow the github repo here: https://github.com/malisper/pgrust

Its funny, as I am still not good at step debugging in lisp, but https://malisper.me/debugging-lisp-part-1-recompilation/ is a great example of basically the same concept there. Really, that whole series was amazing fun to read through.

I would love to see a section about debugging, something like the series at https://malisper.me/category/debugging-common-lisp/ but I suspect it would be difficult to made it general enough (the guide above targets SBCL, Emacs and Slime)... P.S. I should mention that I live in Emacs, but I know there are people that wouldn't touch it with a ten foot pole

I find it funny and also ironic that a lot of modern languages re-discover Lisp's amazing features decades later. The other day I was infuriated to see my Python program halt after 9 hours of API calling to my own home server. The API calls were similar (calling an LLM with a pre-defined prompt template, constrained using grammars). I wanted to save the state of the program and exit it before running the remaining iterations separately. So I needed a way to modify the current running Python code and inspect the variables. I couldn't figure out a way to "dig into" the running Python process and save the state and/or fix it. So I lost 9 hours worth of work. A couple days later, I saw this: https://malisper.me/debugging-lisp-part-1-recompilation/ The fact that CL had this feature built into the language decades ago just blows my mind. Not to mention numerous other features (like the most powerful macro system there is), etc.

Maybe start with https://lispcookbook.github.io/cl-cookbook/emacs-ide.html The 'magic' though isn't really that emacs dependent; if you like vim (like I do) you can use vim and still get that amazing experience. https://lispcookbook.github.io/cl-cookbook/editor-support.html has some details on alternatives. Anyway, the magic or synergy is really a combination of: CL itself being designed to support interactive development, a little server you embed (typically automatically if starting from an editor) in a running CL program called Swank, and a standard set of commands to talk to that server (and the underlying Lisp) and interface with your editor called SLIME (https://slime.common-lisp.dev/doc/html/). The basics of the commands do things like send code over for compilation and redefining things, running things, responding to errors, debugging things, jumping to definition locations.. things you'd normally expect an IDE to do, except because of CL itself so much of that is part of the language and not a separate IDE. Some more editor-specific things come into play that some people really love, and while they may be best done in emacs they're not unique to emacs either. (Example: suppose you messed up the order of your arguments to a function, so it's written like (my-func (list something) some-other-arg) -- editors have support to swap those two arguments in one go, rather than having to backspace and retype or whatever. I personally don't find such features that magical,…

> As someone else mentioned, that sounds more like a work environment problem. Have you ever used any API by, say, a major cloud provider? For example biquery and google docs are both a buggy mess despite being high profile, long established products. Using libraries like pandas is an exercise in figuring out by trial and error how to get it not to corrupt your data and which of the recommended ways of doing things don't involve a 2 orders of magnitude performance defect. I'm in fact mostly using "repl"-driven development for these types of problems, but of course it's inferior to what I mean by debugger driven, a crippled subset to be precise. The scare quotes are because most languages don't have real read-eval-print-loops either, but something less powerful. Unfortunately to the best of my knowledge there are basically only two languages which support this: Common Lisp and Smalltalk (maybe factor or something else really fringe also does) and I haven't used either in years. What I mean by debugger-driven is that you can literally write your whole program without ever restarting it, from the debugger. By contrast python can't even properly reload a changed module definition. Interestingly it's hard to google good explanatory links, but the point is that you can write some code, run into a problem, and fix the problem right there, in the debugger without aborting the current execution or unwinding the stack. Common Lisp and Smalltalk both have a bunch of unique…

Common Lisp is a very pragmatic language, complete with goto, if you want it. This is for the purposes of efficiency rather than because you should use it yourself. Suppose you wanted a really efficient numerical library written in Common Lisp, you could do it. But you wouldn't want to use map/reduce/remove/etc which are all linear time on the sequences they operate on (which can be lists or vectors). So clean looking code turns out to have poor performance because you have multiple O(n) operations, introducing large constants or if they're nested turns them into quadratic or worse operations. For instance, if we wanted to compute dot product we could, in CL, do: (defun dot-product (v1 v2) (reduce #'+ (mapcar #'* v1 v2))) That's O(n) which is the "best" we can do for dot product. But it actually contains two iterations. In C, you'd have: double dotproduct = 0.0; for(int i = 0; i < vector_length; i++) { dotproduct += v1[i]*v2[i]; } While for this small example the difference in performance is minor, in the case of a library making many calls to these functions that double loop would add up. And for more complex operations you're introducing higher constants (or worse) by using the clean looking Common Lisp code. Enter something like Series Series: https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node347.html See here for a demo of how it tranforms functional code to imperative: https://malisper.me/loops-in-lisp-part-4-series/ So you still (as the user)…

Pretty much the only GUI that's StumpWM and not an application is a mode-line, notifications and help. My setup looks like this currently: https://klibert.pl/statics/images/stumpwm.png and here is my mode-line config: https://klibert.pl/posts/stumpwm-better-modeline.html Anyway, StumpWM is great, but not because it's a tiling WM. It's absolutely fantastic because it's a live environment: you can attach a Lisp shell to your running stumpwm instance and immediately get a full-blown programming and debugging environment. See here: http://malisper.me/2015/07/07/debugging-lisp-part-1-recompilation/ (and subsequent posts in the series) for a good introduction to what it really means to work inside a live environment. The problem with this is that it's very, very hard (if not impossible) to show the dynamic, live environment with still images/screenshots. I think it would be good to record a screencast showing a bit of programming and debugging in StumpWM. I wanted to do this a few times in the past, but by the time I remembered I should start recording I already fixed whatever I wanted to fix...

Please consider comparing pg_clickhouse to pgrust. https://malisper.me/how-we-made-postgres-hundreds-of-times-faster-the-…

While I do like pgrust's license, I do feel like it's kind of wrong to port in such a direct way and change the license. I guess this isn't a fork, but it kind of is? It looks like according to this post [0], they did a Claude Fable + Opus re-write. I know that legally this is seemingly a valid way to do things and avoid copyright, but it feels wrong to me. I don't even necessarily think that my feelings are correct, but standing on the backs of giants and using an LLM to "reimplement" the code (not a clean room implementation) feels like it's it _shouldn't_ be a valid way to avoid copyright or allow for license changes. Now, I don't know how MIT -> AGPL re-licensing specifics work, but still. [0] https://malisper.me/postgres-in-rust-regression-suite/

What do you mean, "No"? Letting an LLM loose on Postgres is exactly what happened here: https://malisper.me/pgrust-rebuilding-postgres-in-rust-with-ai/

For other interested spectators, I found this post by the author to be a good example of architectural changes in pgrust (along with motivations): https://malisper.me/the-four-horsemen-behind-thousands-of-postgres-out…

(I'm working with malisper on this), we are now focusing on improving many things about postgres! Some we have written about before [0], and we have much more in mind too. Malis wrote another comment about analytical workloads being 300x faster now than postgres for a version we're working on right now Aiming for postgres compatible database with a 2026 architecture [0] https://malisper.me/the-four-horsemen-behind-thousands-of-postgres-out…

> the code that is currently executing can be manipulated live Typically when people say this about Lisp, they mean that they can redefine functions and objects live, having all the code adapt to use the new definitions, or when their code hits an error, they can not only redefine functions and objects live, but also examine the state of the call stack and change the values of variables live, and then let the code run again. This[0] is a clear explanation that I found. There's also a video[1] that shows the same thing. However, I don't know if those apply to Scheme. Also, I am pretty sure it is unrelated to Lisp being homoiconic, and that it's more about it being dynamically typed and interpretable. [0]: https://malisper.me/debugging-lisp-part-1-recompilation/ [1]: https://youtu.be/jBBS4FeY7XM

https://wiki.postgresql.org/wiki/Loose_indexscan > "Loose indexscan" in MySQL, "index skip scan" in Oracle, SQLite, Cockroach (very recently added), and "jump scan" in DB2 are the names used for an operation that finds the the distinct values of the leading columns of a btree index efficiently; rather than scanning all equal values of a key, as soon as a new value is found, restart the search by looking for a larger value. This is much faster when the index has many equal keys. I ran into a case in a hobby project where a really basic GROUP BY query was performing was worse than it should. I was surprised that https://malisper.me/the-missing-postgres-scan-the-loose-index-scan/ it's been known "missing" since 2017.

> some kind of in-place upgrade system For Common Lisp, here's a toy example[1], and spec[2]. [1] https://malisper.me/debugging-lisp-part-3-redefining-classes/ [2] http://www.lispworks.com/documentation/lw70/CLHS/Body/04_cf.htm

This might help https://malisper.me/category/debugging-common-lisp/ I am not aware of another lang/platform that can offer this kind of flexibility except may be Smalltalk or Erlang, but then they don't have the homoiconicity.

This might be interesting for you: https://malisper.me/category/debugging-common-lisp/

not all lisps have the same level of repl-driven development. clojure seems to be good, but it has nothing on common lisp. common lisp has this designed into it anyway, check out https://www.youtube.com/user/CBaggers/videos https://github.com/rabbibotton/clog/blob/main/LEARN.md edit: https://malisper.me/category/debugging-common-lisp/ (from another poster)

Admittedly I’ve never seriously investigated Clojure debugging tools so it might be that Clojure already supports something similar; but the Debugging Common Lisp[0] series might showcase how Common Lisp (with signals, restarts, mechanisms for redefining classes, p and more…) and SLIME aids debugging in how other ‘usual’ languages don’t. [0]: https://malisper.me/category/debugging-common-lisp/

Check out "An Algorithm for Passing Programming Interviews": https://malisper.me/an-algorithm-for-passing-programming-interviews/ of course, it's important (for better or worse) to just grind out a representative sample until you understand most common patterns, e.g: https://seanprashad.com/leetcode-patterns/

Thanks for your extensive write-up. PHP's definitely not my thing, but I appreciate your view of the process to iteratively dig (and dig, and dig...) into the details. By the way, if you're looking for instant introspection, changes and carrying on, you should have a look at what you get with Common Lisp. You're able to redefine code on the fly, and continue from where things stopped. This blog post series goes into debugging, specifically with SBCL (one implementation), Emacs and an add-on called Slime. https://malisper.me/category/debugging-common-lisp/

Surprisingly little—not many comments but they're pretty good and https://malisper.me/building-fizzbuzz-fractran-bottom/ was a good article. 2017: https://news.ycombinator.com/item?id=14202367 2016: https://news.ycombinator.com/item?id=11894141 2011: https://news.ycombinator.com/item?id=3119937

I have found this series of articles to be very helpful: http://malisper.me/debugging-lisp-part-1-recompilation/

I've worked with common lisp, clojure, python and javascript, and common lisp debuggers are just hands down better in everyway. Check this series for what should be a basic debugging experience by now http://malisper.me/debugging-lisp-part-1-recompilation/

http://malisper.me/2016/06/11/building-fizzbuzz-fractran-bottom/ 1st paragraph: "In this post, I am going to show you how to write Fizzbuzz in the programming language Fractran. If you don’t know, Fractran is an esoteric programming language. That means it is extraordinary difficult to write any program in Fractran. To mitigate this difficultly, instead of writing Fizzbuzz in raw Fractran, what we are going to do is build a language that compiles to Fractran, and then write Fizzbuzz in that language."

The main thing I like is basically you always have a debugger present. When you run into some issue you don't have to restart your program in a debugger and recreate it. Even running in production you can remotely connect to your program and inspect everything about it. Another thing about this is that when exceptions happen you're dropped in the debugger before the stack is unwound, and you generally have options to recover and continue execution without restarting everything and losing your state, which imo is the only sane choice for a dynamic language where you know you will be getting some type errors at runtime in development at least. Everyone knows the frustration of starting some python script, having it run for 10 minutes only to have it crash on some exception that you have to fix and restart. Also I'm not entirely sure why, I think because CL does more checks at compile time, and there is more type checking, I tend to get fewer runtime errors and they're closer to the source of the problem and easier to debug than in other dynamic languages. Then there are questions like what happens to existing instances of classes when you add fields or otherwise change them, which CL handles (https://malisper.me/debugging-lisp-part-3-redefining-classes/), but other language REPLs don't. Regarding the syntax of those type declarations, it's true I also thought CL was verbose and ugly at first, but it is also extremely flexible through macros and reader macros. At the end of…

My relatively amateur take is that the REPL and the debugging experience seem powerful. I would like to know how they compete with other languages. The REPL is the center of everything and it enables you to change functions on a running program. Tracing a function (shows function arguments on every call) is a simple as calling trace(function-name). If a program crashes, it does not really crash... it enters some debug mode which offers possible resolutions, including change the function that failed. The REPL can trivially show you the assembly code of individual functions. You can also add declarations for each function with hints for the compiler (and then check the size of the resulting assembly). I believe this series of articles highlights some of the debugging features. https://malisper.me/debugging-lisp-part-1-recompilation/ There is also a story called "debugging code from 60 million miles away". Other things that I did not manage to explore yet are the macros, which allows you to create your own domain specific languages. My general impression so far is that it is a really powerful language, for lonely hackers. :)

Here's an article that outlines how recording yourself write code can help with improving productivity: https://malisper.me/how-to-improve-your-productivity-as-a-working-prog… > Watching Myself Code > One incredibly useful exercise I’ve found is to watch myself program. Throughout the week, I have a program running in the background that records my screen. At the end of the week, I’ll watch a few segments from the previous week. Usually I will watch the times that felt like it took a lot longer to complete some task than it should have. While watching them, I’ll pay attention to specifically where the time went and figure out what I could have done better. When I first did this, I was really surprised at where all of my time was going. > For example, previously when writing code, I would write all my code for a new feature up front and then test all of the code collectively. When testing code this way, I would have to isolate which function the bug was in and then debug that individual function. After watching a recording of myself writing code, I realized I was spending about a quarter of the total time implementing the feature tracking down which functions the bugs were in! This was completely non-obvious to me and I wouldn’t have found it out without recording myself. Now that I’m aware that I spent so much time isolating which function a bugs are in, I now test each function as I write it to make sure they work. This allows me to write code a lot faster as it…

For Smalltalk, look at Pharo. For Lisp: http://malisper.me/2015/07/07/debugging-lisp-part-1-recompilation/ http://malisper.me/2015/07/14/debugging-lisp-part-2-inspecting/ http://malisper.me/2015/07/22/debugging-lisp-part-3-redefining-classes… http://malisper.me/2015/08/05/debugging-lisp-part-4-restarts/ http://malisper.me/2015/08/19/debugging-lisp-part-5-miscellaneous/