Jean Niklas L'orange

I can't downvote, but might have as the first sentence is an over-simplication and misunderstanding -- particularly as laziness for collections has always been available in clojure.core. Clojure transducers offer an optimization orthogonal to collections best summed above with: "transducers allow you to define steps in collection processing _per item_ rather than having collection processing as a series of transformations of collections". Yes, transducers can be viewed as somewhat of an analog to the Haskell conduit library (as discussed here several years ago: https://hypirion.com/musings/haskell-transducers). However, I think the distractors coming from strongly typed languages are decidedly missing much of the generalization of the transducer model, particularly those conflating transducers exclusively with streams.

Thanks for clarifying. I think we're just using different definitions. The only real definition of an array I've encountered in work and school is that it's just a one dimensional collection of elements, usually of static size. A vector depending on context is usually the same thing as an array but you conveniently change the size dynamically. Lists can whatever you need it to be given the context, just needs to be sequential. Of course you can represent higher dimension structures by linearizing indices (x + row_size * y, etc). I think people are getting confused as most don't consider arrays to be arbitrarily dimensional without some scheme. Completely off topic but you've reminded me of this great article: https://hypirion.com/musings/understanding-persistent-vector-pt-1

See also the excellent 5-part blog post series by Jean Niklas L'orange: http://hypirion.com/musings/understanding-persistent-vector-pt-1

So persistent data structures are conceptually just an extension of the classic lisp cells: You can have a pointer to (a, b, c) and a pointer to (a, b, c, d) simultaneously, without using twice the memory or time. This can be extended to trees. And you can for example make an efficient tree-backed vector, with chunks of values at the nodes. With some thought you can even get good atomic properties so you can have safe parallel operations on the same persistent vector. Here's a good start about Clojure's persistent vector: http://hypirion.com/musings/understanding-persistent-vector-pt-1

That's still not quite right. It's not lazy evaluation that makes it feasible to "make a new tree" in Haskell (although it doesn't hurt), it's the fact that the new tree shares most of the state of the old tree and the data structures make the asymptotic time complexity of both update and retrieval very close to (but not quite) O(n). If you haven't looked at persistent data structures yet then I'd definitely recommend doing so because they are fascinating. A few people have written about Clojure's data structures and the following article looks like it gives a good introduction: http://hypirion.com/musings/understanding-persistent-vector-pt-1

The difference (like the author states) is that JS & family has mutable data by default, while Clojurescript defaults to immutable such. You have to specify that you want your vectors and maps etc to be transient (fancy pants word for mutable). You want immutability - think about it: how large % of your bugs are due to more than one procedure changing the value of one variable, and state getting out of sync? For me in my JS day job, it is definitely a majority of the bugs we create. We also code very defensively to protect us against this. In CLJS on the other hand, you'd have immutable data, and in tight loops where allocating objects would be expensive, you make your data transient, and then mutate it as much as you wish. Living your life with immutable values is a serious improvement. Not only to code, but honestly quality of life. Edit: Clojure doesn't return whole new data structures under the hood, but you never notice that. Here is a great explanation, prepare for (= (mind article) :blown) => http://hypirion.com/musings/understanding-persistent-vector-pt-1

Yes, mutating something at one place in memory is more efficient, because you don't need to allocate new memories. I don't know all to much about other functional languages, as I learned perl -> ruby -> javascript -> little bit C & Java & Go -> now doing Clojure. But I find Clojures collection data structures interesting. The vector (collection like lists or array) type look immutable, but under the hood are trees. When you append to a vector, you seem get a new vector returned. In reality, you added the new element as a node in a tree. Then just modified pointers to that the new and old version share almost all of the pointers & allocations. With simple arrays or lists, you would allocate every element anew. Idk if I can properly explain. Found this blog post very interesting and easier to follow: http://hypirion.com/musings/understanding-persistent-vector-pt-1 Personally, most things in business I find easily parallelisable. You mostly decouple the I/O parts of something with the logic parts of it. But yeah I still have much to learn. Thanks for the link! Interesting :)

There is some prior work on mitigating the performance cost of immutability that you might be interested in. For example, Clojure's persistent vectors allow fast modifications without destroying the original vector, because internally they're wide trees rather than just linear arrays of memory. This allows for assignments to be implemented without a copy of the full vector. https://hypirion.com/musings/understanding-persistent-vector-pt-1

What's your use case for system-versioned tables? You could use some extensions like Periods that support bi-temporal tables: https://wiki.postgresql.org/wiki/Temporal_Extensions Or you could use triggers to build one: https://hypirion.com/musings/implementing-system-versioned-tables-in-p…

I suppose it's quite similar to the idea of Bagwell's persistent data structures[1] — those are particularly useful for lock free coding and with a high enough branching factor could be not as much overhead as a linked list or just plain copying. [1] https://hypirion.com/musings/understanding-persistent-vector-pt-1

After taking a stab at some more functional approach with generics in Go — https://github.com/gtramontina/go-extlib — I do agree that some of it does feel shoehorned. Although for some other constructs, it feels quite nice. As I mentioned on the readme of the linked repository, this post https://hypirion.com/musings/type-safe-http-servers-in-go-via-generics presents pros and cons nicely.

Copy on modify would work, certainly. The term I was forgetting when I wrote that is “persistent” data structures, but I’m not sure whether it’s practical to implement persistent structures in a non-functional language like Python. https://hypirion.com/musings/understanding-persistent-vector-pt-1

This question merits a detailed answer that I can't provide right now, but here are some pointers: https://hypirion.com/musings/lockfiles-are-not-the-only-option https://stackoverflow.com/questions/44521542/why-doesnt-gradle-or-mave…

The implementations of immutable data structures in most concurrency aware functional languages are smarter than that. Take Clojure as an example: "All of the Clojure collections are immutable and persistent. In particular, the Clojure collections support efficient creation of 'modified' versions, by utilizing structural sharing, and make all of their performance bound guarantees for persistent use." https://hypirion.com/musings/understanding-persistent-vector-pt-1

There are better alternatives to linked lists for this use case. 1. Clojure uses wide balanced trees as its general-purpose ordered collection.[0] These have probably already been adapted for Haskell, and they offer persistence and fast random access and some degree of cache-friendliness. In my opinion it makes better trade-offs as a general-purpose structure. 2. By passing an &[usize] in Rust, or a const size_t* in C (i.e. by making g accept a read-only pointer), you can be guaranteed that g doesn't modify the array. [0]: https://hypirion.com/musings/understanding-persistent-vector-pt-1

If you can "submit" your change back to the original datastructure then the original datastructure is not immutable, right? Here's a nice explaination about how the persistant immutable datastructures work: https://hypirion.com/musings/understanding-persistent-vector-pt-1

For those like me who are unfamiliar with Relaxed Radix Balanced trees, here's the paper presenting the algorithm: RRB-Trees: Efficient Immutable Vectors (2012) https://infoscience.epfl.ch/record/169879/files/RMTrees.pdf EDIT: For further reading: Improving RRB-Tree Performance through Transience (2014) https://hypirion.com/thesis.pdf Article: https://hypirion.com/musings/thesis Code: https://github.com/hyPiRion/c-rrb

Someone on HN mentioned the 'magic of ____ arrays' or something like that where search, insertion, merge, etc. were all constant time operations... Ah! Clojure's persistent vectors: https://hypirion.com/musings/understanding-persistent-vector-pt-1 I would use stacks to sort. The book was about 1000 pages? I'll do about 10 stacks. Then figuring out which stack to put a given page in is fast. (Abstract this and you wind up with a binary tree, which is what made me think of the persistent vectors above.)

It's hard. http://hypirion.com/musings/understanding-persistent-vector-pt-1

I'm not sure how it's implemented in Ferret but Clojure's data structures are implemented as trees so there's structural sharing that happens when a vector or hash map is updated or when a new version is produced. So the whole data structure isn't copied but only the relevant parts are added / updated in the tree and the rest is shared with the "new" data structure. Mutable data structures are still more memory effective but I'm assuming its fine since the author has a blog post on using ferret on an Arduino Uno which has 32k of ram. https://nakkaya.com/2017/02/15/bare-metal-lisp-rc-control-using-ferret… Clojure data Sturucture links: Persistent Vectors http://hypirion.com/musings/understanding-persistent-vector-pt-1 Video with Rick hick on them https://youtu.be/wASCH_gPnDw?t=1742

> cloning the entire structure for every applied function sounds like it will be slow? is that not what you will have to do in practice? In short, no - as an example, when an element is added to an immutable map, a big part of the newly produced map is structurally shared with the old one (check out [0].) As for the ordering, you could have a list of geometry -> geometry functions. You could then take the initial geometry and apply the list of functions in order, or a permutation of it, or drop some functions, or apply some others multiple times, and so on. But anyway, it's just an idea - thinking in terms of atomic geometry transformations instead of alternations may or may not be easier in more complex scenarios, I don't know :) [0] http://hypirion.com/musings/understanding-persistent-vector-pt-1

Here's an attempt: http://hypirion.com/musings/haskell-transducers

> Conceptually, it does this by constructing a new copy of the result for every addition, since conj is not destructive That's not how persistent data structures work http://hypirion.com/musings/understanding-persistent-vector-pt-1 > why does efficient set union of two collections require one of the two collections to support fast membership tests I haven't thought much about this but it seems intuitively true to me. You want to take the smaller set, and add each item to the larger set if it doesn't already exist. That last part is why you need a fast membership test. If you're still unconvinced, try loading up two similarly structured database tables with a million rows and get the union of them (with and without indices). The indexed one will be orders of magnitude quicker.

I was introduced to http://hypirion.com/musings/understanding-persistent-vector-pt-1 not too long ago on HN. I don't use clojure so I was paraphrasing my somewhat limited understanding. Thanks for the clarification. That part about speed was more about my general frustration with the "immutable is fast" and "mutable is slow" meme that I hear too frequently. While that can be the case, it isn't always 100% true. Your language and how you use it can play a huge part. I'm hoping to learn more about clojure data structures in the coming months, so much good stuff.

Some of the languages have gotten better! Clojure, for example, has smart data structures that share elements to speed up operations and minimize copying: http://hypirion.com/musings/understanding-persistent-vector-pt-1

This is probably a good start: http://hypirion.com/musings/understanding-persistent-vector-pt-1

Here's an explanation of one of the underlying datastructures that make this library reasonably efficient: http://hypirion.com/musings/understanding-persistent-vector-pt-1

Some time ago, hypirion wrote about how persistent vectors are implemented in Clojure. http://hypirion.com/musings/understanding-persistent-vector-pt-1 There are actually several different implementations of the core data structures: https://github.com/clojure/data.finger-tree https://github.com/clojure/core.rrb-vector Of course, the short answer is that Clojure isn't using classic linked lists, and that regular Java collections can be used and created pretty easily. [0] [1] [0] https://stackoverflow.com/questions/4313505/converting-clojure-data-st… [1] http://blog.raek.se/2011/01/24/executors-in-clojure/

> Immutable arrays, where the entire array must be copied with the change of one element, are an example of that. Only in a naive implementation. Clojure, for example, has a persistent vector that only requires O(log32 n) copying, which grows so slowly as to be effectively O(1). See: http://hypirion.com/musings/understanding-persistent-vector-pt-1

This is interestingly reminiscent of swearjure[1], clojure without alphanumerics. The IRC log in that post is very enlightening to see how the idea evolved into something absolutely frightening. I would like to see the history of JSFuck. 1. http://hypirion.com/musings/swearjure