Jeff Preshing

Jeff Preshing was a longtimer at Ubisoft, and developed for Corel before that in their heyday. His blog makes for good reading: https://preshing.com/

For low-level software, these are great: https://eli.thegreenplace.net/ https://travisdowns.github.io/ https://preshing.com/

Great shoutout to a Montreal hacker called Jeff Preshing http://preshing.com/

Disclaimer: I haven't implemented this yet, this is based on skimming the paper and what I know about lock-free structures. Your run-of-the-mill lock-free code basically operates under the premise of "keep trying until something sticks." For example, with only CompareAndSwap you can implement lock-free bit-field operations as: do { oldValue = Read(variable); newValue = oldValue | 0x01; } while ((CAS(variable, oldValue, newValue) & 0x01) == 0); You can see that we have to keep looping until the new value (which includes the flag) "sticks." The race is allowed to occur between the Read() and the CAS(), where CAS() is used to determine if a race occurred. CAS will check the current value of the address and update it only if it matches (as an atomic operation on the CPU). The worst case here is O(∞) - if another thread always wins, this loop will never exit. The run-of-the-mill lock-free stack code looks quite similar to this (you're basically updating the head until it sticks). This means that in certain circumstances, lock-free code can actually be slower. I have run into this myself[1]: that code uses locks because I couldn't figure out a faster lock-free implementation (that code is 30x faster than my best lock-free effort). There are two operations on a stack: Push() and Pop(). Classically, these both compete to write the head (write-contention). The advantage with this algorithm is that only Push() competes to write the head so you're decreasing the…

I too am a beginner with concurrent programming—it's a difficult topic to get your head around at first. Some resources that I've found super helpful: - Dmitry Vyukov's Lockless Algorithms: https://www.1024cores.net/home/lock-free-algorithms - Jeff Preshing's blog (worth exploring all adjacent articles): https://preshing.com/20120612/an-introduction-to-lock-free-programming… - Bartosz Milewski: https://bartoszmilewski.com/2008/12/01/c-atomics-and-memory-ordering/ - Memory Barriers: a Hardware View for Software Hackers—P.McKenney: https://www.researchgate.net/publication/228824849_Memory_Barriers_a_H… I hope you find some of these useful. I've re-read Paul McKenney's paper every 2-3 months in an attempt to get this stuff to stick! :)

> And I still reckon that it’s closer to 5-6x than it is to 5or6 orders of magnitude. The geometric midpoint (the only one that makes sense here) between the high end of 5-6x and the low end of 5-6 orders of magnitude is about 774×. The best info I can find (unsurprisingly, direct comparisons of performance of long-separated-in-time processors in aggregate is hard to find) is that between 1995-2011, integer performance increased somewhere well over 128 times but less than 256 times and floating point significantly more than 256x, and was trending in the last several years of the period to increase at a pace of 21% per year for both. [0] Seems likely to be close to or significantly past the 774× level between 1993-2023. [0] https://preshing.com/20120208/a-look-back-at-single-threaded-cpu-perfo…

You are personally attacking me for no reason. What does my other comments has to do with this comment? I don't care whether you believe me or not. This is a fake news at it's best. Regarding my comments, [1] That lady is self marketing guru. It's confirmed everywhere even Reddit. [2] Modern C++ is simpler than TypeScript, Go, Rust. Modern C++ become more like Python. https://preshing.com/20141202/cpp-has-become-more-pythonic/ https://web.archive.org/web/20131015192353/http://cpp-next.com/archive… [3] The comment about Christ is a Joke. Can't I make Jokes on HN?

There are great deterministic random number generators, like: https://preshing.com/20121224/how-to-generate-a-sequence-of-unique-ran… Can be used to shuffle a billion points per second on the GPU with sufficient randomness for some use cases.

Yup. With the ability to add a separator (like a space, -, .) I'd switch over to this. Currently I use and recommend https://preshing.com/20110811/xkcd-password-generator/

I've seen it mentioned in several memory model texts that, while ARM's model is more "relaxed" than x86, the DEC Alpha's model is even more relaxed. For example: [1] says "The venerable DEC Alpha is everybody’s favorite example of a weakly-ordered processor. There’s really no mainstream processor with weaker ordering", and [2] says "Some CPUs (such as i386 or x86_64) are more constrained than others (such as powerpc or frv), and so the most relaxed case (namely DEC Alpha) must be assumed outside of arch-specific code." [1] adds, specifically, "[data dependency ordering means that] if you write A->B in C/C++, you are always guaranteed to load a value of B which is at least as new as the value of A. The Alpha doesn’t guarantee that." [1] https://preshing.com/20120930/weak-vs-strong-memory-models/ [2] https://www.kernel.org/doc/Documentation/memory-barriers.txt

I'm actually surprised no one mentioned these two Eli Bendersky http://eli.thegreenplace.net/ Jeff Preshing http://preshing.com/

Modern C++ can be very pythonic. Still not as easy as writing Java in IntelliJ but CLion is becoming quite good (if you succumb to CMake). People write shit code in every language; C++ has my favorite abstraction to performance ratio. http://preshing.com/20141202/cpp-has-become-more-pythonic

I recommend Robin Hood open addressing scheme. It's particularly trivial to implement, and because it relies on linear probing, so it's memory-cache friendly. See more here: http://www.sebastiansylvan.com/post/robin-hood-hashing-should-be-your-… Jeff Preshing has written many interesting blog posts about the subject. For example, see his blog post on Leapfrog Probing( http://preshing.com/20160314/leapfrog-probing/ )

Moore's law "translates" to transistor count. Any other interpretation is your own. I couldn't care less about clock frequency. I care about operations per second, and these are following the numbers I referenced. See: http://preshing.com/20120208/a-look-back-at-single-threaded-cpu-perfor…

They are only better for some workloads and not others. e.g., they are excellent for accessing data in-order, but are worse than a very simple hash table for random access: http://preshing.com/20130107/this-hash-table-is-faster-than-a-judy-arr… (yes, this hash table is vulnerable to timing attacks; point is, for many workloads Judy brings in considerable complexity but is actually inferior to other solutions).

On the contrary, _not_ expecting at least 20% per year (more for generations farther apart) is actually only a recent take for CPUs [1][2] which formed in what I like to call "the great stagnation" of the mid 2010s where AMD bombed, ARM was still a low performance mobile play, and Intel didn't have any stiff competition. After that things have started to pick up a little again now that there is innovation on said fronts [3] I remember there was even a period Moore's law was commonly conflated with "a doubling in performance every 2 years" (implies people being used to an increase of at least 41% per year) instead of a doubling in transistor count. Because Intel's tick-tock model started ~5 years prior to the great stagnation people commonly claimed it's the end of Moore's law and that's why we have tick-tock's now. Of course these days it's common knowledge that Moore's law (the actual transistor count version) has been holding steady all these years still and that the performance lull wasn't related to hitting and innate technical scaling barriers. Marketing departments are definitely going to be marketing departments but they aren't the origin of the idea CPUs can have more than minor increments in performance each year. [1] https://preshing.com/images/integer-perf.png [2] https://preshing.com/images/float-point-perf.png [3] https://i.imgur.com/HlGbJE9.png (note this chart is linear not log)

I was curious so I downloaded a list of id-url pairs from here [0]. It's CSV formatted and contains 1_960_207 entries (last update being 22 feb 2019). It is 134MiB uncompressed and 35MiB compressed using xz, so definitely storable in a web extension. IDs being integers smaller than 10_000_000, they can be stored in 3 bytes and using a 64 bits hash function is enough (using this approximation [1] with k=2_000_000 and N=2^64 gives p=1,08e-7) which accounts for 22MB for 2 million entries. Stats on duplicates would be needed to know the impact of bundling identical hashes together. Definitely doable! Keeping up-to-date would be harder, having a server querying the API to collect and distribute the day-by-day data to every extension-user is probably the best option. [0]: https://console.cloud.google.com/marketplace/product/y-combinator/hack… [1]: https://preshing.com/20110504/hash-collision-probabilities/

There must be some kind of communication error going on. I don't know much about RCU, so I just pulled up this webpage: https://www.kernel.org/doc/Documentation/RCU/whatisRCU.txt In it is: > The rcu_read_lock() and rcu_read_unlock() primitive read-acquire and release a global reader-writer lock. Seems like RCU-operations in the Linux kernel are defined in acquire-barrier and release-barrier terms. I heard a while ago that RCU could be discussed in terms of release-consume semantics (which are slightly faster but harder to understand...) but very few people understand release-consume. As such, release-acquire is probably the memory model of the future. I'm not really aware of anything aside from: Fully Relaxed (unordered), the obscure release-consume, release-acquire, and finally sequentially consistent (too slow for modern systems) --------- Are you perhaps confusing "acquire-release" semantics (which is a memory-barrier / cache coherence principle) with spinlocks perchance? Acquire-release seems to be the "Fastest-practical" memory consistency model. (Since Relaxed doesn't work, and release-consume is too confusing) For more info on acquire-release, Preshing's blogposts are great: https://preshing.com/20130922/acquire-and-release-fences/

You might be being really subtle about python there ! I realize that one can create lines of code like :_ = ( 255, lambda V ,B,c :c and Y(VV+B,B, c -1)if(abs(V)<6)else ( 2+c-4abs(V)-0.4)/i ) ;v, x=1500,1000;C=range(vx );import struct;P=struct.pack;M,\ j ='<QIIHHHH',open('M.bmp','wb').write for X in j('BM'+P(M,vx3+26,26,12,v,x,1,24))or C: i ,Y=_;j(P('BBB',(lambda T:(T80+T9 i-950T 99,T70-880T18+701 T 9 ,Ti(1-T452)))(sum( [ Y(0,(A%3/3.+X%v+(X/v+ A/3/3.-x/2)/1j)2.5 /x -2.7,i)*2 for \ A in C [:9]]) /9) ) ) (stolen from : http://preshing.com/20110926/high-resolution-mandelbrot-in-obfuscated-… which could be seen as retrograde vs assembler (but not really for the very funny and brilliant code above - you have to see in formatted nicely and run it to realize that there are some great people out on the web!) perhaps in fact I would agree with this dig - some people do write horrid bits…

Here are some software engineering blogs from my RSS reader in addition to the ones listed in TFA. I honestly don't know how over or underrated they are, but I guess that varies depending on which communities you're closer to. Mikola Lysenko https://0fps.net Graphics programming, geometry and algorithms. --- Steven Wittens http://acko.net Not underrated in any sense of the word, but I have to include it in case there are people who haven't seen it yet. It's about math and graphics, but also about art and design. Amazingly educational. http://acko.net/blog/how-to-fold-a-julia-fractal/ is a good starting point for new readers. --- Mark Nottingham https://www.mnot.net/blog/ Networking and HTTP. --- Jeff Preshing http://preshing.com Efficient concurrent data structures. --- Bruce Dawson https://randomascii.wordpress.com Software efficiency, performance optimization, fractals and more. --- Yann Collet http://fastcompression.blogspot.com Modern compression algorithms. --- Gil Tene http://stuff-gil-says.blogspot.com Also definitely not underrated, at least not in the Java world. But any engineer who cares about latency and honest benchmarks will benefit. Low latency techniques, GC performance, lock-free programming. --- Tony Finch http://fanf.livejournal.com C and UNIX programming, and more. --- Andy Wingo https://wingolog.org Scheme, Compilers, JavaScript, and much more. --- Aleksey Shipilёv http://shipilev.net JDK internals, low-level optimization.

> It reinforces the idea of Memory Ordering as voodoo I thought there were some very solid bits in there, like the reminder that memory barriers always come in pairs. > it conflates compiler optimizations with memory reordering Well, the code as written is subject to reordering both by the compiler and the CPU. The author is just being honest that this isn't a perfectly crafted example. You could make a version not subject to reordering by the compiler, either by marking variables as volatile or by inserting "compiler barriers" into the code (e.g., empty volatile asm statements). I don't think this is conflation. > Worse, it's just not true --- while this particular example might not cause problems, I think what happened is that you interpreted the author's statement as one about memory reordering in general on the x86 architecture, but I think the author is referring to this specific application, which makes the statement true. (The sentence you quoted is bracketed by two sentences which are more explicitly about this demo application, but it's not explicit in the sentence you quoted.) > memory reordering is still an issue that needs to be dealt with on x86. The article links to another post by the same author with a similar experiment demonstrating reordering on x86. The example is a lot harder to follow, though, since x86 only allows reordering of stores and loads relative to each other. http://preshing.com/20120515/memory-reordering-caught-in-the-act >…

As I understand it, it's the other way around: for x86 (which is TSO) the fence doesn't really matter because the "normal" load is already slowed down, but for ARM it does matter quite a bit, see e.g. [0] (admittedly old blog post) slightly lower on the page. But perhaps we did get to the point where even ARM CPUs and surrounding memory is already performant enough so that spurious fences aren't as noticeable. [0] https://preshing.com/20140709/the-purpose-of-memory_order_consume-in-c…

or a port of the lock-free hash map from https://preshing.com/20130605/the-worlds-simplest-lock-free-hash-table… But really, the premise of using a shared map with concurrent readers and writers seems like a good generator of hard-to-reproduce bugs. IMHO shared-nothing (when feasible) is much easier to reason about, and possibly do periodic merging of thread-local updates, but I would avoid concurrent updates entirely (in particular if 2 threads race to update the same key... that goes to deeper design issues in the application).

> The article explicitly mentions problems with interleaving of instructions between two processes (aka threads) which is the fundamental problem of concurrency. Again, with weak memory models, writes from another thread can be observed out of order due to hardware reordering, even without compiler optimizations. With weak memory models, there can be behaviors that correspond to no valid interleaving of two threads. I am explicitly pointing out that what you are assuming, along with the author, what is often called sequential consistency (https://en.wikipedia.org/wiki/Sequential_consistency) hasn't been true of hardware for decades. It's a common misconception that most systems obey sequential consistency. Your comment just seems to repeat that again, and it isn't true. See here (https://preshing.com/20120930/weak-vs-strong-memory-models/) for a deeper explanation.

For the memory order not covered in this talk, see Fedor Pikus: https://www.youtube.com/watch?v=ZQFzMfHIxng And this whole blog series: https://preshing.com/20120612/an-introduction-to-lock-free-programming…

Number two can be addressed with something like this: https://preshing.com/20121224/how-to-generate-a-sequence-of-unique-ran… You can shrink the range by using a smaller prime number and throwing on out-of-range input. And then you can offset the output to make sure the base-N encoded value is always D digits long.

As someone who has done a lot of atomics programming: stick with sequential consistency unless you are using the variables in a very well known pattern (i.e. the double locking pattern, ring buffers, etc). Additionally, atomic doesn't mean lock free. But if they are lock free [0] you can use atomics in shared memory across a process barrier which is a lot of fun. Another good resource on this is: https://preshing.com/20120612/an-introduction-to-lock-free-programming… [0] https://en.cppreference.com/w/cpp/atomic/atomic/is_lock_free

memory_order_consume has been around since C++11, but discouraged. If you want a good overview of what it is supposed to do, https://preshing.com/20140709/the-purpose-of-memory_order_consume-in-c… goes into more detail with lots of examples and motivation.

Memory Reordering Caught in the Act covers this topic and is very well written: https://preshing.com/20120515/memory-reordering-caught-in-the-act/

Why would error-handling code impact performance? Besides, exceptions in C++ are known to have negative impacts on overall performance even if you don't use them. (see: https://preshing.com/20110807/the-cost-of-enabling-exception-handling/…