Mechanical Sympathy

Check out Peter Lawrey's blog [1] has lots of excellent content coming from high performance trading background. Also some older (probably dated) content on Martin Thompson's Mechanical Sympathy [2] blog. [1] http://blog.vanillajava.blog/ [2] https://mechanical-sympathy.blogspot.com/

Reminds me of one of the best blogs i ever subscribed to, now defunct unfortunately https://mechanical-sympathy.blogspot.com/

Aside: this is a great blog https://mechanical-sympathy.blogspot.com/

Matt Warren's blog is great to learn about the internal mechanism of .Net. I wonder if there is any similar blog for JVM world, so far I have only know Mechanical Sympathy[1] but it doesn't seem as comprehensive. [1] https://mechanical-sympathy.blogspot.com/

I had the pleasure of briefly working with Martin at LMAX. He definitely knows how to get performance out of a system, and although it's not been updated recently I'd recommend his old blog for those that haven't seen it yet and are interested in where the nanoseconds go : http://mechanical-sympathy.blogspot.co.uk/

You're probably already aware of it, but Martin Thompson of LMAX has done a lot of great work on HFT systems and his blog "Mechanical Sympathy" is excellent: http://mechanical-sympathy.blogspot.com/ Although a lot of people think HFT is just a race to the bottom (and they are probably right) it's still great fun to try and wring every last nanosecond of performance out of your code.

1) http://www.altdevblogaday.com/ 2) http://mechanical-sympathy.blogspot.com/ 3) (Not really a blog, but so great) http://lwn.net/ 4) (For entertainment) http://thedailywtf.com/

I don't know why the author called this "Multi-process architectures suck :(" when he really meant "I suck at multi-process architectures :(". Look at what he lists: - "Context Switching" You can find 4 cores in a trivial computer these days (8 cores with hyperthreading). This means you can have 8 processes without context switching at all, and also suggests that if you don't use multiple processes, you can only reach about 10-20% capacity on a multi-core machine. - "Per-process overhead" It's true it has overhead, but you don't have to create thousands of processes, we have light concurrency patterns to use within the actual thread/process. But even then, you don't even have to run more than one primary process per core. You have multiple cores. They can't together run one process. - "We built for a single-core world" and "We lack the tools" Those are outright PEBKAC errors, and not architectural problems. We're way past the stage when multi-process architectures were a hardly understood, confusing problem. We have the right tools, for those who look for them. Cache misses and overly eager context switching are bad, and you can read a lot about this by Martin Thompson on his Mechanical Sympathy blog: http://mechanical-sympathy.blogspot.com/ There are ways to design multi-process systems to take best advantage of all cores, and your machine caches. But even Thompson's Disruptor architecture is based around message passing and multiple processes. Because, again, in a…

This is from the author of https://mechanical-sympathy.blogspot.com/

In general, our industry lacks a sense of "mechanical sympathy"[0]. Whether it's understanding what knobs are tunable in an operating system or using streams for reading data to keep a consistent memory profile, we often don't dig deep. Ditto for folks understanding database systems. I think in very big systems, you see folks dig deep, but it's not encouraged with smaller systems. I think there's a lot of inefficiently run services - not from a price perspective but from a "what could I really get out of 3 compute nodes and a well designed database schema." [0]: https://mechanical-sympathy.blogspot.com/

Here's a nice (old) blog of someone in the industry: https://mechanical-sympathy.blogspot.com/

It's not either/or. Everything needs to be tuned to achieve the final latency requirements. This includes tuning java code to eliminate creating "garbage", immutable string objects mostly, unnecessary object creation, creating short lived objects compared to long lived etc. The other part right sizing jvm generational partitions, using right gc algo etc. Next part would be to find out the code path with lowest latency requirement and making it hot/jitting. Even next part would be to tune OS for things like thread pinning etc etc. You can go further and tune the network layer. Every little thing counts. You can go even further and employ what's called mechanical sympathy (https://mechanical-sympathy.blogspot.com/) and write your code in a way that makes it easier for the processor to process faster. I've never worked with application that use off-heap allocations or no-gc requirements.

you guys need to read this blog https://mechanical-sympathy.blogspot.com/

Mechanical Sympathy by Martin Thompson https://mechanical-sympathy.blogspot.com/?m=1

Java 7 and above provide XX:MaxGCPauseMillis flag for GC conf From Mechanical Sympathy Blog [1] " G1 is target driven on latency –XX:MaxGCPauseMillis=<n>, default value = 200ms. The target will influence the amount of work done on each cycle on a best-efforts only basis. Setting targets in tens of milliseconds is mostly futile, and as of this writing targeting tens of milliseconds has not been a focus of G1. " So in Java world we are talking 100s of milli seconds of worst case which is 3 order of magnitude higher than Go. 1. http://mechanical-sympathy.blogspot.com/2013/07/java-garbage-collectio…

The author Martin Thompson is one of those rare breed of developers who understands the sheer inner workings and the effects of his programs from the source to processor. http://mechanical-sympathy.blogspot.com/

And I think this is insane. We've gone through a series of "obvious, pragmatic" next steps to arrive at a model that seems sane and normal on the surface, but which results in...wat!? http://mechanical-sympathy.blogspot.com/2011/07/false-sharing.html Writing highly concurrent and parallel stuff in Erlang to run on top of multi core systems seems sane to me. In most other languages? Maybe half-sane at best.

Although I guess that is telling that one of LMAX authors (Martin Thompson) has moved back to support C++ alongside Java on his newer product. http://mechanical-sympathy.blogspot.de/ I guess C++ progress in the last years has helped this.

A blog I've really enjoyed on that subject is Mechanical Sympathy: http://mechanical-sympathy.blogspot.com/

[Edit: The LMAX guys showed how much more performance is possible with aligning code with CPU/hardware (in this case for Java) http://mechanical-sympathy.blogspot.de/ ]

If this resonated with you, then check out Mechanical Sympathy [1], a mailing list all about writing code that is aware of how computers work and working with them. Martin Thompson's blog by the same name also has a lot of good information [2]. To the people who say "Who is going to pay for this?" look at all of the places that can measure the cost of slow or inefficient computing e.g. Wall St, Google, Amazon, e.t.c. They all invest in writing low level efficient code where it makes sense. [1]: https://groups.google.com/forum/#!forum/mechanical-sympathy [2]: http://mechanical-sympathy.blogspot.co.nz

Martin Thompson has a blog and a series of talks on "Mechanical Sympathy" -- being in tune with the machine to extract performance, like a race car driver. "How to get 100K TPS with 1ms latency": http://www.infoq.com/presentations/LMAX Blog: http://mechanical-sympathy.blogspot.in

And, of course, the Mechanical Sympathy blog itself: http://mechanical-sympathy.blogspot.com

Very much unfashionable for the HN crowd. Most of the new research in concurrency is happening in Java and funded by the high performance trading industry - an industry which is very far from the HN crowd. The new Java8 stampedlock is a good example. It's possible to implement it in C++ as well, but because of the guarantees required by the lock it is a very difficult lock to integrate into C++ code. On the other hand, the JRE guarantees the correct constraints for Java code making a stampedlock very easy to use [1]. The performance of a stampedlock also seems to be the best case for any multi-reader environment. [2] [1] http://concurrencyfreaks.blogspot.com/2013/11/stampedlocktryoptimistic… [2] http://mechanical-sympathy.blogspot.ca/2013/08/lock-based-vs-lock-free…

2 good JVM performance blogs: http://vanillajava.blogspot.com/ http://mechanical-sympathy.blogspot.com/

Unknown is relative, but here are some of the rss feeds I'm subscribed too: http://nosql.mypopescu.com/ http://lambda-the-ultimate.org/ http://eli.thegreenplace.net/ http://www.stefankrause.net/wp/ http://mechanical-sympathy.blogspot.com/ http://www.cowtowncoder.com/blog/blog.html http://highlyscalable.wordpress.com/ http://hunch.net/ http://perspectives.mvdirona.com/ http://www.scottaaronson.com/blog/ http://www.softwarebyrob.com/ http://blog.mikemccandless.com/

Another good read about increasing disk IO and reducing latency can be found at the Mechanical Symphony blog about the single writer principle: http://mechanical-sympathy.blogspot.com/2011/09/single-writer-principl… We ran into issues where lots of threads attempting a disk write were causing latency problems. We were able to get around this by having a single dedicated thread to disk IO.

I didn't feel bruised, I felt pity ;) People write hard constrained trading software in Java e.g. http://mechanical-sympathy.blogspot.se/ Googling for java avonics has some hits too e.g. http://www.slideshare.net/keugbang/t-50-avionics-embedded-software-dev… I was pretty familiar with the intracities of real-time called iirc "guaranteed service" back when I worked at Symbian. That was a kernel written in dialect of C++. And I've written camera and video capture software for, among others, Android phones. So fairly general experience, I'd say. Still so much more to learn. You, on the other hand Chris, keep calling out Rob, me and anyone else who thinks that PDP-11 is not a necessary requirement for the next decade's "system language" ;) What systems projects have you worked on then?

not dead, this blog is a great example of low level and high level living in harmony: http://mechanical-sympathy.blogspot.com/2011/07/memory-barriersfences.…

Here are my guesses (in addition to your points): - Until late in it's lifecycle, .net runtime performance was worse than JVM. - Also, the Windows TCP stack, re-architected with Vista, had severe latency issues, which probably led to wholesale abandonment of that platform for low-latency applications. Some of these problems were fixed in Windows 8, but it was probably too late by then. (These two points are based on my own experience with Windows/CLR/JVM in the mid/late 2000s and some anecdotes I heard second-hand. So I can't really cite any public sources for them) I'm sure the fact that you can customize Linux to the extreme and the lack of .Net on Linux in those days probably had a lot to do with the choice of Java, in addition to the above. However, based on my reading of blogs like Mechanical Sympathy [1] a while ago (~8-10 years), there does seem to have been some demand for .Net-based solutions in the HFT space. They had code for both Java and C# in their public code. [1]https://mechanical-sympathy.blogspot.com/