Richard Jones

I did a bunch of these benchmarks a while back and the most important factor in a performance test like this is the VM/Language environment. Simply calling into a VM on every IO request will rapidly destroy performance no matter what your IO setup looks like. Erlang's IO stack if just fantastic for this and is generally very well optimized. If you want to go even faster you can code the fast path in C/C++ and call out to a higher level language as needed. A little dated but helpful: http://www.metabrew.com/article/a-million-user-comet-application-with-… I think that Webbit is also pretty good, this benchmark is little contrived.

It's been done before. Here is an article using Erlang and Linux. http://www.metabrew.com/article/a-million-user-comet-application-with-… Part 3 is my favorite. http://www.metabrew.com/article/a-million-user-comet-application-with-…

My favourite prompt-hack is to colourise the $ when the previous command exits non-zero, here's how I have my bash PS1 set up: http://www.metabrew.com/article/updated-bash-ps1 If you exit `top` by pressing q, it exits with non-zero - who knew? :)

You might want to check out the following, which was its own HN entry just a short while ago. This describes testing a million connections using raw Erlang as well as using a libevent/C front-end for the Erlang testing setup to handle connections. Of course, this person was doing C1024K, but you could probably dial down his presets for your purposes :) http://www.metabrew.com/article/a-million-user-comet-application-with-…

Here are my top ten, mostly web-related in some way because of my interests. These are the "loud alpha geeks" - the ones who broadcast their interests via regularly-updated blogs or curated links. (There are plenty of "quiet alpha geeks" who are just as "alpha" but not as easy to follow online.) Brad Fitzpatrick (http://brad.livejournal.com/) - creator of LiveJournal and Danga, now at Google working on PubSubHubbub, the Social Graph API, etc. John Resig (http://ejohn.org/, http://news.ycombinator.com/user?id=jeresig) - jQuery, Mozilla, lots of other JavaScript and browser stuff. Paul Bucheit (http://news.ycombinator.com/user?id=paul) - creator of GMail and FriendFeed, now at Facebook. Ryan Tomayko (http://tomayko.com) - Ruby, Sinatra, now working at GitHub, has an excellent linkblog. Simon Willison (http://news.ycombinator.com/user?id=simonw) - Django, Python, also has an excellent linkblog. Leslie Orchard (http://decafbad.com/) - Previously at Delicious/Yahoo, wrote books on JavaScript, Dojo, RSS, now at Mozilla. Follow his bookmarks at: http://delicious.com/deusx Mark Pilgrim (http://diveintomark.org/) - Atom, HTML5, Linux, Python, Greasemonkey, formerly at IBM, now at Google. Bookmarks at: http://delicious.com/wearehugh John Gruber (http://daringfireball.net/) - Apple news and punditry, linkblog. Sam Ruby (http://intertwingly.net/blog/) - Atom, Python, Ruby, Rails, XML, HTML5, SVG. Read what he reads at: http://planet.intertwingly.net/ Kevin Kelly…

Interesting stuff. I read a few more articles here and walked away with more questions than answers. CassFS definitely looks fun learning project. What I'd really love to see, is an article that explains the goals/niche/maturity/strengths/weaknesses of various distributed storage solutions. Can anyone here offer a recommendation? (For example, here's something similar: http://www.metabrew.com/article/anti-rdbms-a-list-of-distributed-key-v… ) I've been casually researching this topic for months and I still feel so overwhelmed! Not many people are giving a clear picture of what they're trying to accomplish, when/how their system should be used, and how they compare to others. I'm mostly asking because I recently ended up writing my own. It's just a Python script that sends a UDP message to all our servers asking who has the file. First person to respond wins and the file is downloaded via nginx. Replication is accomplished with durable message queues which are consumed during off-peak hours. It took days to code and suits our needs perfectly. I learned from this project that reliable, scalable, distributed data storage is very simple, if you focus on solving a very specific use-case scenario.

https://www.metabrew.com/article/libketama-consistent-hashing-algo-mem… Ketama implementation of consistent hashing algorithm is really intuitive and battle tested.

The Linux tuning part seems to have been inspired by these blog posts from 14 years ago: https://www.metabrew.com/article/a-million-user-comet-application-with… It's almost a little disappointing that beefy modern servers only manage a x5 scale improvement, though that could be due to the differences in runtime behaviour between Erlang and the JVM.

It's called a C1M problem. From 2013: https://www.metabrew.com/article/a-million-user-comet-application-with… It's pretty easy. Been working on a product doing this for 500k connections per machine (MQTT) back in 2014. EC2, m3.xlarge instances back then.

Erlang is a terrific platform, but it isn't very accessible to the wider population. Timely, here's antirez on this topic when Redis was announced on HN ten years ago [0]: > There are a lot of projects similar to Redis in Erlang actually: http://www.metabrew.com/article/anti-rdbms-a-list-of-distributed-key-v… > Apart from speed what I like of C for this kind of projects is the self-contain-ness and the fact that most developers that may help in the development are probably able to read/write C but not Erlang. [0] https://news.ycombinator.com/item?id=494903

This has been posted twice before[1][2]. It should be noted that this benchmark was done over localhost, i.e. packets did not actually leave the computer, so who knows what kind of optimizations were done by the kernel / networking stack. When looking at examples where packets actually leave the computer, at least one person achieved 1M concurrent connections with Erlang in 2008, and WhatsApp were hitting 2.8M concurrent connections with Erlang in a production system in 2012[4]. Of course memory usage was a lot higher in these examples, but at least they were exercising the networking stack. [1]: https://news.ycombinator.com/item?id=5127251 [2]: https://news.ycombinator.com/item?id=5474331 [3]: http://www.metabrew.com/article/a-million-user-comet-application-with-… [4]: http://www.erlang-factory.com/upload/presentations/558/efsf2012-whatsa…

See http://www.metabrew.com/article/a-million-user-comet-application-with-…

WhatsApp had over 2 million users connected to their (Erlang) server last year. http://blog.whatsapp.com/index.php/2012/01/1-million-is-so-2011/ Here is the same in Erlang for reference (from a few years ago, I would be interested to see if there is a more efficient way now): http://www.metabrew.com/article/a-million-user-comet-application-with-…

Similar article here, going to 11: http://www.metabrew.com/article/a-million-user-comet-application-with-…

Interesting example from 2008: http://www.metabrew.com/article/a-million-user-comet-application-with-… I'd also add that this shouldn't be taken as something to say that Erlang is totally superior to node or that Erlang makes scaling to 1M concurrent connections a piece of cake. If you're working at that level, there's no magical out of the box solution.

If you're looking for a less functional Hubot that's been around for years, why not use Last.fm's irccat? http://www.metabrew.com/article/how-we-use-irc-at-lastfm It already does the whole "kick of a script with these params from irc" thing and can easily be piped to from any langage that can open a port.

Are you referring to http://www.metabrew.com/article/a-million-user-comet-application-with-… http://www.metabrew.com/article/a-million-user-comet-application-with-… http://www.metabrew.com/article/a-million-user-comet-application-with-… By Richard Jones of the Last.fm? Thank you for your replying. The original C10k from Dan Kegel was a guideline for me when I worked on project during 2003 - 2006. I am wondering what is the latest progress in this area. I am glad this problem is solved to you. I will check out niver tech in future when the problem comes up.

Nice. Last.fm has a nice post at http://www.metabrew.com/article/how-we-use-irc-at-lastfm. We also have separate channels for each message type. IRC is pub-sub, yay! Plus you can direct-message a bot and it will DM you back, to not pollute the channel you're currently in. Do you have any bots that take commands? I think that's where it gets really useful.

Keep in mind that select basically sucks for handling more than a few sockets -- it's slow and limited to max. 1024 concurrent sockets (unless you change FD_SETSIZE and recompile everything, which may cause unexpected bugs). An almost always better choice is using epoll -- it's usually faster and can handle any number of concurrent sockets (up to a million and more). * http://docs.python.org/library/select.html#edge-and-level-trigger-poll… * http://scotdoyle.com/python-epoll-howto.html * http://blog.urbanairship.com/blog/2010/08/24/c500k-in-action-at-urban-… * http://www.metabrew.com/article/a-million-user-comet-application-with-…

I'm curious what kind of OS tuning was required for 500k concurrent connections. It's obvious that some ulimit tuning is necessary, but I'm wondering if any other hard limits beyond that were hit? Also, in case the original author is reading, you might find this interesting as well. It's an article on trying to reach 1MM concurrent connections with Erlang. http://www.metabrew.com/article/a-million-user-comet-application-with-…

You can see this in the 1M connection test done here: http://www.metabrew.com/article/a-million-user-comet-application-with-… Look at the "Turning it up to 1 Million" section where he details the need to use 17 IPs for the client side.

Last.fm has a pretty nifty notification system that uses netcat and an IRC bot. http://www.metabrew.com/article/how-we-use-irc-at-lastfm/

http://www.metabrew.com/article/anti-rdbms-a-list-of-distributed-key-v… http://themindstorms.blogspot.com/2009/05/quick-reference-to-alternati… The comparisons in those posts are probably outdated soon with so many new nosql data stores.

Thanks for the quick reply. How many machines are running MySQL for you? I was reading this website - http://www.metabrew.com/article/anti-rdbms-a-list-of-distributed-key-v… I have not tried HBase and HyperTable myself yet, but the blog post says that they still have latency issues. What are your views?

Here's a good comparison of key/value database alternatives: http://www.metabrew.com/article/anti-rdbms-a-list-of-distributed-key-v…

There are a lot of projects similar to Redis in Erlang actually: http://www.metabrew.com/article/anti-rdbms-a-list-of-distributed-key-v… Apart from speed what I like of C for this kind of projects is the self-contain-ness and the fact that most developers that may help in the development are probably able to read/write C but not Erlang.

If you really want to play... you could use erlang Nitrogen - http://github.com/rklophaus/nitrogen/wikis <- quite new Erlang web - http://www.erlang-web.org/ <- serious enterprisey guys. Erlyweb - http://erlyweb.org/ <- ROR for Erlang And one more for the scalability fetish - http://www.metabrew.com/article/a-million-user-comet-application-with-…

We use Freenode, #grooveshark is an open channel for random techie users who want to talk directly to developers (or each other), and people working on 3rd party extensions/add ons to Grooveshark. We have other internal invite-only channels for coordinating with external developers on internal projects. We also have github spamming a channel now whenever changes to a repo that a lot of people are coordinating on get pushed. The #redis channel saved our asses at least once when Pieter Noordhuis wrote a script to help us with a particular issue we were having. :) And the #facebook channel has been a useful resource as well. I was wondering if anyone here uses IRCcat: http://www.metabrew.com/article/how-we-use-irc-at-lastfm I know flickr also uses it and it seems like it would be insanely useful to have notifications about all kinds of things going into various channels. I know flickr also uses it: http://velocityconference.blip.tv/file/2284377/ Haven't spent much time working on it but so far the bot gets kicked for flooding immediately. May only be a problem because we're using freenode for testing. Long term want to run our own IRC server, but for testing it was easier to just use freenode...

These "NoSQL" comparisons are a tricky thing, inasmuch as this is a very volatile space with a lot of really interesting development happening. Furthermore, benchmarking a handful of storage options that are really only alike in that they share the same thing which they are not (relational, SQL) is a lot like running road tests on a handful of cars that are not Honda Civics. I made the foolish decision a while back to choose a data storage option based on benchmarks and hype, and I'm currently paying for it by having achieved spectacular launch failure and now completely rearchitecting my backend. NoSQL emerged only after a decade or so of webscale challenges helped us finally realize that everything only looked like relational nails because the only tool we had was SQL hammer, and that tackling every data warehousing problem with SQL was coming at the cost of money, engineering hours, and headaches. Now that viable, production-ready alternatives have become available it's important that design decisions to go beyond seeing problems as not-nails and seeing them instead for what they are. Are you looking for reliable, versioned master-master replication for embedded devices? Might want to consider Couch. An obscenely fast caching tier that need not persist to disk? Sure, check out Scalaris. A fully distributed and pluggable map-reduce architecture capable of storing a metric fucktonne of data? Hbase might be for you. And, it should go without saying, there are still and…