Iron.io Blog

> So making your code 100x faster saves you more than 100 servers. The first thing that came in my mind is the following article from back in 2013: https://blog.iron.io/how-we-went-from-30-servers-to-2-go/

I came across http://blog.iron.io/2013/01/queue-webhook-events-with-ironmq.html This is pretty sweet, but it seems like it might be more useful for the baked-in reliability to be on the remote end. That is, the side sending the POSTs should notice that it didn't get a 200 reply and retry some number of times with backoff. Also, the POSTs need to be idempotent (include a sequence number or nonce or something) so that if the "200 OK" gets lost and the sending side retries, you can discard the duplicates on the receiving side. That's of course a lot harder since you have to convince everybody whose service you want to use that this is worth the time/effort :)

Look into Iron.io http://blog.iron.io/2012/05/best-practices-scaling-image-processing.ht…

Iron.io | San Francisco, CA | Full Time | ONSITE | Operations Engineer at Iron.io HQ Hey all, We're looking for Operations engineers to work on large scale distributed systems at Iron.io HQ in San Francisco. We're the makers of IronMQ (our cloud message queue) and IronWorker (our asynchronous task execution service). Here are some fancy numbers to describe us: * IronWorker has scheduled and launched over 1,100,000,000 Docker containers to date since we started using Docker (see http://blog.iron.io/2014/10/docker-in-production-what-weve-learned.htm… for more)) * All of those containers run 3rd party code on behalf of our customers * We execute over 500k process hours worth of user code per month * We serve over 5 billion API requests/month * We run services in many regions across multiple cloud providers Here's our StackOverflow company page with more details about us. (check https://careers.stackoverflow.com/company/iron-io) If you're interested in the job, send email to [email protected] with: * Your Github handle * Your resume or description of your experience and projects * A short description of your favorite operations related project and why it's interesting

IMO if you build a product and it is working, there really is not much reason to redo your efforts in a higher performance framework unless your needs require you to do so. If you are at your current hardware's limits and you can't throw more hardware at it (for whatever reason), it may make sense in that case. If you want to reduce the amount of servers you currently have, it also makes sense there. http://blog.iron.io/2013/03/how-we-went-from-30-servers-to-2-go.html This is a great post talking about going from 30 servers using Ruby to 2 servers using Go. Use case example - Twitter was using Rails on the front-end for a while but it got to the point where using it made little sense. Part of Rails' performance is dependent on caching - which is really hard, if not impossible, to do well when you are dealing with live data. FYI, Gemini is our framework that we've built and maintain internally. You probably won't find reference to it (except for out blog). edit This is the link to Twitter's blog about Rails http://engineering.twitter.com/2011/04/twitter-search-is-now-3x-faster…

Funny that you should post this, and the next day on the front page of Hacker News an article talking about Ruby taking 30 servers, each of which running at 50% CPU utilization -- and still Rails would go to 100% utilization and crash the cluster when usage spiked -- while Go could run on a single server without even seemingly using CPU (5% was typical). [1] That's greater than a 40x speed improvement -- closer to 300x, going from 5% on one to 50% on 30. >We have come to understand that developer time is vastly more valuable than computing resources I get that, but still, 30 servers is pretty expensive. Having a full time engineer just to manage the cluster and write load balancing code is pretty expensive. Having down time because of a usage spike is pretty expensive. I'd rather pay for the smarter dev team myself, but I'm an engineer, so I'm biased. Though fundamentally I wasn't talking about servers above. I was talking about desktops, and mostly Windows desktops. And having to install Python because that's what one developer prefers, and .NET because of another preference, and Perl because of a third, and Ruby because of a fourth, and Java because of a fifth...where does it end? My own "scripting language" of choice is Lua. It's TINY, it's faster than all of the above (except, under some circumstances, Java -- but give Mike Pall another year or two and I bet LuaJIT will be beating Java in all the benchmarks instead of just many of them), and using it gives developers…

Docker support is a new feature to enable more complex jobs where you need to control the entire stack, for instance, maybe you need imagemagick installed on the core machine, or ghostscript. IronWorker can support that. It also allows you to test your workers locally before uploading: http://blog.iron.io/2015/03/the-new-ironworker-development-workflow.ht… That said, you don't need to think about Docker if you don't want to and an IronWorker can be a few lines of code too, for example: https://github.com/iron-io/dockerworker/blob/master/ruby/hello.rb Or node: https://github.com/iron-io/dockerworker/blob/master/node/hello.js

Has anyone used IronWorker in production? It seems to be more flexible than AWS Lambda. http://blog.iron.io/2015/01/aws-lambda-vs-ironworker.html

It's nice to see how fast Go is already. It'll probably be a couple years ago, but it should improve once they start working on the optimizing compiler. People have already demonstrated how Go can be used on the server: http://blog.iron.io/2013/03/how-we-went-from-30-servers-to-2-go.html

If a language is 30x less efficient than another language then you would likely need 30x more servers. Many folk are simply not prepared to spend 30x more than they need to on hardware. It's the difference between 20 servers and 600 servers. Case in point: "How We Went from 30 Servers to 2": http://blog.iron.io/2013/03/how-we-went-from-30-servers-to-2-go.html

You might consider looking again. Check out this post (http://blog.iron.io/2012/05/new-ironworker-command-line-interface.html… about new .worker files and ng gem (https://github.com/iron-io/iron_worker_ruby_ng). Docs are really improving rapidly too. Finally, don't hesitate to stop by the support channel (http://get.iron.io/chat) with any questions.

Here are a few updated differences. disclosure I work for Iron.io Lambda: node, java IronWorker: node, ruby, python, php, java, binaries, .NET, go (binaries), and more (specific language runtime versions available) Lambda: allows 500mb of local disk space to persist your task IronWorker: 10 gb of local disk space available Lambda: only current version of code IronWorker: versions all uploads and allows you to revert Lambda: no built in scheduler IronWorker: a flexible scheduler out of the box to run tasks Lambda: maximum execution time 60 seconds IronWorker:maximum execution time 1 hour (customizable up to 24 hours for dedicated users) Lambda: 100 maximum concurrent request. (The maximum can be higher, can someone point me to the absolute maximum if it exist?) IronWorker: concurrency maximum can go much higher as needed (250 for a production plan). You can also limit concurrency per IronWorker, aka set a worker to only run up to 50 concurrent tasks. (useful if you have bottlenecks) Lambda: 90 inactive code retention period IronWorker: no limit to inactive code retention period Lambda: available on aws IronWorker: choose your cloud provider: AWS, Rackspace, Microsoft Azure, private clouds, and more. We've also enabled users to pause the execution of their task queue incase they need to patch or revise their worker code. supporting your own docker…

Here are some examples just off the top of my head. Go: http://blog.gopheracademy.com/moving-to-go http://blog.iron.io/2013/03/how-we-went-from-30-servers-to-2-go.html http://blog.gopheracademy.com/day-03-building-a-twelve-factor-app-in-g… Clojure: http://blog.getprismatic.com/blog/2013/1/14/bringing-functional-to-the… http://www.infoq.com/presentations/Why-Prismatic-Goes-Faster-With-Cloj…

> Apart from Google (who uses it for Youtube), Go hasn't really been used in large projects. I don't know how large each of these projects is, but a growing number of companies are using it for serious projects: Iron.io: How We Went from 30 Servers to 2 with Go http://blog.iron.io/2013/03/how-we-went-from-30-servers-to-2-go.html Behind the Scenes: Big Data at Torbit http://torbit.com/blog/2013/02/19/big-data-at-torbit/ pool.ntp.org DNS server in Go http://news.ntppool.org/2012/10/new-dns-server.html Juju at Canonical http://www.reddit.com/r/programming/comments/18atce/juju_canonical_109… Go at bitly http://word.bitly.com/post/29550171827/go-go-gadget CloudFlare blows hole in laws of Web physics with Go and Railgun http://arstechnica.com/information-technology/2013/02/cloudflare-blows… dl.google.com now served by Go https://groups.google.com/forum/?fromgroups=#!topic/golang-nuts/BNUNbK… Go At Conformal https://www.cyphertite.com/blog.php?/archives/7-Go-at-Conformal..html Go at Novartis https://plus.google.com/114945221884326152379/posts/d1SVaqkRyTL Go at the BBC http://www.quora.com/Go-programming-language/Is-Google-Go-ready-for-pr… Go at SoundCloud http://backstage.soundcloud.com/2012/07/go-at-soundcloud/ Go at CloudFlare http://blog.cloudflare.com/go-at-cloudflare Go at Heroku http://blog.golang.org/2011/04/go-at-heroku.html Building StatHat with…