Bitly Engineering Blog
word.bitly.com · Company blog · on HN since 2012
The Bitly engineering team writes about distributed systems, monitoring, Hadoop, and open source tools like the NSQ message queue.
What HN saysHN commenters point to its NSQ post for the design background of the message queue, and compare it with beanstalkd and other engineering blogs.
16HN people shared it
8linked it in comments
10stories submitted
246story points
9 years agolast shared
2012people per year2026
Most-shared posts
- 1Bitly's realtime distributed message queue: processing at scale4 people · 4 points · first shared 2012 · discussion on HN
- 2Go go gadget3 people · first shared 2012
- 3Joining Bitly Engineering2 people · 2 points · first shared 2014 · discussion on HN
- 4Things We Forgot to Monitor1 people · 232 points · first shared 2014 · 61 comments on HN
- 5Lessons learned while post-processing 1.75 billion lines of Hadoop output1 people · 2 points · first shared 2014 · discussion on HN
- 6Speeding things up with RedShift1 people · 2 points · first shared 2014 · discussion on HN
- 7Building NSQ Client Libraries1 people · 1 points · first shared 2014 · discussion on HN
- 8Front End Dependency Management with Browserify1 people · 1 points · first shared 2014 · discussion on HN
- 9Introducing the Bitly Tech Podcast1 people · 1 points · first shared 2015 · discussion on HN
- 1010 things Bit.ly forgot to monitor1 people · 1 points · first shared 2014 · discussion on HN
- 11Homepage1 people · first shared 2017
- 12Clickatron1 people · first shared 2012
What HN people wrote when they linked it
JayeshSidhwani · 2017-04-21
I had asked a similar question on Quora [1] few years back. Here is the summary:
- Facebook: https://www.facebook.com/Engineering?sk=notes
- Google: http://google-engtools.blogspot.com/
- Twitter: http://engineering.twitter.com/
- Linkedin: http://engineering.linkedin.com/blog
- Dropbox: https://tech.dropbox.com/
- Instagram: http://instagram-engineering.tumblr.com/
- Netflix: http://techblog.netflix.com/
- Flickr: http://code.flickr.com
- Etsy: http://codeascraft.etsy.com/
- Yelp: http://engineeringblog.yelp.com/
- Heroku: https://blog.heroku.com/
- Airbnb: http://nerds.airbnb.com/
- Bitly: http://word.bitly.com/
- Evernote: https://blog.evernote.com/tech/
- Highscalability: http://highscalability.com/
[1] https://www.quora.com/What-are-the-companies-that-have-some-of-the-bes…
matticakes · 2012-10-09
beanstalkd is a great project and it certainly inspired some of our thinking. I think of beanstalkd as a more fully functional alternative to simplequeue (what we built/used prior to NSQ - more details in our blog post http://word.bitly.com/post/33232969144/nsq).
Some of the goals of NSQ transcended just replacing our specific daemon that buffered and delivered messages (most importantly the interactions with the lookup service). Because of that, we felt that owning that piece would make it easier to achieve those goals.
Additionally, one of the most important properties of nsqd (the queue component of NSQ) is that data is pushed to the client rather than polled (like in beanstalkd).
justinsaccount · 2017-05-31
probably better off looking at
http://word.bitly.com/post/33232969144/nsq
http://nsq.io/
haarts · 2012-10-26
I do believe marketing has something to do with it. I don't believe a random Google employee is allowed to disclose stuff about their infra permission is granted. This is not a bad thing mind you. Google is merely trying to win minds in the most friendly way possible.
Neither did I said that this was a 'major' differentiator between Go and D. Just 'a' differentiator and not a major one at that.
I have become slightly myopic with regard to the little announcements. I'm confusing it with announcements like Bitly's: http://word.bitly.com/post/29550171827/go-go-gadget or Cloudflare's: http://blog.cloudflare.com/cacheing-the-uncacheable-cloudflares-railgu… But the point still stands, I see more of these posts related to Go than posts related to D (or Clojure/Scala/Rust/...).
jehiah · 2012-10-09
Some of the background behind our decisions are on our blog post http://word.bitly.com/post/33232969144/nsq
In terms of manual de-duping, we strive internally for idempotent message processing so it's fairly irrelevant, but to handle cases where it matters, all of our messages have unique id's added to them outside of NSQ.
The actual cases where messages are handled multiple times is limited to when a client disappeared during message processing (a hard restart) or it passed the allowable time window to respond and was given to another client.
If there are any specific numbers you are curious about, please ask.
thezilch · 2013-08-27
There are plenty accounts of Go in production: (Google's dl.google.com) https://groups.google.com/d/msg/golang-nuts/BNUNbKSypE0/E4qSfpx9qI8J, (CloudFlare's Railgun) http://blog.cloudflare.com/go-at-cloudflare, (Bit.ly) http://word.bitly.com/post/29550171827/go-go-gadget, (SoundCloud) http://backstage.soundcloud.com/2012/07/go-at-soundcloud/, (Disqus's realtime-backend) http://blog.disqus.com/post/51155103801/trying-out-this-go-thing, etc
mseepgood · 2013-04-04
> Apart from Google (who uses it for Youtube), Go hasn't really been used in large projects.
I don't know how large each of these projects is, but a growing number of companies are using it for serious projects:
Iron.io: How We Went from 30 Servers to 2 with Go
http://blog.iron.io/2013/03/how-we-went-from-30-servers-to-2-go.html
Behind the Scenes: Big Data at Torbit
http://torbit.com/blog/2013/02/19/big-data-at-torbit/
pool.ntp.org DNS server in Go
http://news.ntppool.org/2012/10/new-dns-server.html
Juju at Canonical
http://www.reddit.com/r/programming/comments/18atce/juju_canonical_109…
Go at bitly
http://word.bitly.com/post/29550171827/go-go-gadget
CloudFlare blows hole in laws of Web physics with Go and Railgun
http://arstechnica.com/information-technology/2013/02/cloudflare-blows…
dl.google.com now served by Go
https://groups.google.com/forum/?fromgroups=#!topic/golang-nuts/BNUNbK…
Go At Conformal
https://www.cyphertite.com/blog.php?/archives/7-Go-at-Conformal..html
Go at Novartis
https://plus.google.com/114945221884326152379/posts/d1SVaqkRyTL
Go at the BBC
http://www.quora.com/Go-programming-language/Is-Google-Go-ready-for-pr…
Go at SoundCloud
http://backstage.soundcloud.com/2012/07/go-at-soundcloud/
Go at CloudFlare
http://blog.cloudflare.com/go-at-cloudflare
Go at Heroku
http://blog.golang.org/2011/04/go-at-heroku.html
Building StatHat with…