Server Fault Blog

Stack Exchange (Stack Overflow) barely made it out. We are in the same datacenter but we just finished building out and testing a secondary datacenter in Oregon literally last weekend. We did an emergency failover last night after the datacenter went to generators. Read more at http://blog.serverfault.com

Not at all. The Windows OS now has command line accessible management points that are similar to Linux. There is still a great deal of difference in their management models (I blogged about this a while back - http://blog.serverfault.com/2013/06/03/cross-platform-configuration-ma… That I was hired as a Windows specialist was so that we can go deeper on the OS side and the PowerShell side, just as we have a Linux expert to go deeper on the Linux side. Our sysadmin team was just more tilted with experience on the Linux side (though you wouldn't know it - as almost everyone I work with would qualify as a senior admin in any Windows shop in the world).

> Read more at http://blog.serverfault.com Thank you for this discussion!

You only pay for licenses over 4 OS and 2 SQL Server (or outright new ones) after graduating BizSpark. 2 SQL Servers (OS licenses are chump change by comparison) go a loooong way* too, as you're presumably scaling up rather than out. http://www.microsoft.com/bizspark/about/Graduation.aspx Through the grapevine I hear the "review" process (for more free licenses at graduation) is extraordinarily accommodating, though I'd guess if you're spinning up a box per customer or something insane you'd get denied. Lots of people get this wrong, I'm surprised Microsoft hasn't carved this correction into a mountain somewhere... *Stack Exchange is running 2 + failover ( http://blog.serverfault.com/2011/09/30/the-stack-exchange-architecture… ), and was running on 1 not too terribly long ago.

For those who aren't reading it the ServerFault blog http://blog.serverfault.com/ is a treasure trove of information. The work they are doing and blogging about on the systems side is something a lot of people take for granted.

We use a LOT of open source stuff too. We believe in the best of both worlds -- http://blog.serverfault.com/post/stack-exchanges-architecture-in-bulle…

Although we use MS for our core we do use Linux and open source projects as part of our stack. The routers/firewalls, redis for caching, and haproxy for load balancing are all run on Linux. In addition we use Linux for a lot of the management such as Bacula for backups and Nagios for monitoring. We try to be open about our infrastructure and both George and I blog about the changes we make at http://blog.serverfault.com

First of all, massive kudos for the Stack Exchange-like technical transparency. Definitely consider a massive upgrade album like http://blog.serverfault.com/2015/03/05/how-we-upgrade-a-live-data-cent… and http://imgur.com/a/X1HoY! GitLab is awesome. I'm really sad that in the past two or three months I've only found one GitLab link on HN to click. There really needs to be more. (I'm not sure if this is because I'm browsing in AEDT or if GitLab isn't used a lot on here.) I wondered about how you guys might do advertising to get more mindshare, and then I realized one possible explanation about why you're doing this: getting technical advice from the community means everyone's had a part to play, and they're likely to remember that. Good move ;) --- In my case, I have little (okay, 0) practical experience; a lot of the following is mentioned experimentally, to see how these ideas would handle the described environment. It's pretty much all stuff I've read online. I welcome replies that shoot down any of these ideas. > Disk Disks can be slow so we looked at improving latency. Higher RPM hard drives typically come in GB instead of TB sizes. Going all SSD is too expensive. To improve latency we plan to fit every server with an SSD card. On the fileservers this will be used as a cache. We're thinking about using Bcache for this. There's already been another brief comment (https://news.ycombinator.com/item?id=13153317) about ZFS. So, I'll ask. Why not ZFS? You don't have…

When you look at something like how Stack Exchange moved physical servers to a new datacenter, you can see where AWS benefits you. Not all startups have the server and networking know-how to pull that kind of stuff off, or even set it up in the first place. https://blog.serverfault.com/2015/03/05/how-we-upgrade-a-live-data-cen…

Relevant: https://blog.serverfault.com/2017/01/09/surviving-the-next-dns-attack/

From my feedly: http://techblog.netflix.com/ http://githubengineering.com/ http://blog.serverfault.com/ https://engineering.quora.com/ http://corner.squareup.com/

Not as detailled on the tooling, but it sows how much hardware they use, Stackoverflow did a blog post on their datacentres move: http://blog.serverfault.com/2015/03/

Those drives still cost about 20k or so to purchase, for only about a 25% increase for reads and 10% for writes (http://blog.serverfault.com/2011/02/09/our-storage-decision/) why would we pay that extra for so little gain? Also, with FusionIO you are putting all of your eggs in one basket - drive wise. If that card dies you are done. In short we don't _need_ the FusionIO so why would we put out that kind of money for it? Also we went to 96GB about a year or so ago when we setup those boxes it was a reasonable amount of memory to put in now - we are maxing the thing out at 288GB now.

It turns out that I haven't been the only person surprised by the TCP incast pattern, including folks who don't use Erlang at all (e.g. Hadoop users). I was happy to write about it, to help spread the word. Even if I'd stumbled upon Kyle Brandt's posting http://blog.serverfault.com/2011/06/27/per-second-measurements-dont-cu… ("Per Second Measurements Don’t Cut It"), I don't think I'd've realized the full consequences of packets dropped by microbursts of traffic when there's "obviously plenty of bandwidth available".

Wonder if anyone noticed how crazy their engineers are: http://blog.serverfault.com/post/performance-tuning-intel-nics/

Someone asked the question in the comments: http://blog.serverfault.com/post/stack-exchanges-architecture-in-bulle…

Well, this answers my question: http://blog.serverfault.com/post/stack-exchanges-architecture-in-bulle… About 1/10th Reddit's traffic, but also, I would think, a much lighter usage pattern (i.e. a lot more writes on Reddit, more quickly changing pages, etc.)

Scaling has been cheap, and fun. Read about our newest data center: http://blog.serverfault.com/post/1432571770/stack-overflows-new-york-d… Read about our SQL Server hardware: http://blog.stackoverflow.com/2010/10/database-upgrade/