Backblaze Blog

I love Backblaze too. They also have a great blog and have posted about their custom hardware a while back http://blog.backblaze.com/

Supposedly Backblaze was at one point approached about their 45 drive chassis' by a federal agency. http://blog.backblaze.com/2014/03/19/backblaze-storage-pod-4/ 45 drives * 6TB drives = 270TB of storage in one chassis (more than a month's worth of recordings). How much does it cost to get a whole rack colo'd in the Bahamas? As you mentioned, if you have the horsepower, you can do speech to text right on the box. Plenty of space for ElasticSearch as well.

Ugh. Backblaze is one of those companies with an extraordinarily poor design that they flout and "open source" as if anyone would follow their lead. Take a look at the physical design of their system and combine that with the published data. Consider that to remove any harddrive from their setup requires physically removing a 4u rackmount storage pod from the rack. http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v2-0reveali… Also, no hardware raid, battery, or cap. Source: worked at Eye-Fi, built 2PB storage

Perhaps 100s of TBs but 100 of GBs is not a difficult problem to solve, perhaps once you get to more than 100 TB you'd be beyond the capabilities of a single chassis. Even a petabyte should fit in a rack or two. I really fail to understand how a business could acquire that much data and not be able to sell it. http://blog.backblaze.com/2013/02/20/180tb-of-good-vibrations-storage-… Here ya go, 180 TB for $10K in 4U, which means 10 to a rack which means 1.8 PB per rack. Who has 180 TB of database that isn't worth $10K?

That sounds like a great way to boost a struggling supply chain. I wonder if Backblaze could have done this in 2011 (http://blog.backblaze.com/2012/10/09/backblaze_drive_farming/) -- instead of buying huge numbers of drives themselves, getting customers to buy them and mail them in in exchange for cheaper storage.

Hey Brian, thank you for explaining the reasons of why the Linux version hasn't seen the light yet, I have been keeping an eye on Backblaze since this blog entry you wrote http://blog.backblaze.com/2008/12/15/10-rules-for-how-to-write-cross-p… quite a while now, and it sucks that there's still no Linux version available, so as others have mention please consider releasing something. PS: I'm already one of your costumers and I'd love to be able to use your service in my linux machines.

A matrix chart sounds like a great idea. Anyone know if Backblaze handles this correctly? Here is a post describing their setup: http://blog.backblaze.com/2008/11/12/how-to-make-strong-encryption-eas…

The argument seems to be that average online storage needs are simply growing beyond what can be provided by a flat-fee unlimited plan. I don't know if that's true, but there's something important the post doesn't address: the potential declining costs of providing online storage. Might the two not balance each other out for the foreseeable future? I refer to this most excellent post by BackBlaze, which outlines how they do storage: http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-buil… While we might not see many additional leaps of over 90% reduction in cloud storage costs, I (a) wonder how much headroom such innovation bought Backblaze, and (b) whether their main costs, hard drive, will keep pace with user demands.

BackBlaze, an online backup company, posted a great blog article a while back about building their custom 67TB servers for just under $8,000. They even include all the how to, and a link to the custom case manufacturer if you wanted to do it yourself: http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-buil…

I meant figure as in: numbers. They (you?) switched from an expensive platform to another expensive platform. In an article about a transition I hope to find hard data about the performance- and cost-tradeoffs that were made. How many hosts, how much data, how much traffic, how many queries, how much latency, how many dollars? Instead I got a generic "AWS is awesome" boilerplate along with a generic AWS best-practice bullet-list... FWIW, here's a not-so-old chart from backblaze about relative storage cost: http://blog.backblaze.com/wp-content/uploads/2009/08/cost-of-a-petabyt… S3 ranges way up there and knowing that figure first-hand (and also their traffic rate) I sort of expected cost considerations to be at least mentioned in a document titled "XXX transition to ... storage-system".

WOW... that is impressive. 67TB in a 4U. They put a lot of thought into those servers. http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-buil… Edit: Impressive service and great price point. It looks like the only thing that I would want to have as well is geographic redundancy too.

Rack space is an ongoing cost while the hardware is a fixed cost. Presumably they've decided it's worth having a slightly higher fixed cost per GB in exchange for a reduced ongoing cost per GB. (There's probably other benefits as well, such as reduced workload and just getting the space online now.) Let's take a look... They do have an oldish post on hosting costs: http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v2-0reveali… A rack in 2011 cost them $2,100 per month (it's more now, guaranteed). Their racks look to hold about 11 machines. Each pod is then about $191/mo. Each rack thus needs 1.375 extra pods at "low density". That's $262.50 a month. Since you now need about 10% more pods, you're paying about $29 more in hosting costs per month per pod. Your savings are not $688, since you also need to construct 1/8 pod later. A low-density pod will cost about $2719, with is $340 split 8 ways. So real hardware savings is $348. Hosting costs eat that in exactly a year, and they are certainly expecting a longer life than that. (That's 2011 hosting costs, which are certainly lower.) So it's worthwhile to spend a little extra for the additional drives. (Of course, if they could get the extra five in with more modest hardware, as seems quite possible, that would be a sure win. But sometimes ease of development/assembly/spare parts wins over hardware cost.)

> I wonder if Amazon hasn't cut Dropbox a huge discount to keep them around. I can't imagine why they would. Dropbox dedupes all their data before hitting S3, but Amazon can still leverage their bulk hardware discounts that Dropbox likely isn't big enough to score. Amazon would have the upper hand in such a deal, and I don't see why it would be friendly. > http://blog.backblaze.com/2013/02/20/180tb-of-good-vibration... looks like an interesting solution to storage. AFAIK those boxes are and have usually been tuned for lukewarm or maybe even lukecold storage to cut costs. Dozens of 5400 rpm drives loaded on bare minimal support planes have a genuine rating of Shit for reliable seek times. It's great for backup, when you only expect a small fraction of users to actually make a query and be thankful enough they're pulling down a recovery. For the kind of direction Dropbox wants to take, this is absolutely out of the question.

Backblaze wrote about it here (2011): http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v2-0reveali… and here (2013): http://blog.backblaze.com/2013/02/20/180tb-of-good-vibrations-storage-… We are constantly looking at new hard drives, evaluating them for reliability and power consumption. The Hitachi 3TB drive (Hitachi Deskstar 5K3000 HDS5C3030ALA630) is our current favorite for both its low power demand and astounding reliability. The Western Digital and Seagate equivalents we tested saw much higher rates of popping out of RAID arrays and drive failure. Even the Western Digital Enterprise Hard Drives had the same high failure rates. The Hitachi drives, on the other hand, perform wonderfully. In the second article they say that WD-RED are 2nd in reliability (WD-RED did not exist 2011). I'm happy that I've got a cheap Hitachi Ultrastar. But who knows. As a personal anecdote: WD-Green failure rates are huge here. 24/7 Desktop machines, 240 drives. I've replaced in last 12 month at least 20 drives.

Check your math. You're talking about recording the keystrokes and URLs people visit on smartphones. That adds up to such a phenomenally-small fraction of the data they actually download, storage for all 141+ million users could probably be accomplished by your average geek with a checkbook. For scale, lets assume that each user sends, oh, 100 megabytes of data over a year. Roughly equivalent to 100 million characters, or (going with a 5mb ascii version of the bible I just googled) 20 bibles worth of text, or 1 million URLs at 100 characters long (that's roughly one URL every 30 seconds), so this is likely an overestimate, possibly substantially. And they have all 141 million users for the full year. Punch in the numbers, and you get 141 terabytes of data. Without compression. That's microscopic. I can buy hard drives for that off the shelf for not too much money. Here, Backblaze sells that much for $7.4k in a single box, which is absolute pennies to a company with 141 million customers: http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v2-0reveali… Meanwhile, Amazon has pricing tiers going into the petabytes for S3, and very likely receives far more than our theoretical 141TB in a single day. And they're not the size of the moon.

There are pretty much three major costs to consider when it comes to video online. They are storage, delivery and encoding. Storage is pretty cheap. S3 starts at $0.14 per GB for the first TB and then drops to $0.125 for the next 50TB and so on. There might be some cheaper cloud storage options out there. If you don't need the durability that S3 provides, you can use their reduced redundancy option which is like 40% cheaper. There are other cloud storage provides out there worth considering like Rackspace's Cloud Files, although I think S3 is the cheapest. CDNs also often include their own storage so that might also be something to consider. You might be able to get a good rate if you committed to a minimum storage and bandwidth package. If you wanted, you could even go the route of building your own storage units (http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v2-0reveali… which presents it's own pros and cons. Delivery also isn't that expensive. CloudFront, for instance, charges $0.12/GB for the first 10TB and $0.08/Gb for the next 40TB. If you're streaming over 10TB per month, you can start to get discounts by committing to monthly minimums. Again, there are other providers out there, such as limelight and akamai, but as far as I can tell, they don't publish their prices. That usually means you probably need upfront commitments with them. Because of that, there is probably a lot of room to negotiate the price of bandwidth. Delivery will most…

This isn't my area of expertise, but when you factor in redundancy, you're looking at a petabyte of data just for what you have now. You'll need it to failover plus have at least one full backup at a separate location. Commodity hardware almost seems prohibitively expensive. You'll have the hardware, 600+ 2tb drives just to start, space, electricity, cooling, maintenance (with that many drives, you'll probably be replacing them often). If you went with the solution that BackBlaze rolled for themselves, you'd be looking at 15 of their servers (for redundancy), without even adding any more data in the future. Using the solution they came up with, that would run almost $120,000. I have no idea what the monthly maintenance expenses would be. http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-buil… You are undertaking a nearly herculean feat, here. I don't think any of the professional data storage solutions out there will be any cheaper, either. In fact, a petabyte of storage (especially if you need doubles of all hardware to place at another location) would probably run nearer a million bucks or more. It might even be worth, as someone else linked, considering BackBlaze themselves. At least find out what they could do for you. They offer a business plan at something like $50/computer for unlimited storage. Obviously they're not going to store 0.5-1.0 petabytes of data for that price, but on the scale that you need, you might be able to…

You're right, my math was flawed, though possibly not in the way you were implying, because: I added a T1 ($250/month) to a building rental ($2000/month), and then said that they'd have $10,000/month left over. Why would I do such a thing? Why, so that: They could upgrade to redundant DS3s (or better!) They could buy extra drives! They could pay someone to live within a few miles and carry a cell phone that accepts text messages! ...etc. Doubly funny, to me, is that "disk failure" is being mentioned in the comments on an article about how all of their disks failed all at once. So, no, I certainly did not forget about disk failure. Indeed it's one of the reasons that I'd rather not host certain things as EC2 instances. As far as scalability: Well, how far would 67TB carry you? Because you could deploy 67TB every single month and still have money left over. [1] To get to the meat of my argument: I think that what's both great and horrible about EC2-like services is that they allow programmers to deploy Really Big Things without having to employ or deal with system administrators. As a result, it's a solution that's being misapplied in some cases. [1]: http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-buil… -- it even has pretty pictures

There's some info on the hardware on Netflix Open Connect site [1]. They mention it's heavily influenced by Backblazes Storage Pod[2]. [1]: https://www.netflix.com/openconnect/hardware [2]: http://blog.backblaze.com/2014/03/19/backblaze-storage-pod-4

This post is also on BackBlaze's blog - http://blog.backblaze.com/2008/12/15/10-rules-for-how-to-write-cross-p…

How to save 95%: don't use AWS long-term, buy dedicated hardware. http://blog.backblaze.com/2013/02/20/180tb-of-good-vibrations-storage-…

A more accurate comparison would be a BackBlaze Storage Pod [0] [0] http://blog.backblaze.com/2013/02/20/180tb-of-good-vibrations-storage-…

Interestingly, we think they do as well if not better (http://blog.backblaze.com/2013/12/04/enterprise-drive-reliability/). We think it's because external drives are meant to be jostled around by Joe Schmo, so they do well in our storage pods, whereas internal drives and especially enterprise drives are meant to live in a perfectly Utopian environment.

> and in some cases...externals that were made internal! :) I remember reading about this a while back[1] (fun read!). Have you guys noticed a big difference in life expectancy of the repurposed external drives or do they generally match up with their internal equivalents? [1]: http://blog.backblaze.com/2012/10/09/backblaze_drive_farming/

That's definitely true, that you wouldn't buy all your capacity upfront, but you would need to have a minimum amount of capacity on hand upfront, and to do that in multiple datacenters and build the software to manage it would cost money. That explains why they started on S3 but it doesn't explain why they are still there now. looking at the economics of it, I wonder if Amazon hasn't cut Dropbox a huge discount to keep them around. But even still, with $250 million in the bank, I bet the discussion to move in house is happening right now. After a certain point (X users, Y revenue, Z $ VC money), I would look at hosting in-house. http://blog.backblaze.com/2013/02/20/180tb-of-good-vibrations-storage-… looks like an interesting solution to storage.

They've published some stuff on disks: https://static.googleusercontent.com/external_content/untrusted_dlcp/r… The backblaze guys also have a big data set: http://blog.backblaze.com/2013/11/12/how-long-do-disk-drives-last/

The original "Drive Farming" link is at least as interesting as the new article: http://blog.backblaze.com/2012/10/09/backblaze_drive_farming/

In the parent article they said they switched because of cost: "When Hitachi drives got too expensive we switched to Seagate and Western Digital. When internal drives got too expensive we turned to external drives. When hard drive availability became an issue we turned to Drive Farming." From their article on v2 they said: "The Western Digital and Seagate equivalents we tested saw much higher rates of popping out of RAID arrays and drive failure. Even the Western Digital Enterprise Hard Drives had the same high failure rates. The Hitachi drives, on the other hand, perform wonderfully." http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v2-0reveali… There are other articles where they mention that they do testing and some brands perform better than others.

The extrapolations in the article were pretty optimistic to begin with. I mean, when you look at the dotted trendline in this graph: http://blog.backblaze.com/wp-content/uploads/2013/11/blog-cost-trend-1… you'd think by 2016 drives would cost $0 per gigabyte and by 2020, drive companies would be paying you to take drives off them.

I guess that comment was cynical, but just to be sure: http://blog.backblaze.com/2013/11/12/how-long-do-disk-drives-last/. Especially in the beginning there is a higher failure rate for disk drives. So it is actually the opposite: backups should be one of the first things to arrange for new machines.