StorageMojo

Excellent. I generate lots and lots of communications every day and find that being misread is a way to score my effectiveness. Paul Graham points out that his essays can't really have pages of footnotes to cover all the possible interpretations of the writing and he is correct. We strive to make it understandable to the widest possible audience and go from there. I suggest you really closely examine this thought : "The reality is they are working for a company that desires to increase their bottom line just like any other business-- as they should. My point is because of this, the ecosystem is click driven. Which means a much greater frequency of startups will be covered. Which means even if a startup is doing nothing new and the technology is rather unimpressive, the brand attached to it means clicks ($$) so it gets coverage." There is a whole lot of assumption going on in there, and while you do have to assume some things [1] its not true that you can assume all things. To check yourself you can try to reason your way from first principles to the conclusion that you expressed above. You can stipulate that businesses want to be profitable. Some will argue with that saying some businesses are non-profits but the point is sufficiently accepted that you can get away with just stipulating it. Then you claim based on that one stipulation that the desire for profit makes the business "click driven." That bit of reasoning is weak, on what basis can you claim that the only…

Do we know now how Glacier actually works? Tape robots, spun-down disks, racks of optical media? Best source I could find was: https://storagemojo.com/2014/04/25/amazons-glacier-secret-bdxl/

There are plenty of startups in storage space, it's a very active area for obvious reasons. Look at the Storage Mojo blog for coverage of them: http://storagemojo.com/ E.g. the very first item as I write this is on industry giant NetApp buying a Canadian company that has to be a startup (the company's CTO and blogger's hair is otherwise too long :-). Virtualization is also hot and there are startups in this space, including the above company. Xen and VMWare have only recently exited "startup" status. Security that gets pretty deep into systems also has a lot of startups.

It all depends on what your definition of "high-volume" is, and just how "archival" your access patterns really are. Amazon Glacier runs on BDXL disc libraries (like a tape library). There's nothing truly expensive about producing BDXL media, there just isn't enough volume in the consumer market to make it worthwhile. If you contract directly with suppliers for a few million discs at a time, that's not an issue (you did say high-volume, right?). https://storagemojo.com/2014/04/25/amazons-glacier-secret-bdxl/ For medium-scale users, tape libraries are still the way to go. You can have petabytes of near-line storage in a rack. Storage conditions are not really a concern in a datacenter, which is where they should live. (CERN has about 200 petabytes of tapes for their long-term storage.) https://home.cern/about/updates/2017/07/cern-data-centre-passes-200-pe… If you mean "high-volume for a small business", probably also tapes, or BD discs with 20% parity encoding to guard against bitrot. Small users should also consider dumping it in Glacier as a fallback - make it Amazon's problem. If you have a significant stream of data it'll get expensive over time, but if it's business-critical data then you don't really have a choice, do you?

Blueray disks are thought to be the key: https://storagemojo.com/2014/04/25/amazons-glacier-secret-bdxl/comment… Some people disagree though. It’s still an unknown.

Some have theorized that this is what's behind Amazon Glacier, and that they want to keep it to themselves: https://storagemojo.com/2014/04/25/amazons-glacier-secret-bdxl/ I also crave for such cheap storage, perfect for archival/backups, especially when combined with modern snapshotting. HVDs never really materialized, despite being promising, and actually on the market at some point, IIRC: https://en.wikipedia.org/wiki/Holographic_Versatile_Disc A lot of other medium were developped, never commercialized: https://en.wikipedia.org/wiki/3D_optical_data_storage A new optical storage medium is long overdue. This time, it will probably be without support from the media industry.

Rumor has it Blu-Ray (specifically BDXL): https://storagemojo.com/2014/04/25/amazons-glacier-secret-bdxl/ Though a comment on that page mentions this HN comment claiming custom low-speed HDDs: https://news.ycombinator.com/item?id=4416065

> > Retries won't help you when the data was corrupted on its way to be stored. > Why not? 1. Data is fine in memory 2. Data is fine over PCIe. 3. Data is corrupted by firmware bug in storage controller. 4. Data (now corrupted) is sent to disk, which generates checksums for each LBA, and sends the bits to the storage medium. 5. Some time later the data is requested, disk reads bits from sectors, checksums pass, corrupted data is sent back up the stack to the application. This is not a fantasy scenario: * http://www.cs.toronto.edu/~bianca/papers/fast08.pdf * https://storagemojo.com/2007/09/19/cerns-data-corruption-research/

Lots of theories but it may have been licensing, a possible NetApp patent infringement lawsuit, or maybe they thought they could do better. https://arstechnica.com/gadgets/2009/10/apple-abandons-zfs-on-mac-os-x… https://storagemojo.com/2009/08/31/why-did-apple-drop-zfs/

Assumes: - 3 year support - $1000 monthly hosting fees - admin costs for non-cloud items - 10% data accessed each month (for S3 and Glacier) It would have been much better to have made a simple calculator to adjust how much if any of these items to include. The 10% accessed for S3 and Glacier is excessive and without it I think Glacier could be very competitive with tape. And also missing from this chart is optical as Glacier is thought to be implemented. https://storagemojo.com/2014/04/25/amazons-glacier-secret-bdxl/

In my opinion, Excel is not a good platform for documenting research because it is very time consuming to audit and try to reproduce the results since there is no standard way for inputting data. There is a long list of cases where companies and governments have made huge decisions that were based on Excel spreadsheets where the models had fundamental flaws [0][1][2]. [0] https://www.forbes.com/sites/timworstall/2013/02/13/microsofts-excel-m… [1] https://storagemojo.com/2016/08/26/excel-may-be-dangerous-to-your-heal… [2] http://www.jamaicaobserver.com/business/Excel-is-dangerous-_19130803

Google did some tests/research/study a few years back and they noticed that hard drives fail around 40% of the time without showing any signs with SMART attributes. EDIT: Ok, few years = 10. http://storagemojo.com/2007/02/19/googles-disk-failure-experience/

Not likely tape. Looks like evidence points to BXDL https://storagemojo.com/2014/04/25/amazons-glacier-secret-bdxl/

I suspect that Amazon will not only be dropping prices, but will be adding new ways to store data with variable pricing (e.g. new tiers as well as reduced redundancy). Not to mention that if you have 100PB in AWS you might get preferential pricing? Google recently increased their spread and now have 4 tiers. There is also an interesting article about Glacier storage and pricing here https://storagemojo.com/2014/04/25/amazons-glacier-secret-bdxl/.

It is rumored that Amazon denied that Glacier uses tape. It's still not that clear what it does use. https://storagemojo.com/2014/04/25/amazons-glacier-secret-bdxl/ (also read the comments)

I think this is a good guess, http://storagemojo.com/2014/04/25/amazons-glacier-secret-bdxl/

RAID 6 is not an option with drives big enough because of rebuild time. https://storagemojo.com/2010/02/27/does-raid-6-stops-working-in-2019/

It's often assumed that Amazon uses tape for glacier but that's actually unfounded. Amazon's keeping the details of glacier as a trade secret[1] [1]: http://storagemojo.com/2014/04/25/amazons-glacier-secret-bdxl/

Here's one theory: https://storagemojo.com/2014/04/25/amazons-glacier-secret-bdxl/ and here's the HN thread: https://news.ycombinator.com/item?id=7647571 Facebook uses Blu-ray discs with a robotic storage system for its cold storage: http://www.datacenterdynamics.com/it-networks/robot-digs-through-blu-r…

It was never confirmed, but I recall this[1] article making a pretty persuasive case Glacier was using optical disks. [1]https://storagemojo.com/2014/04/25/amazons-glacier-secret-bdxl/

Pretty sure Amazon said publicly that they are not using tape for Glacier, I can't find the source except this second hand one: https://storagemojo.com/2014/04/25/amazons-glacier-secret-bdxl/

EMC XtremeIO is giving Pure a strong run for its money. EMC's biggest problem in storage isn't on the All-Flash side (XTremeIO is working as intended), it's that there's fewer reasons to buy the high end, high margin (VMAX) storage these days, and EMC needs to come up with a big margin patch... which it thinks it might have with DSSD [1] [2]. [1] https://storagemojo.com/2014/11/11/pure-vs-emc-whos-winning/ [2] https://storagemojo.com/2015/02/18/dssd-hiring-is-exploding/

Cold storage is probably safe. I've seen the assertion that it's tape from someone who I wouldn't have thought would get it wrong but I'd now bet against it whether or not Amazon's denial seals the deal. Robin Harris has argued in favor of BDXL but he's speculated about a number of things. http://storagemojo.com/2014/04/25/amazons-glacier-secret-bdxl/ My money would be on the simplest: cold disk. Some support here: http://mjtsai.com/blog/2014/04/27/how-is-amazon-glacier-implemented/ but it's all speculative and anecdotal. I don't have a problem with the thought that Amazon is getting out ahead of the drive economics, especially given that I'm not sure how widely used Glacier is today. I do find it interesting that this has been apparently kept under wraps from anyone who would share more widely.

It sucks when hard drives fail, but it isn't that "uncommon" for drives to fail in under two years. A lot of disk manufactures only offer a 1 year warranty for a lot of their models nowadays. Google did a study on failure rates: http://storagemojo.com/2007/02/19/googles-disk-failure-experience/ The trick is to buy stuff with a credit card that offers warranty extensions (usually 1 year). Just call up the credit card company and they will tell you what you need to do.

State of autonomous cars: http://en.wikipedia.org/wiki/Autonomous_car And I might forward you to Formula 1 racing which is one of the most technologically sophisticated activities on the planet. The car companies involved e.g. Ferrari, Mercedes, McLaren are constantly dealing with big data, machine learning and real time simulations. http://storagemojo.com/2012/11/02/big-data-in-formula-1-racing/ The idea that nobody can handle IT apart from web companies is a bit silly.

http://storagemojo.com/2007/02/19/googles-disk-failure-experience/ Google did some research on this with their datacenters. There is apparently no correlation between heat & diskfailures.

Someone else reports experiences of switches rebooting after 497 days: http://storagemojo.com/2011/11/07/how-fault-tolerant-are-sans/#comment…

> 1) Macs do tend to run hotter than PCs, and this could decrease HDD lifespan. Google found that temperature didn't much matter according to their published research: http://labs.google.com/papers/disk_failures.pdf > In the lower and middle temperature ranges, higher temperatures are not associated with higher failure rates. This is a fairly surprising result, which could indicate that data center or server designers have more freedom than previously thought when setting operating temperatures for equipment that contains disk drives. Non-PDF summary/discussion: http://storagemojo.com/2007/02/19/googles-disk-failure-experience/

Related: Does RAID 6 stop working in 2019? http://storagemojo.com/2010/02/27/does-raid-6-stops-working-in-2019/

More on the ZFS situation http://storagemojo.com/2009/08/31/why-did-apple-drop-zfs/