Jim Salter

Agreed, mirror vdevs are the way to go. Here's a post that explains the tradeoffs between RAIDZ and mirror vdevs in great detail: http://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/

ZFS detects corruption. A very long ago someone named cyberjock was a prolific and opinionated proponent of ZFS, who wrote many things about ZFS during a time when the hobbyist community was tiny and not very familiar with how to use it and how it worked. Unfortunately, some of their most misguided and/or outdated thoughts still haunt modern consciousness like an egregore. What you are probably thinking of is the proposed doomsday scenario where bad ram could theoretically kill a ZFS pool during a scrub. This article does a good job of explaining how that might happen, and why being concerned about it is tilting at windmills: https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/ I have never once heard of this happening in real life. Hell, I’ve never even had bad ram. I have had bad sata/sas cables, and a bad disk though. ZFS faithfully informed me there was a problem, which no other file system would have done. I’ve seen other people that start getting corruption when sata/sas controllers go bad or overheat, which again is detected by ZFS. What actually destroys pools is user error, followed very distantly by plain old fashioned ZFS bugs that someone with an unlucky edge case ran into.

> It's possible to create the same amount of redundant storage space with only 6-8 hard drives with RAIDZ2 (RAID 6) redundancy. I've given up on striped RAID. Residential use requires easy expandability to keep costs down. Expanding an existing parity stripe RAID setup involves failing every drive and slowly replacing them one by one with bigger capacity drives while the whole array is in a degraded state and incurring heavy I/O load. It's easier and safer to build a new one and move the data over. So you pretty much need to buy the entire thing up front which is expensive. Btrfs has a flexible allocator which makes expansion easier but btrfs just isn't trustworthy. I spent years waiting for RAID-Z expansion only for it to end up being a suboptimal solution that leaves the array in some kind of split parity state, old data in one format and new data in another format. It's just so tiresome. Just give up on the "storage efficiency" nonsense. Make a pool of double or triple mirrors instead and call it a day. It's simpler to set up, easier to understand, more performant, allows heterogeneous pools of drives which lowers risk of systemic failure due to bad batches, gradual expansion is not only possible but actually easy and doesn't take literal weeks to do, avoids loading the entire pool during resilvering in case of failures, and it offers so much redundancy the only way you'll lose data is if your house literally burns…

A NAS built from generic x86 hardware and some disks. Use ZFS, it's a bit of a rabbit hole but an excellent choice for both reliability/redundancy and as a tool to backup. I'm not gonna explain how this works, just describe what you can do, it's an option. ZFS: - ensures you don't get bit rot - manages both disks (raid/mirrors &c) and the filesystem, it's an all-in-one solution - supports block level replication to local and remote systems, after the first backup it's fast - can create dynamic partitions to group files together and build replication strategies around Choose either RAID or mirrored drives (https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz… I've gone mirrored but more for flexibility and performance. Use a calculator to see what options of disks you have https://jro.io/capacity/ (and google 'ZFS calculator' for others) For backup get a second machine somewhere else in your house with a smaller setup and use ZFS replication to keep it up to date with everything on the main box you need backed up. Currently I use a raspberry pi with a USB disk but this is perhaps cutting it fine. You wanna keep this online so ZFS can periodically check the health of the data on the disks. Fully offline backups can be a risk. Finally for a 3rd backup use some of those external drives, format to ZFS and use replication. Plug them in on a schedule and take a backup. If you want to backup to remote systems (cloud/a box in your parents house) it also…

> If you run a scrub/whatever to check validity of old data and your system has a new memory issue, it could destroy everything very quickly even without new file writes. No, it won't. https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/ > Let’s assume that we have RAM that not only isn’t working 100% properly, but is actively goddamn evil and trying its naive but enthusiastic best to specifically kill your data during a scrub. First, you read a block. This block is good. It is perfectly good data written to a perfectly good disk with a perfectly matching checksum. But that block is read into evil RAM, and the evil RAM flips some bits. Perhaps those bits are in the data itself, or perhaps those bits are in the checksum. Either way, your perfectly good block now does not appear to match its checksum, and since we’re scrubbing, ZFS will attempt to actually repair the “bad” block on disk. Uh-oh! What now? > Next, you read [a copy of the same block from another disk]. Now, if your evil RAM leaves this block alone, ZFS will see that the second copy matches its checksum, and so it will overwrite the first block with the same data it had originally – no data was lost here, just a few wasted disk cycles. OK. But what if your evil RAM flips a bit in the second copy? Since it doesn’t match the checksum either, ZFS doesn’t overwrite anything. It logs an unrecoverable data error for that block, and leaves both copies untouched on disk.

Looks like a decent basic rundown of what was another nice jump forward (2020 fwiw, 2.0 wasn't just introduced it launched about 1 year ago, current version is 2.1.1). Worth noting a number of these have significant implications for tuning performance in common use cases, so some resources from even a few years ago are now a bit out of date. I know a lot of people for example have followed JRS' "ZFS tuning cheat sheet" [0], and a lot of that still applies. But given the increasing usage of enormously faster but pricier solid state drives, commonly a relative surfeit of processing power, and the performance of ZStandard, I think most people should now consider it over lz4 by default when using solid state drives unless using a very weak system or every last cycle is needed for other tasks (VMs perhaps). A bunch of detailed performance graphs comparing different compression algorithms were posted in the ZSTD feature introduction [1] which can help give a better idea of the right balance. HN had a long discussion on the kind of improvement it can make at scale just a few days ago [2]. Persistent L2ARC has also potentially changed the math there in some cases, though that one in turn may well be obsoleted entirely for SSD pools. And not new here I think but somewhat more recent, for those who do still have things that make use of atime ZFS now includes the Linux style 'relatime' support which does a nice job of splitting the difference between noatime and full atime. Also,…

> To add to the downsides, you can't expand RAIDZ vdevs. > If you start with a 6 disk RAIDZ2 and want to add a couple more drives, you can't. The only way to add capacity to the pool is to add an entire new vdev. This is mostly true, and in many situations for all practical purposes is true. Not fully understanding this recently cost me a few hundred dollars (2x 8TB external drives) when I needed to expand my 4 disk raidz2 to a 6 disk raidz2. However, what _is_ possible is expanding a pool by incrementally replacing drives with larger capacity equivalents. This wouldn't work in my situation as I already was using the largest consumer drives available - but the next time I need to expand in a few years it may be a possibility. > ... expandable file system that handles parity/erasure coding as well as bitrot that can be used right now Unfortunately I believe this to be the case. However in practice, so long as you know up front your data store is not expandable, it's fairly easy to work around without it being too much of a hassle. If having two zpools when you need to expand is a burden for your use case because the data necessarily will be partitioned across pools (ie: in multiple directories), there are tools that will present multiple filesystems as a single mount point to linux. Alternatively, mirrored vdevs [0] as opposed to a classic raidz2 configuration are much easier to expand, as you only have to replace 2 drives to expand your total storage space (a single…

The important key point is that you will not lose anything by using ZFS without ECC compared to using any other file system without ECC. The myth is that you specifically need ECC if you’ll be using ZFS, not that ECC is a really good idea for any file system. https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/

> ZFS is notorious for corrupting itself when bit flips That is a notorious myth. https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/

Data that's about to be written to disk often resides in ram for some period of time - bit flips in non-ECC ram can silently corrupt the data before writing it out. ZFS doesn't prevent this though it might detect it with checksumming. https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/

AIUI the slog is only for synchronous writes; most people using zfs at home don’t do any of those (unless you set sync=always which is not the default). https://jrs-s.net/2019/05/02/zfs-sync-async-zil-slog/

That's not the case it mirrored vdevs. There is no degredatuon of the array with a failed drive in a mirrored vdev, it continues humming along perfectly fine. Also resilvers are not as intensive when rebuilding a mirror as you are just copying from one disk in the vdev to the other, not all X other drives and recalculating parity at the same time. This means less reads across the entire array and much much quicker resilver times, thus less window for drive failure. But don't just take my word for it. This is a blog post that go much into much more detail https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz…

> ZFS is nearly perfect but when it comes to expanding capacity it's just as bad as RAID. if you don't mind the overhead of a "pool of mirrors" approach [1], then it is easy to expand storage by adding pairs of disks! This is how my home NAS is configured. [1] https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz…

I believe this is the article I read when I started: https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz…

> An important note is that you CANNOT add extra disks to an existing VDEV. Not true for mirror vdev, but you don’t get the added capacity. You may add vdevs later, however. Also some good arguments are made for using mirror vdevs over RAID-Z [1]. The extra space afforded by a 1:3 RAIDZ compared to 1:2 mirror isn’t much. Given the performance gain over a mirror resilvering and reduced recovery time, I prefer mirroring. [1] https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz…

On ZFS, I don't think there's any reason why it needs ECC more than any other filesystem [0]. [0] - https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/

Please always use ECC because your Filesystem-cache (for example xfs) is the same as your ARC (on ZFS)...it's really the same. But you can activate Arc check-summing at least. That's a better article: https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/

ZFS doesn't need ECC but it does benefit from it. If your pool was becoming corrupt every six months, look to a shitty drive controller or cabling first. I've run ZFS for quite a few years now on laptops and lower-end machines that can't use ECC memory and I've never had corruption, unrecoverable files/pools, etc. https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/

ZFS doesn't need ECC ram to work correctly. This is an incorrect but commonly repeated myth. See: https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/

While looking up for ZFS ncopies option to see what it actually does I stumbled onto this blog post: https://jrs-s.net/2016/05/02/zfs-copies-equals-n/ "zfs: copies=n is not a substitute for device redundancy!"

There are concerns with ZFS of the "scrub of death" on a system lacking ECC ram: https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/ There is some debate on this question: https://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=26… I'm curious if the situation is improved for BtrFS.

I fell for the ECC myth as well when using ZFS on XigmaNAS, but got schooled here on HN a while back: https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/

You might be interested in this article which does mention valid points for choosing mirrored data instead of any RAIDZ levels in ZFS: https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz… It certainly made me reconsider my ZFS practices.

I'm so sick of this argument every time a ZFS article comes up. This is by the inventor, Matthew Ahrens: “There’s nothing special about ZFS that requires/encourages the use of ECC RAM more so than any other filesystem.” https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/ Now give me my karma back.

Were you using a WiFi adapter? Intermittent drops often come from undervoltage. You may need a 3.5v power adapter. Also, slow ssh can come from dns troubles: http://jrs-s.net/2017/07/01/slow-ssh-logins/

ZFS has no special reliance on ECC RAM: http://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/ TL;DR is that a few stuck bits is won't corrupt your pool like the forum post claims. Also, in the highly unlikely event that bad RAM would make ZFS corrupt data, how would btrfs perform better on the same hardware?

ECC is nice to have but absolutely not a show stopper if you don't have it. Plenty of articles and discussions online, example: http://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/ Claim that you can only use ZFS with ECC memory to feel safe is false.

There is nothing special about ZFS that makes it require ECC. It's a myth. See: http://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/

Thanks for catching that - my statement was definitely too strong in implying data invulnerability. I was hinting that you still get a bit of extra protection with ZFS over other filesystems, both on Non-ECC RAM. There is a chance with a stuck/corrupt bit that checksum will fail and you will get a read error. I interpreted OP as saying ECC is a must for ZFS, but I don't think you are more prone to corruption that any other FS? I'm basing my understanding on this: http://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/