Delphix Blog

FULLTIME or INTERN. Offices in San Francisco, Menlo Park, and Boston Delphix is a data virtualization company that is doing for databases what VMware did for servers - this is a massive market, and we are on track for similar success. We're looking for software generalists to help build our full stack, from the operating system and filesystem, to the Java management stack, and the GUI. The engineering team at Delphix is composed of the inventors and architects of the VMware platform, ZFS filesystem, DTrace, Oracle RAC, DataGuard, and Flashback. We've built an engineering culture where anyone with a good idea can have a voice and drive unique projects. Whether it's developing new abstractions in the filesystem, designing an architecture to interoperate with a novel database, or developing a new cloud paradigm for virtualizing data, there is no lack of hard problems and opportunities at Delphix. The Delphix platform has already established itself as the premier platform for structured data management in the enterprise world. In our first year of selling, we added 50 large corporate customers, including many from the fortune 500 (Proctor & Gamble, RBS, Fidelity, Deutsche Bank, Comcast, Staples, Qualcomm, among others). We have offices in Menlo Park, San Francisco, and Cambridge, MA. Check out our technical blogs at http://blog.delphix.com. News coverage: http://www.prnewswire.com/news-releases/delphix-raises-25-million-in-s…

Ah, yes -- now we're talking about a meaningful way in which ZFS can be improved! (But then, you know that.) Metaslab fragmentation is a very real issue -- and (as you point out) when pools are driven up in terms of utilization, that fragmentation can become acute. (Uday Vallamsetty at Delphix has an excellent blog entry that explores this visually and quantitatively.[1]) In terms of fixing it: ZFS co-inventor Matt Ahrens did extensive prototyping work on block pointer rewrite, but the results were mixed[2] -- and it was a casualty of the Oracle acquisition regardless. I don't know if the answer is rewriting blocks or behaving better when the system has become fragmented (or both), but this is surely a domain in which ZFS can be improved. I would encourage anyone interested in taking a swing at this problem to engage with Uday and others in the community -- and to attend the next ZFS Day[3]. [1] http://blog.delphix.com/uday/2013/02/19/78/ [2] http://permalink.gmane.org/gmane.os.illumos.devel/5203 [3] http://zfsday.com/zfsday/

Maybe off topic, but Alex Reece from Delphix had a blog post about a year ago about the ability to remove a disk from a pool. http://blog.delphix.com/alex/2015/01/15/openzfs-device-removal/ This would imply some kind of rebalancing, if I understand correctly. Which maybe could be considered a form of indirect defragmentation.

OpenZFS has resumable send and receive -- so even if they don't have it today, they could have it in the future. http://blog.delphix.com/matt/2015/03/25/resumable-zfs-sendreceive/ https://www.illumos.org/issues/2605

Actually, you can shrink pools now. http://blog.delphix.com/alex/2015/01/15/openzfs-device-removal/

ZFS has supported autodetecting 4k drives since day one, with the caveat that certain drives that were programmed to try to fool OSes that only supported 512 byte sectors would occasionally fool ZFS as well [1]. Additionally, ZFS can be set as case-insensitive, to match HFS+ behavior. Finally, I'd argue that ZFS is useful even to the one-drive user. The snapshot feature would make Time Machine backups even more transparent, compression would make things a bit smaller, and filesystem sending/receiving would make backing up to other devices a lot simpler. Modern filesystems just make sense, even for the regular user. [1] http://blog.delphix.com/gwilson/2012/11/15/4k-sectors-and-zfs/

Everything that matters about vdev size and configuration is explained here, complete with charts that indicate the actual overhead of different arrangements: http://blog.delphix.com/matt/2014/06/06/zfs-stripe-width/

vdev removal is in the canonical source of illumos at this point: http://blog.delphix.com/alex/2015/01/15/openzfs-device-removal/

Things are looking a little brighter on the ZFS pool shrinking front thanks to Delphix working on device removal: http://blog.delphix.com/alex/2015/01/15/openzfs-device-removal/

So you add a large amount of data, you snapshot, you delete that data and at that point you send the snapshot but don't want to send the deleted data? Why not just re-snapshot and send the new snapshot instead of the old one? > And I'm also unsure whether it's even possible interrupt a transfer and resume it later, if the underlying data has changed There is no "underlying data has changed" in a snapshot, it's a revision of the filesystem. As for interrupting and resuming later, Delphix apparently has contributed a resumable zfs send/receive to OpenZFS, though I don't know if it's available in yours or rsync.net's distribution: http://blog.delphix.com/matt/2015/03/25/resumable-zfs-sendreceive/ As for the concerns of sending small files first, I don't think ZFS could even use that: I don't believe it's content-addressed, so if you send a set of blocks from snapshot 1, interrupt and send snapshot 2 instead the blocks common to both will still be re-sent, ZFS only knows the block commonality from snapshots and since snapshot 1 hasn't been pushed its contents can't be used for snapshot 2. Unless the resumable replication support uses content addressing but that doesn't seem to be the case.