Cyrille Rossant

This is good info. I've been wary of hdf5 for some time. Nothing concrete (until this bug) but from my research it just consistently smelled fishy. The main turnoff for me was the possibility of data corruption bricking the entire dataset. Pity, as it has on paper a lot of great concepts and features. Maybe it'll be mature enough someday, though my money is on something better from the ground up coming along. Honestly, most of the portability advantage is moot nowadays. Chunk s3-like storage, smb, and ability to copy files from ext to ntfs (at least on nix) means that sharing your data across platforms isn't the struggle it used to be. Windows is rapidly becoming/already is a second class citizen in science-data heavy workflows. I ended up going with a NAS and just file system primitives for my computer vision image workflow, works great. https://stackoverflow.com/questions/35837243/hdf5-possible-data-corrup… https://cyrille.rossant.net/moving-away-hdf5/

HDF5 is a clunky file format and dependency. There's a whole host of usual complaints, many of which have already been mentioned: http://cyrille.rossant.net/moving-away-hdf5/ My biggest personal annoyance is that HDF5 isn't thread safe^, so it only supports parallel reading and writing via multiple processes. This makes parallel computing a pain. This is especially annoying when using HDF5's built-in compression, which hogs a lot of CPU. Inter-process communication is slower than reading from SSDs, so that isn't a great alternative: http://matthewrocklin.com/blog/work/2015/12/29/data-bandwidth There's a lot to be said for file formats that you can simply memory map, and that's exactly what Feather/Arrow are. Building out-of-core workflows on top of should be a joy. Wes -- does the Python library for Feather already release the GIL? ^ you can use and/or compile HDF5 with a global lock, but the underlying library still isn't thread safe.

Congrats for a serious engineering achievement! I had experimented with a similar idea in an infinitely more primitive way more than ten years ago (https://cyrille.rossant.net/numpy-browser-llvm/). I'm glad to see so much progress since then.

Interesting article, thanks. This is about the legal aspects of Shaken Baby Syndrome, and how Robert Roberson's was sentenced to death in Texas on the basis of now-obsolete science. It's almost impossible for the medical profession to admit that mistakes were made, especially when the mistake takes on a life of its own. I guess the judicial system has a similar problem. I visited /u/rossant's profile, and found a blog post about his book on this subject: https://cyrille.rossant.net/sbsbook/

The one potential requirement I'll caution you on is resilience against write failures, in case you're collecting time-series data and can't afford to lose a "session" or spend time messing with recovery options. HDF5 is not made for that. Binary and SQLite are better in that respect. SQLite wins on usability against binary and HDF5. https://cyrille.rossant.net/moving-away-hdf5/

Before moving to HDF5: https://cyrille.rossant.net/moving-away-hdf5/

Another way to do this would be to use numba package to generate LLVM IR from Python bytecode (it supports a rich set of instructions including numpy stuff, has type inference, etc), and then run it in the browser via emscripten, with some adjustments to the IR if required. Quick googling yields a related blog post: http://cyrille.rossant.net/numpy-browser-llvm/