HTTrack Blog

Also useful: https://blog.httrack.com/blog/2013/11/15/everything-you-always-wanted-…

Because that is not portable and/or has performance problems. I'm not kidding you. I know the conventional wisdom is that a rename is enough. It is not. Depending on the filesystem, you may have to fsync() the directory entry too, or fsync the parent directory entry too. Or on other filesystems, don't do these because doing these may cause problems. And do all this in the right order, because if you get it slightly wrong then it doesn't do what you want in the event of a crash. The problem is maddening. There is an article out there that explains all this in much more detail. I think it is this one but I don't quite remember whether it is the same as the one I read: http://blog.httrack.com/blog/2013/11/15/everything-you-always-wanted-t… The performance problem comes from the fact that you need two fscyncs.

fsync() has never been as reliable as the name implies, since flushing the disk cache kills performance system-wide. Fuzziness on what fsync does has... side effects: https://danluu.com/file-consistency/ http://blog.httrack.com/blog/2013/11/15/everything-you-always-wanted-t… Filesystems optimized for flash, and for battery-backed systems in general, (laptops, phones) have some history: https://en.wikipedia.org/wiki/Flash_file_system

Unwinding under Android, especially from JNI is also a mess. http://blog.httrack.com/blog/2013/08/23/catching-posix-signals-on-andr…

For large allocations, realloc can remap virtual memory instead of doing a naive copy: http://blog.httrack.com/blog/2014/04/05/a-story-of-realloc-and-lazines…

Yup, I reported the same issue a year ago (http://blog.httrack.com/blog/2014/08/31/i-found-a-bug-in-strncat/)

This argument in this article is mainly based on a naive realloc() that does allocate-copy-free, but on a system with virtual memory the memory allocator should be able to reallocate by modifying page table entries, which is far faster than copying the data around; in the (highly improbable) unfortunate case that there are no contiguous VAs on every reallocation, this method still has to move a quadratic number of PTEs, but for the given example of growing to 1M, assuming 4K pages that's only 256 PTEs in total - and 1 + 2 + 3 + ... 256 is 32896. Assuming each PTE is 4 bytes, that gives 131584 total bytes moved, in the pathologically worse case. And if you’re lucky the OS’s virtual memory system will do some magic with page tables to make the copying cheap. But still, it’s a lot of churn. There is no actual copying of data, and as the numbers show, a little over 128K for zero-copy reallocations via VM is still 16x less than the 2M of a doubling, copying buffer. Thus, a better strategy could be allocate-copy-free with doubling size for small buffers, and resizing in page-sized increments allowing realloc() to do zero-copy via VM for large buffers. Related links: http://stackoverflow.com/questions/16765389/is-it-true-that-modern-os-… http://blog.httrack.com/blog/2014/04/05/a-story-of-realloc-and-lazines… (discussed previously at https://news.ycombinator.com/item?id=7541004 )