Yann Collet

zstd has improved a lot since it was brought in under the facebook roof, but those are incremental changes, it already existed before then. Here's the good old blog about that http://fastcompression.blogspot.com/

This didn't originate from within Facebook. It is an evolution of the work done in late 2013/early 2014 around FSE[1]. Facebook later hired Yann to support his work around this (pretty amazing!) tool. The ability to define your own dictionary is useful beyond simply large indexable datasets. Anything application-specific can usually benefit from user-defined dictionaries. [1] http://fastcompression.blogspot.com/2013/12/finite-state-entropy-new-b…

I'm guessing you are referring to: http://fastcompression.blogspot.com

The author has actually written a lot of great material on his blog: https://fastcompression.blogspot.ca/p/lz4.html?m=1 Not sure what you'd consider suitably documented beyond what is already out there. Not being patented or published in a journal look like positives to me.

There is mention in the blog of the creator about window size. It can be configured down to 4KB to it can operate in low memory environments: http://fastcompression.blogspot.com/ see under "Custom window sizes" section

I found his (Yann's) blog posts the most helpful for getting a basic idea of what was going on with FSE: https://fastcompression.blogspot.com/2013/12/finite-state-entropy-new-… and the followup https://fastcompression.blogspot.fr/2014/01/fse-decoding-how-it-works.…

It hasn't been long that I fetched zstd from https://github.com/Cyan4973/zstd Upon looking further, it turns out that the author Yann Collet [0] works with Facebook now; the repo would have been thus transferred to Facebook. Author's blog [1]. [0] https://twitter.com/Cyan4973 [1] http://fastcompression.blogspot.in/

I think you should look at the author's blog [0]. There is lots of in depth explanation of not just what he is doing but how he got there, the failed steps, the intermediate steps. [0] http://fastcompression.blogspot.in/

"LZFSE" most likely refers to a Lempel-Ziv modeling phase that converts the input bytes into a series of either literal or copy instructions, followed by a Finite-State-Entropy encoding scheme which "provides the same level of performance as [an] Arithmetic coder, but only requires additions, masks, and shifts". Here's some explanation of FSE: http://fastcompression.blogspot.com.au/2013/12/finite-state-entropy-ne…

Very nice introduction. Recently I've been following the progress of the Finite State Entropy algorithm[1]. From the readme on the github project [2]: >FSE is a new kind of Entropy encoder, based on ANS theory, from Jarek Duda. >It is designed to compete with Huffman encoder and Arithmetic ones. >While huffman is fast but can only represent probabilities in power of 2 (50%, 25%, etc.) arithmetic coding can represent probabilities with much better accuracy, but requires multiplications and divisions. > FSE solve this dilemma by providing precise probabilities, like arithmetic does, but using only additions, masks and shifts. > This makes FSE faster, on par with Huffman speed, and even suitable for low-power CPU environment. [1]: http://fastcompression.blogspot.com/2013/12/finite-state-entropy-new-b… [2]: https://github.com/Cyan4973/FiniteStateEntropy

Discussed a couple years ago here: https://news.ycombinator.com/item?id=5037089 And the followup discussed here: https://news.ycombinator.com/item?id=5075370 I repost because I'm becoming more and more frustrated with C, and wondered if others feel the same. Not because it's too low-level, rather because it seems to be trying (with limited success) to move to being a higher level language. There are times when I really do want a "thin layer over assembly", and C seems to have evolved to a place where that is no longer possible. I explain my distress more completely in the comments of Yann's article here: http://fastcompression.blogspot.com/2014/11/portability-woes-endianess…

Depending on how many times you're going to decompress the same data and the bandwidth you'll use to transmit the compressed data, different things will be "best". http://fastcompression.blogspot.com/p/compression-benchmark.html

Nice writeup! FSE, one of the parts of Zstandard, is a very elegant entropy coder. If you're following this section of the article, you might be wondering how it's encoded, since the encoding process is somewhat less obvious than for Huffman coding. What causes trouble is simply, "Once you've emitted a symbol, how do you know that the next symbol is in the range [BL, BL+2^NB)?" The answer is that the encoding is performed backwards, starting from the end of the input, and the table is constructed so that for any symbol emitted, you can find a previous symbol so that the current symbol is in the previous symbol's [BL, BL+2^NB) range. There's another writeup about FSE here, which goes into more depth about FSE (rather than talking about Zstandard in general): http://fastcompression.blogspot.com/2013/12/finite-state-entropy-new-b… My intuition tells me that there's probably a homomorphism from Huffman coding to FSE that preserves the encoded size exactly, but I haven't done the math to check.

Here are a bunch of blogs from my RSS reader that I think are worth reading. Some of them haven't updated in a while, but the archives may be worth a look anyway. * Blogs with a math focus: http://blog.sigfpe.com http://bit-player.org http://mathr.co.uk/blog/ https://www.johndcook.com/blog/ * Blogs with a programming languages focus: https://existentialtype.wordpress.com https://bluishcoder.co.nz http://blog.ezyang.com http://akkartik.name http://lambda-the-ultimate.org http://mrale.ph * Blogs with a UI design focus: http://www.brandonwalkin.com/blog/ http://ignorethecode.net/blog/ http://interuserface.net * Blogs with a graphics / 3D rendering focus: http://c0de517e.blogspot.com http://diaryofagraphicsprogrammer.blogspot.com https://directtovideo.wordpress.com http://graphicrants.blogspot.com http://www.reedbeta.com http://www.realtimerendering.com/blog/ https://blog.demofox.org http://eelpi.gotdns.org/blog.wiki.html * Other interesting / more general programming blogs (many of these have a "systems" focus): http://danluu.com http://ewontfix.com http://chneukirchen.org/blog/ http://neverworkintheory.org http://apenwarr.ca/log/ https://www.snellman.net/blog/ https://www.pvk.ca http://psy-lob-saw.blogspot.com https://randomascii.wordpress.com http://fastcompression.blogspot.com https://fgiesen.wordpress.com http://blog.plover.com http://chneukirchen.org/trivium/ http://rachelbythebay.com/w/

Here are some software engineering blogs from my RSS reader in addition to the ones listed in TFA. I honestly don't know how over or underrated they are, but I guess that varies depending on which communities you're closer to. Mikola Lysenko https://0fps.net Graphics programming, geometry and algorithms. --- Steven Wittens http://acko.net Not underrated in any sense of the word, but I have to include it in case there are people who haven't seen it yet. It's about math and graphics, but also about art and design. Amazingly educational. http://acko.net/blog/how-to-fold-a-julia-fractal/ is a good starting point for new readers. --- Mark Nottingham https://www.mnot.net/blog/ Networking and HTTP. --- Jeff Preshing http://preshing.com Efficient concurrent data structures. --- Bruce Dawson https://randomascii.wordpress.com Software efficiency, performance optimization, fractals and more. --- Yann Collet http://fastcompression.blogspot.com Modern compression algorithms. --- Gil Tene http://stuff-gil-says.blogspot.com Also definitely not underrated, at least not in the Java world. But any engineer who cares about latency and honest benchmarks will benefit. Low latency techniques, GC performance, lock-free programming. --- Tony Finch http://fanf.livejournal.com C and UNIX programming, and more. --- Andy Wingo https://wingolog.org Scheme, Compilers, JavaScript, and much more. --- Aleksey Shipilёv http://shipilev.net JDK internals, low-level optimization.

This looks very interesting. Before everybody starts making the obvious comparisons, note that this is from the same guy that made LZ4, and he is someone who clearly knows what he's doing. I've been following his work for some time. It looks like this is the evolution of Zhuff, an experimental (closed-source) compressor [1]. It is basically LZ4 followed by a fast entropy coder, specifically FSE [2], that is a flavor of arithmetic coding that is particularly suited for lookup-table based implementations. From a quick look at the source code it seems that the entropy stage uses 3 probability tables, one for literal bytes, one for match offsets, and one for match lengths. This is not dissimilar from gzip (which however uses Huffman). EDIT: from a second look it seems that the LZ77 compression stage is basically LZ4: it uses a simple hash table with no collision resolution, which offers very high compression speed but poor match search. I'm surprised he didn't implement (yet?) an HC variant as for LZ4, it could even beat gzip compression rate with no overhead in decompression speed/memory requirements. [1] http://fastcompression.blogspot.fr/p/zhuff.html [2] https://github.com/Cyan4973/FiniteStateEntropy

ZStd started out as ZHuff, which was first announced in 2008; see https://encode.su/threads/521-Zhuff-fast-compression and http://fastcompression.blogspot.com/p/zhuff.html I've been using LZ4 in commercial software since 2011, so unless I am some kind of luminary Google's compression experts must have been aware of both LZ4 and ZHuff when they started working on Brotli.

I think zstd is originally not a Facebook project. https://fastcompression.blogspot.com/2015/01/zstd-stronger-compression… I believe Yann Collet, author of lz4, eventually went on to work for Facebook and finished it there.

Explanations from Zstd author: http://fastcompression.blogspot.com/2013/12/finite-state-entropy-new-b…

I tried a fairly obscure and technical query, "what's a Finite State Entropy Encoder?" and it came up with https://news.ycombinator.com/item?id=7040951 which is about "Finite State Entropy - A new breed of entropy coder" http://fastcompression.blogspot.com/2013/12/finite-state-entropy-new-b… Neither google nor DDG found that. Pretty neat. :)

LZ4 explained for those who got interested http://fastcompression.blogspot.com/2011/05/lz4-explained.html

This is explanation from author http://fastcompression.blogspot.com/2011/05/lz4-explained.html

xxHash is not really "new", it's been available since 2014. At least, that's the oldest Python bindings available on PyPI: https://pypi.org/project/xxhash/0.0.1/ Sure, as it developed, xxHash variants appear. The latest in the family is XXH3 and it's already being used at least since 2019: http://fastcompression.blogspot.com/2019/03/presenting-xxh3.html?m=1

For the second situation, I prefer the compress/transmit/decompress metric (as a function of transmit pipe speed), as described here: http://fastcompression.blogspot.com/p/compression-benchmark.html

On a purely compression algorithm front, the recently format-stabilized [1] zstd [2] beats the pants down Brotli, in terms of compression speed, decompression speed, and compression ratio. Zstd isn't some random effort either, but a more flexible (in terms of compression time-for-ratio) effort by the creator of the popular and insanely fast lz4 compressor[3]. [1] http://fastcompression.blogspot.com/2016/06/zstandard-reaches-final-fo… [2] https://github.com/Cyan4973/zstd [3] https://github.com/Cyan4973/lz4

WWDC video talking about it: https://developer.apple.com/videos/play/wwdc2015/712/ Apple's docs: https://developer.apple.com/library/mac/documentation/Performance/Refe… Generic info on finite state entropy compression: http://fastcompression.blogspot.com/2013/12/finite-state-entropy-new-b… The last link was submitted to HN: https://news.ycombinator.com/item?id=7040951 The repo's README seems pretty clear. Google provides the rest. Not really sure how much more there is to say about a reference compression algorithm.

And (perhaps more interestingly) Finite State Entropy coding: http://fastcompression.blogspot.com/2013/12/finite-state-entropy-new-b…

More about xxhash here, it's by Yann Collet, the author of LZ4: http://fastcompression.blogspot.fr/2012/04/selecting-checksum-algorith…

I though lzfse is deflate with finite state entropy instead of huffman: http://fastcompression.blogspot.de/2013/12/finite-state-entropy-new-br…

How does it compare with Zstandard? http://fastcompression.blogspot.com.es/2015/01/zstd-stronger-compressi…