Charles Bloom

For anyone who already has at least a surface level understanding of compression and wants to take a deeper dive, check out Charles Bloom's blog: http://cbloomrants.blogspot.com Unfortunately it has been dormant for some time but there are years worth of useful information there and he is an uncommonly good presenter of technical knowledge through the written word.

RAD's Charles Bloom has a blog (https://cbloomrants.blogspot.com) which is excellent for anyone interested in high-performance programming. Especially compression.

If you are interested in compression at all, be sure to take a trip through Charles Bloom's blog [1]. It's an incredible read, he covers everything from the basics all the way through state of the art algorithms. A great example is this post [2], where he talks about how to correctly implement a Huffman encoder/decoder. It's a lot tricker than it is made to sound in most books. For example, most Huffman codes that are used in practice are length limited, to allow the decoder to use smaller lookup tables. There are a bunch of surprisingly interesting tricks to get that to work well from the encoding side (which symbols do you choose to be smaller than they would be otherwise?). [1] http://cbloomrants.blogspot.com/ [2] http://cbloomrants.blogspot.com/2010/08/08-12-10-lost-huffman-paper.ht…

I’d love to see Charles Bloom add LZFSE and ZSTD to his “pareto frontier” charts: http://cbloomrants.blogspot.com (read down a few posts)

One of the oddworld devs, Charles Bloom, is a guru of compression and has a really fascinating blog you should follow: http://cbloomrants.blogspot.com

> In-Place Calculation of Minimum-Redundancy Codes Or in general, refer to "On the Implementation of Minimum Redundancy Prefix Codes" by Moffat and Turpin (1997), as strongly recommended and later explained by Charles Bloom [1]. [1] https://cbloomrants.blogspot.com/2010/08/08-12-10-lost-huffman-paper.h…

Of course there is comparison with LZ4: http://www.radgametools.com/oodlemermaidselkie.htm LZ4HC is LZ4 compressed with the slow compressor to achieve better compression. It is decompressed using regular LZ4 decompressor. They also compare with zstd and lzma, which are the open source competition. I don't know what "specs" you think are missing. They do say which architectures are supported, what the memory use is, whether multi-threading is supported, etc. No public price, true. You can read the blogs of the people who work on this code to get a feel for the caliber: https://cbloomrants.blogspot.com/ https://fgiesen.wordpress.com/ These are known people with known reputation. The product itself is well known in the industry.

Isn't the magic kernel simply a 2× bilinear downscale filter? For example, if you use 2× bilinear downscaling in Tensorflow (as opposed to bilinear downsampling which would end up being just a box filter) using tf.image.resize it uses the magic kernel [1,3,3,1]. There is an article [0] that points this out, although later it goes into recommending odd filters, which insidiously break many applications where you would want to use downscaling by shifting the image by a fraction of a pixel. [0] https://cbloomrants.blogspot.com/2011/03/03-24-11-image-filters-and-gr…

Not just high-performance, his "Library Writing Realizations" post is also well worth a read whatever domain you're working in. http://cbloomrants.blogspot.com/2015/09/library-writing-realizations.h…

Lots more stats and graphs on Charles Bloom's blog (RAD employee working on compression) - http://cbloomrants.blogspot.co.uk/

JPEG has shown amazingly good staying power. I would have assumed "JPEG is woefully old and easy to beat" but Charles Bloom did a good series of blog posts looking at it, and my (non-expert and probably hopelessly naive) takeaway is that JPEG still holds its own for a 20+ year old format. http://cbloomrants.blogspot.com/2012/04/04-09-12-old-image-comparison-…

Interesting that, despite flaunting her edginess, the author never once does actually type out an unadulterated or censored "fuck". The complaint I have with this sort of copy is not being offended, or of being obscene, or what have you--it's that the sort of diction used tends to make me think that whoever wrote it is simply not that bright. If you'd like a good example of decently-written swearing, look at Zed Shaw ( http://programming-motherfucker.com/ ), jwz, or Charles Bloom ( http://cbloomrants.blogspot.com/ ). Those folks clearly rage early and rage often, but they have some skill and class.

In the 90s, Lena was pretty much one of the images you spent a lot of time looking at if you were in compression research. It was part of a standard set of images we were always comparing agains. I want to say the last time I ran across mention of Lena was either here on HN or through a rant of Charles Bloom -- http://cbloomrants.blogspot.com I do find the story fascinating each time I read it.

Entertaining rants, compression (data & image) & general sweng. http://cbloomrants.blogspot.com/

For those interested in the topic, I highly recommend the series of posts from Charles Bloom: http://cbloomrants.blogspot.com/2023/07/notes-on-float-and-multi-byte-… > TLDR: > For S16 deltas, bias by 0x8080 , for S32 deltas, bias by 0x80808080. > De-interleaving multi-byte integers into homogenous streams can help, particularly with weaker back-end compressors. > TLDR: > For floats, just reinterpret F32 as U32. > If the floats have a mix of positive and negative, convert the sign bit to two's-complement signed S32. > Consider lossy elimination of negative zero -0.f > If you didn't actually want the huge precision for floats near zero, a simple lossy encoding is just to do float += constant, which works for non-negative floats where you don't know the high end of the range so you can't just use fixed point. > TLDR: > The best way to compress numeric data that is larger than bytes (F32,F16,S32,S16) is usually to delta them in their original size integer, then de-interleave after the delta. > Sometimes no filter or no deinterleave is best, particularly with stronger compressors, so being able to select filter on/off per-file can give big wins.

See: cbloom's rant on quantization for deeper investigation: https://cbloomrants.blogspot.com/2020/09/topics-in-quantization-for-ga… "While in theory there are cases where you might want to use either type of quantization, if you are in games don't do that! The reason is that the GPU standard for UNORM colors has chosen "centered" quantization, so you should do that too."

bzip2 is particularly slow because the transform it depends on (BWT2) is "intrinsically slow" - it depends on cache-unfriendly operations which can't easily be parallelized: https://cbloomrants.blogspot.com/2021/03/faster-inverse-bwt.html

I tried out hardware accelerated compression on a z15 and it just speeds it up a stupid amount, something like 20x times in decompression and 200 times with compression [0], which is not something that is done many other places yet. Though the PlayStation 5 does have it [1] [0]: https://github.com/rust-lang/libz-sys/pull/72#issuecomment-904020105 [1]: http://cbloomrants.blogspot.com/2020/09/how-oodle-kraken-and-oodle-tex…

http://cbloomrants.blogspot.com/2018/06/zstd-is-faster-than-leviathan.… is versus zstd 1.3.3. There are others too on that blog. Oodle and zstd are both very impressive, it's a shame the former is not free.

If you're interested in this, Charles Bloom has a lot of articles on compression applied to games, in particular UDP packets: http://cbloomrants.blogspot.com/2013/11/11-14-13-oodle-packet-compress…

There is LZNA (http://cbloomrants.blogspot.se/2015/05/05-09-15-oodle-lzna.html) that uses vectorization to update and query its statistical models.

Now here things gets really interesting [0]. And the fact that the claim is that higher level of compression has negligible impact on decode speed making Kraken really impressive for one time compression and lots of decompression situation. [0] http://cbloomrants.blogspot.ca/2016/07/introducing-oodle-mermaid-and-s…

> I think he is the first one to write a practical fast arithmetic coder using ANS. I don't think he is the first; although RAD game tools has been cagey about the details, many strongly suspect their recently announced Kraken, etc. use ANS in some form and some of their previous products; see the discussion here: https://news.ycombinator.com/item?id=11583898 ...and here: http://encode.ru/threads/2492-Kraken-compressor http://cbloomrants.blogspot.ca/2016_05_01_archive.html

The RAD guys have entries in that dept too. "Selkie is 1.3X faster to decode than LZ4 and gets 12% more compression" [1] though "Selkie does not currently have a super fast encoder" [2] [1] http://cbloomrants.blogspot.com/2016/07/introducing-oodle-mermaid-and-… [2] http://cbloomrants.blogspot.nl/2016/07/oodle-selkie.html

There is an attempt to patent something ANS-related: http://cbloomrants.blogspot.com/2015/05/05-21-15-software-patents-are-…

This article has some comparisons between Oodle's LZNA and LZHAM (at least as of a year ago): http://cbloomrants.blogspot.com/2015/05/05-09-15-oodle-lzna.html

FYI: The magic kernel isn't taken nearly as seriously by people in the field of signal processing (almost my field). I'm always surprised as to why it keeps popping up. One of the many articles on the internet explaining why the magic kernel isn't really that magic: http://cbloomrants.blogspot.com/2011/03/03-24-11-image-filters-and-gra…

Sorry guys, but no magic is involved here: http://cbloomrants.blogspot.com/2011/03/03-24-11-image-filters-and-gra…

It's a terrible metric that fails basic dimensional analysis. Anytime you see a ratio of logs, you should already be suspicious. Say algorithm A is twice as fast as algorithm B. Then we'll have alpha r/R log(T)/log(t). The alpha r/R is constant, so the only interesting part is log(T)/log(t) = log(2t)/log(t) = log2/log(t) + 1. We can make this apparently unitless number take any value we want just by changing the unit used to measure time (or if a unit were fixed, by changing the amount of data used to test). It's sort of vaguely superficially sensible, but the idea of being able to reduce comparison of compression algorithms (which fundamentally trade off speed for compression ratio) to a 1-dimensional value is laughable. Charles Bloom's Pareto frontier charts (http://cbloomrants.blogspot.com/2015/03/03-02-15-oodle-lz-pareto-front… are one of the more reasonable options.