Matt Mahoney

jsaxton86's Test Result's: Input File: Title: The Simpsons Movie - 1080p Trailer.mp4 Kind: MPEG-4 movie Size: 147,399,359 bytes (147.4 MB on disk) Video Dimensions: 1920 x 800 Codecs: H.264, AAC Duration: 02:17 Audio Channels: 6 Output Cram: Title: The Simpsons Movie - 1080p Trailer.mp4.cram Kind: Document Size: 113,620,340 bytes (113.6 MB on disk) Time to Cram File: Seconds: 3313.306152 (55 Minutes ~13.3 Seconds) Memory Usage: GB (Rounded): 24.67 GB Hardware Overview: Model Name: iMac Model Identifier: iMac11,1 Processor Name: Intel Core i7 Processor Speed: 2.8 GHz Number of Processors: 1 Total Number of Cores: 4 L2 Cache (per Core): 256 KB L3 Cache: 8 MB Memory: 16 GB Processor Interconnect Speed: 4.8 GT/s Boot ROM Version: IM111.0034.B02 SMC Version (system): 1.54f36 We're really proud to try our best on your test jsaxton86, and we know that test results from the person showing you something always come with a healthy pinch of salt. So we'll ask the community to give it a try, and back us up here. Scientific testing means repeatable parameters, so we urge others to take his test too. (--EDIT: Turns out this file is too big 50MB in the limit of Github push-- We're also making our Cram file available on github, check it out with the decompressor or in a Hex editor: https://github.com/Cramcore/Crams) From these results you can clearly see areas we aren't happy with either. Time and memory, but this is just the beginning for…

I love it! This was what the ultra marathon community used to be like before it became commercialized and professionalized (https://mattmahoney.net/ultra/, especially the story of Gravel Man https://mattmahoney.net/ultra/death600.txt). Now you can't run 50k without a crew. EDIT: And this is what a serious amateur can do on that route: https://www.latimes.com/california/story/2022-03-10/astrophysicist-cro…

It's worth noting that the benchmark has not been updated as frequently for the past several years, and some versions of compressors are quite far behind the current implementations (http://www.mattmahoney.net/dc/text.html#history). The one instance I double-checked (zstd) I don't recall it making a massive difference, but it did make a difference (iirc, the current version was slightly smaller than what was listed in the benchmark).

For anyone interested in the equivalence between AI and compression, take a look at the Hutter Prize :) http://prize.hutter1.net/ Also worth a look is the Large Text Compression Benchmark http://mattmahoney.net/dc/text.html - currently the world's best compressor is a neural network made by ... the renowned Fabrice Bellard, creator of ffmpeg and QEMU! And I really dig these pages' refreshingly appropriate text-only style!

That being said: the best results currently on the large text compression benchmark utilize ML: http://www.mattmahoney.net/dc/text.html

> As an example of a numerical value, GPT-2 achieves 1 bit per character (=token) on a Wikipedia data set https://towardsdatascience.com/perplexity-of-language-models-revisited… (I have not checked this claim, it's just what I found from googling "best large-language model character-perplexity 2023".) A token in a LLM is generally more than one character, so I would guess that the entropy is a bit lower than that. Shannon estimated it at 0.6-1.3 bits/character in 1950 (https://mattmahoney.net/dc/entropy1.html)

Might be worth looking at a few of the better ones from this list: http://mattmahoney.net/dc/text.html Some 200 programs tested against a large set of wikipedia data.

I'd recommend checking out zpaq[1], it purposed for backups, and has great compression (even on low setting) for large 100GB+ file collections. However for smaller stuff I use zstd at level 22 in a tar for most things since it's much faster, though a little heavier. [1] http://mattmahoney.net/dc/zpaq.html

I'm far from an expert in this subject but doesn't this ranking of large text compression algorithms with NNCP coming first suggest that neural-nets are pretty great at compression? http://mattmahoney.net/dc/text.html https://bellard.org/nncp/ I don't see examples of high performing symbolic AI based compression algorithms anywhere, but again I am very ignorant, do you have examples?

This is very interesting but for people who are not from the field note 1. The compressors he is comparing to *are* neural networks themselves (PAQ) 2. He is using a rather small corpus enwik8 (100,000,000 bytes), instead of the common enwik9 3. He is not publishing the amount of RAM or hardware he used http://mattmahoney.net/dc/dce.html (seems down) https://web.archive.org/web/20210216231934/http://mattmahoney.net/dc/d… It seems Bellard is starting to play with these. I bet he will find interesting things and I would love to see him interact in the Russian compression webforum (please don't link). Big names in compression like Yann Collet, Matt Mahoney, and Jarek Duda used to hang out there.

Here's great book on this http://mattmahoney.net/dc/dce.html The guy was using neural networks for compression long time ago before it was a thing again. EDIT: Oh, somebody mentioned it already (but it's really good, free & totally worth reading)

DEFLATE compressed data consists of blocks. Each block can be either uncompressed, or compressed with a pre-defined Huffman table (the variant mentioned in the parent comment), or compressed with a dynamic Huffman table. Many DEFLATE compressors would consider all three options (at higher -j values) and choose the best. So, one does not need to tune DEFLATE to get the advantage of dynamic Huffman tables. And any kind of arithmetic encoding (or the new hotness, Assymetric Numeral Systems) would do a better job. For a competition like this, all simple approaches have probably been tried already. The current leader has been tuning their approach for 5 years: http://mattmahoney.net/dc/text.html#1159

reminds me of zpaq [1]. It is an public domain compression tool, which save the decompression algorithm as IR into the compressed archive. It is written by Matt Mahoney, best known for the PAQ archiver series. [1]: http://mattmahoney.net/dc/zpaq.html

Because the author compares to the best domain-specific compressors (by compression ratio) for each dataset instead. For text, he used CMIX[1]. For DNA sequences, he used MFCompress[2]. [1] http://mattmahoney.net/dc/text.html [2] http://bioinformatics.ua.pt/software/mfcompress/

My favorite post-race report is from the infamous Andrew Thompson run of 2005, in which a promising young runner headed into the fifth loop with a good time and then completely lost his mind from sleep deprivation, spending the next hour vividly hallucinating and wandering aimlessly through the woods. http://www.mattmahoney.net/barkley/2005/andrew.html

A similar approach has also been developed for use with zpaq, a compression format which stores the decompression algorithm as bytecode in the archive: [The configuration] "jpg_test2" by Jan Ondrus compresses JPEG images (which are already compressed) by an additional 15%. It uses a preprocessor that expands Huffman codes to whole bytes, followed by context modeling. http://mattmahoney.net/dc/zpaqutil.html

It's interesting to note that in every year that someone finishes, they add additional parts to the course making it more difficult for the competitors in the following year. In 2013, Nick Hollon finished in 57:41 and Travis Wildeboer in 58:41. A new hill was added, "foolish stu", increasing the total climb to over 60,000 feet. In 2014, Jared Campbell got his second finish in 57:50. Another new hill, Hiram's Vertical Smile, brings the total climb to 62,680 ft.[1] 1- http://www.mattmahoney.net/barkley/

Zstd and LZ4 look amazing. Here's another compressor I really like: http://mattmahoney.net/dc/zpaq.html

BWT-based compressors like bzip2 do best when their inputs have highly repetitive structure. In a JSON file with many repeated keys, the information that `"k` is usually followed by `ey_1": "` is compressed very effectively. For similar reasons, bzip2 tends to outperform gzip on executables-- it can more effectively model the conditional probability of opcode sequences. See http://mattmahoney.net/dc/dce.html#Section_55 for more discussion

Quite a bit. For excellent introduction to the topic, read this: http://mattmahoney.net/dc/dce.html For fast lossless compressors, see this article using an Apple II to compare the best of today's (LZ4) with what was available when the hardware was popular/new.

That's a good classic, but don't forget to also read his excellent writings of the connections between data compression and artificial intelligence: http://mattmahoney.net/dc/rationale.html Also, his book "Data Compression Explained" was a great help and eye opener for me: http://mattmahoney.net/dc/dce.html

For people who want to understand the link between compression and artificial intelligence: The Hutter Prize pages have some nice descriptions. (http://prize.hutter1.net/index.htm) It would be interesting to see if enwiki8 can be compressed further. (http://mattmahoney.net/dc/textdata.html) Matt Mahoney has a lot of great information about compression! The compression curiosities page has some nice gimmicks. A file that compresses well in with one software, but hardly at all with another; a tiny compressed file (115 bytes) that expands to 5 MB; compressed files decompressing to themselves. (http://www.maximumcompression.com/compression_fun.php)

For any algorithm that’s heuristically searching a very large space (in this case, the space of model parameters to compress a given string), you can expect it to do better given more time. So it would actually be surprising if the best compressors, in terms of compression ratio, were fast. In practice, the competitions that people like to use to measure which compressor is “best” have resource limits, and compressors from the PAQ family are at or near the top of most rankings. http://mattmahoney.net/dc/text.html has some nice charts of the Pareto frontier for at least one competition. This lets you see the most efficient compressor for a given compression ratio and vice versa. The log scale gives a sense of the diminishing returns: it’s really easy to compress most structured data to 0.5 its original size, but getting from say 0.214 to 0.213 can be a hell of a lot of work.

Interesting approach. The fastest of the 4 presented compressors ("LSTM (small)") is 24 times slower than xz, and their best compressor ("LSTM (large1)") is 429 times slower than xz. Let alone gzip or, presumably, zstandard (not shown in paper). They also ran the models on different CPUs (a Core i5 and a Xeon E5) so the results are not even comparable within the same paper. Another site list's the author's decompression times, which are even worse: xz decompresses twelve thousand times faster (50MB/s vs. 4kB/s) when nncp has an Nvidia RTX 3090 and 24GB RAM available to it, which apparently speeds it up by 3x compared to the original CPU implementation. At half the size of xz's output, there can be applications for this, but you need to: - not care about compression time - not be constrained on hardware requirements (7.6GB RAM, ideally let it run on a GPU) - not care about decompression time either - and the data must be text (I can't find benchmarks other than from English Wikipedia text, but various sources emphasize it's a text compressor so presumably this is no good on e.g. a spacecraft needing to transmit sensor/research data over a weak connection, even if the power budget trade-off of running a GPU instead of pumping power into the antenna were the optimal thing to do) [1] https://www.mattmahoney.net/dc/text.html

I'm doing my PhD in compression-based machine learning, wanted to contribute a few clarifying points. The relationship between probabilistic modeling and lossless compression is very direct. A model that predicts the next symbol with probability p can on average losslessly compress that symbol with the help of an entropy coder (e.g. arithmetic coding) in -log(p) bits. Therefore improved probabilistic models immediately translate into improved lossless compressors. There are two ways people have used compressors for ML: the first is based on the Minimum Description Length (MDL) principle [0] which says that the best model is the one which provides the shortest description of the data, counting the size of the model itself. This is similar to the technique used in this blog post (argmin code length across class-conditioned compressors) except for counting the model size. Basically you can train a compressor per class and then choose the class compressor which best compresses the test data. It is a maximum likelihood argument because of Shannon's source coding theorem: a code length L corresponds to a probability 2^-L. The second way is the Normalized Compression Distance (NCD) [1], which uses code lengths to calculate information-theoretic distances which can then be plugged into distance-based algorithms like kNN. MDL interprets compressed lengths as likelihoods, while NCD interprets them for distance calculations. The theoretical foundation is Kolmogorov complexity which…

It's actually not the best at enwik8 or 9. The results at https://www.mattmahoney.net/dc/text.html explicitly add the size of the compressor itself to the result. Note the "enwik9+prog" column. That's what it's ranked on. The reason to do this is that it's trivial to create a compressor that 'compresses' a file to 0 bytes. Just have an executable with a dictionary of enwik9 that writes that out given any input. So we always measure what is effectively the Kolmogorov complexity. The data+program as a whole that produces the result we want. So those results add in the compressor size. The programs there generally have no dictionary built in or in the case of LLM based compressors, no pre-trained data. They effectively build the model as they process data. Not compressing much at all at the start and slowly compressing better and better as they go. This is why these programs do better and better with larger data sets. They start with 0 knowledge. After a GB or so they have very good knowledge of the corpus of human language. This program here however is pre-trained and shipped with a model. It's 150MB in size! This means it has 150MB of extra starting knowledge over those models in that list. The top models in that list are the better compressors, they'll quickly out learn and overtake this compressor but they just don't have that headstart. Of course measuring fairly this should be listed with that 150MB program size added to the results when doing a comparison.

When people claim that there is such a thing as "X% accuracy in reasoning", it's really hard to take anything else seriously, no matter how impressive. AI (and humans!) aside, claiming that there was an oracle that could "answer all questions" is a solved problem. Such a thing cannot exist. But this is going already too deep IMO. When people start talking about percentages or benchmark scores, there has to be some denominator. And there can be no bias-free such denominator for - trivia questions - mathematical questions (oh, maybe I'm wrong here, intuitively I'd say it's impossible for various reasons: varying "hardness", undecidable problems etc) - historical or policital questions I wanted to include "software development tasks", but it would be a distraction. Maybe there will be a good benchmark for this, I'm aware there are plenty already. Maybe AI will be capable to be a better software developer than me in some capacity, so I don't want to include this part here. That also maps pretty well to "the better the problem description, the better the output", which doesn't seem to work so neatly with the other categories of tasks and questions. Even if the whole body of questions/tasks/prompts would be very constrained and cover only a single domain, it seems impossible to guarantee that such benchmark is "bias-free" (I know AGI folks love this word). Maybe in some interesting special cases? For example, very constrained and clearly defined classes of questions, at…

Interesting, I've recently spent an unhealthy amount of time researching archival formats to build the same setup of using SQLite with ZStd. My use case is extremely redundant data (specific website dumps + logs) that I want decently quick random access into, and I was unhappy with either the access speed, quality/usability or even existence of libraries for several formats. Glancing over the code this seems to use the following setup: - Aggregate files - Chunk into blocks - Compress blocks of fixed size - Store file to chunk and chunk to block associations What I did not see is a deduplication step for the chunks, or an attempt to group files (and by extend, blocks) by similarity in an attempt improve compression. But I might have just missed that due to lack of familiarity with Pascal. For anyone interested in this strategy, take a look at ZPAQ [1] by Matt Mahoney, you might know him from the Hutter Prize competition [2] / Large Text Compression Benchmark. It takes 14th place with tuned parameters. There's also a maintained fork called zpaqfranz, but I ran into some issues like incorrect disk size estimates with it. For me the code was also sometimes hard to read due to being a mix of English and Italian. So your mileage may vary. [1]: http://mattmahoney.net/dc/zpaq.html [2]: http://prize.hutter1.net [3]: https://github.com/fcorbelli/zpaqfranz

Impressive! But an order of a magnitude slower then non-LLM compression techniques. I try to compare with the results from the top 9 of this enwik8 compression test by Matt Mahoney: https://www.mattmahoney.net/dc/text.html The durilca compressor by Dmitry Shkarin (what happened to him?) has been the fastest compressor in the top since it's debut in 2006. The original size, enwik8, is 10^8 bytes. The compressed size is the compressend enwik8 plus the size of a zip archive containing the decompressor. The bpb value of rwkv_430M on enwik8 is 0.948 bpb, the 7B models will be lower, perhaps around 0.800. So if my calculations are correct, the LLM's can perform 50% better then the best performing conventional compressors excluding the zipped LLM decompressor code (I am unsure about the size). The bpb ratio for each existing program can be calculated as: bpb ratio=(Original size (enwik9)Total size (enwik9+prog) )×8 nncp v3.1: Given compressed size is not provided, we cannot calculate bpb for nncp v3.1 using enwik8. cmix v19: bpb=(14,837,987+223,485108)×8bpb=(10814,837,987+223,485 )×8 bpb≈1.205 tensorflow-compress v4: bpb=(15,905,037+55,283108)×8bpb=(10815,905,037+55,283 )×8 bpb≈1.272 cmix-hp 10 Jun 2021: Given compressed size is not provided, we cannot calculate bpb for cmix-hp using enwik8. fast-cmix: Given compressed size is not provided, we cannot calculate bpb for fast-cmix using enwik8. starlit 31 May 2021: bpb=(15,215,107108)×8bpb=(10815,215,107…

Sure! The current best achieved compression for English text (well, specifically, Wikipedia articles) is Fabrice Bellard's nncp, which uses... transformers. http://mattmahoney.net/dc/text.html#1085 However, most research into reducing the model perplexity of text (increasing the compression ratio when the model is used for encoding) uses fixed language models which are learnt offline. Obviously, they use transformers too. You can see some of them here: https://paperswithcode.com/sota/language-modelling-on-wikitext-103 However (token) perplexity is only comparable across models if the same tokenizer is used. The above page seems to compare papers using the GPT-2 tokenizer. I would assume that if you controlled for differences in tokenizers, GPT-4 without RLHF would achieve the highest compression. It should be obvious that RLHF increases perplexity because it changes the training objective to something other than predicting text.