Jean-Marc Valin

For a related (but less complicated) project, see Mozilla's RNNoise project, which uses neural networks to better tune classical signal processing algorithms to reduce noise in recorded speech: https://people.xiph.org/~jm/demo/rnnoise/

Additionally, I'd love to see a decoder add noise proportional to the uncertainty and surrounding structure as over smoothed images look unnatural as well. See activity masking demo https://people.xiph.org/~jm/daala/pvq_demo/

> SILK that seemed to be the parent of OPUS That's right - SILK was a codec tuned for voice chat, not music. CELT is the other parent of OPUS and is tuned for music. (Opus uses a hybrid strategy that switches between codecs as necessary.) Don't just look at the benchmarks, at the bottom of the page you can click on the links and listen to a bunch of 64 kbps Opus samples. They're very, very good. And no, they aren't indistinguishable from a CD. As I said in the original comment, they're more than good enough for high quality music streaming for a video chat. > I'd really love to find out that codecs have halved audio size in the past half decade. This has happened, to my ears at least. A little over a decade ago, I could not-infrequently tell a V2 MP3 (~196 kbps) from the FLAC. Took 256 kbps (~V0) to be reliably transparent. (I think it's improved marginally since then, V2 might be transparent almost all the time now.) These days, it's quite infrequent I can tell a 128 kbps Opus file from the FLAC. And the best codecs are quite passable even at 64 kbps. Actually, Opus is "okay" even down to like 48 kbps. Try the sample here: https://people.xiph.org/~jm/opus/opus-1.2/

Update: I did some searching and found some interesting demos of a hybrid of neural nets and more conventional DSP called LPCNet: https://people.xiph.org/~jm/demo/lpcnet_codec/ Sure enough, it was discussed on HN when it came out last year. I think I missed it then. For those who didn't catch this from the URL, this is by Jean-Marc Valin, of Speex and Opus fame.

There's good software to extract voice from audio though! One time I had this webpage [0] open, forgot about it, until a colleague started a phone conversation from the other side of the open space, my laptop's mic picked it up and I got his sharp voice in my headphones. [0]: https://people.xiph.org/~jm/demo/rnnoise/

Perhaps LPCnet may be of use to you. If is a neural net for generating speech based on google's research. The demonstrate using it to intelligently slow down or speed up audio. perhaps you will find 3x audio more natural with it. maybe a smart accelerator could be made that speeds up quiet parts more. https://people.xiph.org/~jm/demo/lpcnet/

GP is right though. It's possible to add options that don't pay for themselves in terms of the entropy of encoding "don't use option X here." Here's an example of a feature that didn't make it into the Daala codec: https://people.xiph.org/~jm/daala/paint_demo/index.shtml

> the audio quality is incredible, though required compute power is ridiculous. LPCnet gives similar quality with much less compuation needed: http://www.rowetel.com/wordpress/?p=6482 https://people.xiph.org/~jm/demo/lpcnet/ https://people.xiph.org/~jm/demo/lpcnet_codec/

It's just personal observation. I noticed the same thing in audio artifacts with the RNNoise demo from xiph.org: https://people.xiph.org/~jm/demo/rnnoise/

Note that the demo link is incorrect, and should be https://people.xiph.org/~jm/demo/lpcnet_codec/

The noise suppression demo is incredible! Scroll down to "Show Me the Samples!" (about 3/4 of the way down) on https://people.xiph.org/~jm/demo/rnnoise/ and hit play and try the buttons. My heavens; this sort of technology would enable, say, a robot to have a much better time understanding what a person is saying in a public place (as well as having possibilities for better chatting for people over the internet).

yes, this is an active area of study. You could check out https://people.xiph.org/~jm/demo/rnnoise/

I enjoyed experimenting with RNNoise for real-time microphone noise suppression. I liked the write-up too [1]. [1]: https://people.xiph.org/~jm/demo/rnnoise/ EDIT: Just realized perhaps you are asking for a Keras-based example? If so, this one's not Keras or TF based.

I'm actually unsure what's meant by reference here. But VP9/10 is the base codec on which AV1 is being built, but many of the most significant innovations from Daala are being folded into AV1. Daala isn't dead and will continue to have value, at the very least as an experimental/development codec, but it may also form the base of a future static image format (it compares very well against JPG and HEVC/BPG. This is comparison is over a year old so it's even better now: https://people.xiph.org/~jm/daala/revisiting/compare.html

Many pieces of Daala are being folded into AV1, which is an industry effort supported by most large software and hardware companies to produce an HEVC replacement. This post summarizes the results of the different experiments: https://people.xiph.org/~jm/daala/revisiting/ Unfortunately some of the most innovative techniques around frequency-domain prediction ended up not panning out. :(

Actually, high quality JPEGs are just as good as BPG, VP8, etc: https://people.xiph.org/~jm/daala/revisiting/subset1_psnrhvsm.png so when you use a better encoder like mozjpeg, JPEG is actually competitive with the newest formats, given enough quality it gets slightly grainier results with blocking artifacts, but BPG and WebM look like they had a painting algorithm blur out all of the details

The goal is to provide a similar enough image to base any residual coding. Xiph.org folks have a good example of exploiting such redundancy in the video compression format [1]. [1] https://people.xiph.org/~jm/daala/paint_demo/ (was later abandoned due to the high cost, and now we have https://people.xiph.org/~jm/daala/deringing_demo/ )

> Therefore, on average, unless the lossless algorithm has a perverse encoding that uses more bits than necessary to encode the possible outputs, no lossless algorithm can beat it on average I think you mean "unless the lossy algorithm has a perverse encoding." But your argument proves too much. It's true that no lossless algorithm can beat a lossy algorithm if averaged across all possible images, for precisely the reason you say. But for the same reason, no lossless algorithm can beat the "compression algorithm" of no compression at all, like BMP but without redundancy in the header, averaged across all possible images. The unknown question — which I think we don't even have a good estimate of — is how much incompressible information is in the images we're interested in compressing. Some of that information is random noise, which lossy algorithms like JPEG can win by discarding. (Better algorithms than JPEG can even improve image quality that way.) But other parts of that information are the actual signal we want to preserve; as demonstrated by Deep Dream, JPEG doesn't have a particularly good representation of that information, but we have reason to believe that on average it is a very small amount of information, much smaller than JPEG files. If it turns out that the actually random noise is smaller than the delta between the information-theoretic Kolmogorov complexity of the signal in the images we're interested in compressing and the size of typical JPEG files…

There's a few recent papers actually that show minor improvements by integrating LPC prediction into deep methods ([0], [1]). In my experience (some of which comes from reproducing these, some of which comes from my own experiments), this isn't actually too useful, at at most offers a minor modeling benefit. The main difference between something like Festival and what we have now is the amount of domain-specific engineering. (This is generally the promise of deep learning -- replace hand-engineered features with simple-to-understand features and a deep model.) If you go and read the Festival manual, you're going to find tons of domain-specific rules and heuristics and subroutines; for example, there's a page on writing letter to sound rules as a grammar [2]. Nowadays, we may have a pipeline that resembles Festival at the high level, but each step of the pipeline is learned as a deep model from data rather than being carefully hand-engineered by many people over the course of years. This yields much more fluid speech as well as much, much faster iteration and experimentation times, leading to faster progress as well. [0] https://arxiv.org/abs/1811.11913 [1] https://people.xiph.org/~jm/demo/lpcnet/ [2] http://www.festvox.org/docs/manual-2.4.0/festival_13.html#Letter-to-so…

One thing I'd love to see follow from this is an image format based on its intra coding. The companies involved expressed interest (https://www.cnet.com/news/google-mozilla-av1-photo-format-could-outdo-… and tests (https://people.xiph.org/~tdaede/av1stilldemo/) look great. The .heic format now used by iPhones is the same idea (https://en.wikipedia.org/wiki/High_Efficiency_Image_File_Format). But wider use of that is going to be constrained by the cost problem HEVC has in general (two patent pools to deal with). Some of WebP is based on VP8 (not VP9) intra coding too. There are a lot of tools packed in AV1's intra coding (and HEVC's, though I've read less about that). Block sizes range from 4x4 to 64x64 and there's a mix within one image, so the encoder can use the right size for the level of detail in each area. There are more ways to predict a block's content from what's to the top and left, which leaves less work for the JPEGish DCT part. There's clever de-ringing post-processing that, in effect, blurs away many of the attention-getting JPEG-y DCT artifacts around edges, while 1) being aware of the direction of the main edge itself to avoid blurring that away and 2) using contrast thresholds to preserve as much other legitimate detail as it can--more about deringing at https://people.xiph.org/~jm/daala/deringing_demo/ . (There's some good detailed discussion at https://parisvideotech.com/wp-content/uploads/2017/07/AOM-AV1-Video-Te… and in…

There was a blog post a few months ago: [0] It looks like there have been a few dead ends -- clever ideas that for one reason or another didn't pan out -- but overall, it's making solid progress. It remains, however, very experimental and is far from complete. One area in which Daala works very well right now is in compression of still images. There's a demo at the bottom of the page which shows it as much better than HEVC for that purpose, and you can also see how the codec has improved over time. Of course, the relative goodness of codecs isn't necessarily consistent over widely varying bitrates. In some of the earlier posts the Daala researchers expressed the belief that the quantitative metrics often don't accurately correspond to subjective perception of quality. Netflix seems to be putting a lot of work into developing better metrics [1], but the Daala researchers seem to rely on subjective A-B tests as the 'gold standard'. [0]: https://people.xiph.org/~jm/daala/revisiting/ [1]: http://techblog.netflix.com/2016/06/toward-practical-perceptual-video.…

> Interesting to see they're also applying Daala to still images. It's a very different set of constraints and optimizations than video. Video codecs have to encode keyframes, which is roughly the same problem as still images. Many still image codecs are modified intraprediction modes from video codecs. So it's really a subset[0] of the constraints and optimizations, not a totally different set. It's also the case we've recently spent a lot of time working on intraprediction issues (and related topics). One reason for that is that the lapping filters make intraprediction very difficult compared to traditional codec designs. Demo2[1] explains much of this difficulty in the context of a solution that we tried and had to abandon. The paint demo[2] explains another solution we tried, which also failed, but which is now the basis of a post-filtering technique. [0] Modulo things like color format support, metadata, etc. [1] http://people.xiph.org/~xiphmont/demo/daala/demo2.shtml [2] https://people.xiph.org/~jm/daala/paint_demo/