Jean-Marc Valin

I'm often thinking the same, and am only aware of a single popular library doing that: rnnoise [0] This library combines "classic" digital signal processing with a smaller RNN. As a result, it's smaller, faster and probably also has less uncanny edge cases than approaches that use an RNN for the complete processing chain. I think many use cases could benefit from this approach. [0] https://jmvalin.ca/demo/rnnoise/

A codec that uses AI isn't necessarily going to be using it to synthesize content. It could use it for things like improving rate-distortion optimizations and early-skip heuristics. Modern video codecs are complex beasts. It's not as simple as "take a macroblock and find some motion vectors that minimize residual below a certain threshold, else code as intra-block". They have hundreds of mutually-exclusive techniques to compress a certain area of the frame. Determining which technique will require the smallest residual is done with fast early-skip heuristics that often make the wrong decision. The official manual for x265, the H.265/HEVC encoder (https://x265.readthedocs.io/en/master/cli.html), has literally hundreds of options, almost all about tuning these myriad of heuristics for your particular input. AI can be used to enhance things like early-skip heuristics. "This block looks like it'll benefit from a really-detailed motion search in this particular area" or "we'll save bits if we bypass the DCT step and quantize the block directly" or "this frame should definitely be a B-frame". Encoders already use heuristics to do this (brute forcing all possible decisions to find which is optimal is too slow), but they don't always make the best decision. An AI could be used to improve that. Now when I say AI, I'm not talking about massive, multi-billion weight monstrosities that synthesize nonsense, but extremely simple neural networks with a few thousand weights. The popular…

Also, while the author complains that there is not a lot of high quality data around [0], you do not need a lot of data to train small models. Depending on the problem you are trying to solve, you can do a lot with single-digit gigabytes of audio data. See, e.g., https://jmvalin.ca/demo/rnnoise/ [0] Which I do agree with, particularly if you need it to be higher quality or labeled in a particular way: the Fisher database mentioned is narrowband and 8-bit mu-law quantized, and while there are timestamps, they are not accurate enough for millisecond-level active speech determination. It is also less than 6000 conversations totaling less than 1000 hours (x2 speakers, but each is silent over half the time, a fact that can also throw a wrench in some standard algorithms, like volume normalization). It is also English-only.

Might be useful to split it up to use the same speech features as some off the shelf vocoder, such as FARGAN used by RADE (https://freedv.org/radio-autoencoder/) and DRED (https://jmvalin.ca/papers/valin_dred_journal.pdf).

There's also the LPCNet Codec (2019), which does wideband speech at 1.6kb/s by using a recurrent neural network. https://jmvalin.ca/demo/lpcnet_codec/

Some "didn't make it in" things that looked promising were the perceptual vector quantization[1], and a butterfly transform that Monty was working on, IIRC as an occasional spectator to the process. [1] https://jmvalin.ca/daala/pvq_demo/

Created a few years ago by Jean-Marc Valin of xiph/mozilla (who by the way is also the author of Opus codec among other things): https://gitlab.xiph.org/xiph/rnnoise/ Overview of RNNoise from the horse's mouth is here: https://jmvalin.ca/demo/rnnoise/ Used as a Wasm module! In some ways the web is becoming more opaque. Is this the future then, a hodgepodge of binaries doing things behind the scenes? Though in this case it happens to be OSS, and it may well be a moot point -- backend is already a blackbox to the enduser, now parts of frontend are blackboxes. The practical implication is probably just that some measure of customizability is gone.

I'm curious how https://jmvalin.ca/demo/rnnoise/ would do. Might work.

I believe iOS 15 includes a little-touted feature: "Voice Isolation" mode; I haven't tested it comprehensively, but other similar technology (e.g. RNNoise [1]) has very impressive results in the scenarios you allude to. [1] https://jmvalin.ca/demo/rnnoise/

opus has an ambisonics profile: * https://datatracker.ietf.org/doc/html/rfc8486 * https://jmvalin.ca/opus/opus-1.3/

The documentation is a bit poor. The original data is available for download (with more info about the entire process, most of which is outside of my grasp as I am not an ML person) in the demo blog post: https://jmvalin.ca/demo/rnnoise/ (towards the bottom of the page)

For me noisetorch [0] (based on RNNoise [1][2]) works pretty good. I'm using RTX Voice while I'm working on Windows and noisetorch while working on manjaro. RTX Voice seems to work a bit better, but noisetorch compares really well to it. I have a microphone which has static noise and is not really mechanical decoupled from my table. Both applications are good at cancelling that kind of noise. Noisetorch is a bit more focused on suppressing noise while you're not talking, which is the use case of the posted article. RTX Voice is better at suppressing noise while talking, which you can hear in Nvidias demos as well. [0] https://github.com/lawl/NoiseTorch [1] https://github.com/werman/noise-suppression-for-voice/ [2] https://jmvalin.ca/demo/rnnoise/

Opus uses a neural network to determine which codec to use: https://jmvalin.ca/opus/opus-1.3/

From the developer of RNNoise, which is the technique being used here: "As strange as it may sound, you should not be expecting an increase in intelligibility. Humans are so good at understanding speech in noise that an enhancement algorithm — especially one that isn't allowed to look ahead of the speech it's denoising — can only destroy information. So why are we doing this in the first place? For quality. The enhanced speech is much less annoying to listen to and likely causes less listener fatigue" https://jmvalin.ca/demo/rnnoise/

Here are some demos, I believe this is the same algorithm: https://jmvalin.ca/demo/rnnoise/

Codecs like AV1 do a LOT of motion and transformation detection. That's generally where a lot of that ram goes, They are storing a metric ton of matrix data from prior (and future? I'm not sure if AV1 does this) frames so that when calculating the current frame stream they can quickly look up that info. Needless to say. figuring out which piece of data should be plucked out and transformed from prior frames is really computationally expensive. Here's a paper on exactly what sort of tools are available for the codec. https://jmvalin.ca/papers/AV1_tools.pdf edit I should note that the data stream for H.264 is FAR simpler. It doesn't support nearly the number of transformations that AV1 does.

Apologies that should be "inequity caused by gop length" 1) AV1 has significant advantages in its inter compression vs HM. AV1 also has explicit tools for dealing with short GOPs that where not tested. See: https://jmvalin.ca/papers/AV1_tools.pdf for inter prediction. 2) AV1 and HM both have a different set of tools for intra prediction and as such different coding efficiency on key frames. 3) Yeah and two second gops are really bad for video quality. 4-6s is far more common except on latency sensitive applications. I'll also point out that the reported MOS scores have several oddities, like an 8mbps av1 stream being rates lower than it's own 6mbps stream.

This page implies that the grain synthesis step is informed by analysis of the grain in the uncompressed video: https://bitmovin.com/cool-new-video-tools-five-encoding-advancements-c… That is confirmed by this article: https://jmvalin.ca/papers/AV1_tools.pdf

Interesting -- and unexpected. I also wrote the Speex suppressor and one of the things that specifically annoyed me about it was the robotic noise and the pseudo-reverberation it adds to the speech, but it seems like some people (like you) like that. Trying to understand exactly what you don't like about RNNoise... is it how the remaining background noise sounds or how sharply it turns on/off? I did a quick hack to RNNoise to smooth out the attenuation and prevent it from cancelling more than 30 dB. I'd be curious if it improves or makes things worse for you (compared to the samples in the demo): https://jmvalin.ca/misc_stuff/rnn_hack1/

Daala's methodology of video compression differs from HEVC, namely it optimizes the perceived quality of the image. So in theory it can be computationally lighter, because it can save on areas which are affecting perception less. But it's not there yet. Here is an overview of this idea: http://jmvalin.ca/video/spie_pvq_abstract.pdf I'm not sure though how exactly it translates into still images compression efficiency. For video they do plan to eventually beat HEVC both on quality and algorithmic delay.

https://jmvalin.ca/demo/rnnoise/ says: > So what should you listen for anyway? As strange as it may sound, you should not be expecting an increase in intelligibility. Humans are so good at understanding speech in noise that an enhancement algorithm — especially one that isn't allowed to look ahead of the speech it's denoising — can only destroy information. So why are we doing this in the first place? For quality. The enhanced speech is much less annoying to listen to and likely causes less listener fatigue. > Actually, there are still a few cases where it can actually help intelligibility. The first is videoconferencing, when multiple speakers are being mixed together. For that application, noise suppression prevents the noise from all the inactive speakers from being mixed in with the active speaker, improving both quality and intelligibility. A second case is when the speech goes through a low bitrate codec. Those tend to degrade noisy speech more than clean speech, so removing the noise allows the codec to do a better job. I do think that for direct listening, the jitsi.org speech samples would be slightly more intelligible if the noise removal was tuned to pass through frequencies with mixed noise and signal. I don't know if that would be worse in a video conference. Does the speaker or listener get to choose between conservative and aggressive noise removal?