Diary Of An x264 Developer

But they do not provide SSIM comparisons, they compare PSNR. Here is "how to cheat with codec comparisons", http://x264dev.multimedia.cx/?p=472 which exaplains (partially) why PSNR isn't that great when judging image quality. SSIM: http://en.wikipedia.org/wiki/Structural_similarity

Another article that recently reaffirmed my distaste for software patents is this article on the future of web video by an x264 developer: http://x264dev.multimedia.cx/?p=292 In a summarising comment: "The best choice [for web video] will always be a deeply patent-encumbered format until someone gets rid of software patents. There is no other feasible solution."

Especially since this states H.264 compliance, I'd be willing to say this can't possibly live up to it's claims. Video encoders are notorious for their bloated claims and faulty comparisons. After a quick perusal of their site I see they compare themselves only to web-m, and use generalized %'s to claim differences between products and processes. They say they've optimized for human perceptual quality. Perhaps they have. But did they really do a better job than the open source H.264 alternative (x264) used by everyone from YouTube and Hulu, to Facebook? One thats been in development for _years_ now and beats just about everything thrown at it in a fair comparison? I think not. I'd love to be wrong though. When they do eventually come out with comparisons, I hope they have read this first : http://x264dev.multimedia.cx/archives/472

That he has. My test clip choice was inspired by him[1], and I also link to another blog post of his[2] in the conclusion. That VP8 blog post is almost three years old now, though, and the comment I'm replying to in my article claimed that VP8 is better than H.264 "at this point". This is why I did my test with the latest and greatest encoders available for both formats today. [1] http://forum.doom9.org/showthread.php?t=154430 [2] http://x264dev.multimedia.cx/archives/472

His blog is very insightful. There is also another great post where he talks about how to cheat on video comparison tests ( http://x264dev.multimedia.cx/archives/472 ). One of the things brought up in there is for still frame comparisons. If the frame came from an I frame rather than a B* or P frame in the output video, the test will be unfair. *VP8 doesn't have B frames, similar to the h.264 baseline profile.

I was under the impression that Wavelets were something of a dead-end. As far as I picked it up the wavelets need to be tuned for your output. People have tuned them for great PSNR output but not for SSIM or other perceptual outputs so you get output that is blurry sometimes even compared with JPEG. I'm not sure if this is a fundamental problem with the technique, or just something that's currently impracticable compared with less elegant but working alternatives. There's more here (though it's talking about video issues too): http://x264dev.multimedia.cx/archives/317

required reading: http://x264dev.multimedia.cx/archives/541 - don't know how relevant it is now, but still a good read.

Are there any recent third party comparisons between x264, WebP and JPEG encoded images like this one from September 2010 [1] by Jacob Garrett-Glaser? From the WebP gallery [2], it looks like JPEG and WebP are pretty comparable for those selections of images (with Nærøyfjorden, Norway having the greatest difference). The comparison study [3] looks favorable to WebP at all comparable SSIM. Is anyone familiar offhand with the algorithmic complexity of WebP compared to JPEG? And finally, does the 16383x16383 maximum WebP resolution limit WebP (compared to JPEG at 65535x65535), or will that limit increase as the ability to easily create gigapixel images increases? [1] http://x264dev.multimedia.cx/archives/541 [2] http://code.google.com/speed/webp/gallery.html [3] http://code.google.com/speed/webp/docs/webp_study.html

The following blog posts from an x264 developer point out problems with the spec, including discrepancies with the implementation and chunks of cut&pasted C code: http://x264dev.multimedia.cx/archives/377 http://x264dev.multimedia.cx/archives/486 Unfortunately, the VP8 Internet-Draft seems to be just a plaintext version of the bitstream guide complained about above, including the disclaimer that the implementation takes precedence and the snippets of C code. It would need a lot of work to actually be published as any kind of standard, this perhaps being a reason Google is going the individual submission RFC route.

> The only operative difference between the two codecs is that WebM is free. by moving many features of the codec into postprocessing, the video format becomes scalable; a decoder can do “less work” while still playing back the file, albeit at a lower quality. -- http://x264dev.multimedia.cx/archives/292 I have naively taken that to mean that the trade-off of resolution for battery life can be handled in the decoder; that fewer encodings need to be stored on the server to target the same wide range of devices, vs. h264. I would love to know whether this is true, and what other practical differences there are between the two codecs from the POV of content producers. edit: VP8 ... Technically, it produces output on par with H.264 High Profile, while maintaining a low decoding complexity on par with H.264 Baseline. -- http://diveintohtml5.org/video.html#vp8

To get clarity of thought on the debate, go back and read http://x264dev.multimedia.cx/archives/377 Everybody seems to be crying "but hardware support is coming soon!" without seeing it's difficult: "The unfortunate problem with this is that it’s a nightmare for hardware implementations, greatly increasing memory bandwidth requirements." It comes down to WebM/VP8 being inferior (in specification and implementation) to an existing widely deployed (in bits and gates) codec. Feel free to argue the fallacies of the osnews questions if your time is worth nothing.

From the bits and pieces that I have read, there are some very real patent questions still looming. Jason Garett-Glaser put together a great comparison of h.264 vs VP8/WebM. Check it out here: http://x264dev.multimedia.cx/archives/377 His summary: the spec is really terrible, the performance leaves a lot to be desired (though, there is a lot of room for optimization), and it 'copies too much from H.264 for comfort'.

WebM is not that great technically (http://x264dev.multimedia.cx/archives/377) and is unlikely to remain royalty-free if it ever takes off as there are many patents that could apply to it's code. H.264 will play in both flash and the browser. It's a standard codec that is used in most workflows already. On top of that, MPEG-LA has announced it will always be royalty free for video that is free to end users (http://www.macworld.com/article/153692/2010/08/h264_royalties.html). There's no reason for Firefox and Opera to not pick up H.264. It is the only choice that makes sense if HTML5 video is going to take off, and they are holding the web back. Thumbs up to Microsoft for actually trying to push us forward this time.

A very good article about vp8 vs x264 vs jpeg: http://x264dev.multimedia.cx/archives/541 The comments are also worth reading.

Page not coming up for me, but over at http://lwn.net/Articles/407884/#Comments there was a link to another less-than-thrilled appreciation: http://x264dev.multimedia.cx/?p=541 (by Jason Garrett-Glaser, aka Dark Shikari, x264 Developer)

There's more than one "objective" test for image quality, and some of them are closer to human perception than PSNR. The trouble with comparisons like this is that the JPG encoder might be tuned to minimize PSNR, but if the WebP encoder is tuned to optimize for something else, of course it won't perform as well when you compare the PSNR! More info on the issues with this type of benchmark from Jason Garrett-Glaser (a.k.a. Dark Shikari), an x264 developer: http://x264dev.multimedia.cx/?p=458

This is awesome. Congratulations to the always-incredible developers of ffmpeg. One thing is worrisome about this: since H.264 (the current industry standard video codec) and VP8 are highly similar, we can share code (and more importantly: optimizations) between FFmpeg’s H.264 and VP8 decoders So it looks like VP8 is so similar to H.264 that decoders for the two can share quite a bit of code (something Jason Garrett-Glaser suggested might be the case: http://x264dev.multimedia.cx/?p=377 ). Have to admit that this doesn't seem to bode too well for VP8's claim to relative freedom from patent litigation.

“What I’d heard from ex-on2 folks was that there is some philosophical disagreement about how to optimize [encoder] tuning, and the tune for PSNR camp mostly won out.“ Apparently around the time of VP6, On2 went the full-retard route and optimized purely for PSNR, completely ignoring visual considerations. This explains quite well why VP7 looked so blurry and ugly." -- http://x264dev.multimedia.cx/?p=292

In the last Theora/H.264 thread on HN, I recall somebody saying that VP8 was better than H.264... when VP8 was new, and H.264 encoders were still largely experimental. Nowdays, things like x264 and commercial codecs shipped with QuickTime are better than VP8. There's an interesting post about the problems with VP8 posted by an x264 developer here: http://x264dev.multimedia.cx/?p=292

I don't think there are technical limitations holding this back. You can already get the necessary low-latency video encoding as open-source software, even: http://x264dev.multimedia.cx/?p=249 Gaikai seems to have pretty good business model to make it happen, too - better than OnLive anyway.

As the lead developer, I can say quite confidently that the encoder matters a huge amount more than the standard. See: http://x264dev.multimedia.cx/wp-content/uploads/2009/08/quality_chart1… http://x264dev.multimedia.cx/?p=164 http://x264dev.multimedia.cx/?p=102 As you can see, a bad H.264 encoder (e.g. Apple's) can barely compete with Xvid. The entirety of my work on x264 is contained in the past 2 years of development. The psychovisual optimizations alone probably account for at least a 50-70% improvement in compression quality.

I'm pretty sure h264 is the superior codec in all regards except for that pesky license issue. It encodes faster and creates smaller files that look better. From (lead x264 developer and HN user) DarkShikari[1]: "Overall, VP8 appears to be significantly weaker than H.264 compression-wise. The primary weaknesses mentioned above are the lack of proper adaptive quantization, lack of B-frames, lack of an 8×8 transform, and non-adaptive loop filter. With this in mind, I expect VP8 to be more comparable to VC-1 or H.264 Baseline Profile than with H.264. Of course, this is still significantly better than Theora, and in my tests it beats Dirac quite handily as well. ... Finally, the problem of patents appears to be rearing its ugly head again. VP8 is simply way too similar to H.264: a pithy, if slightly inaccurate, description of VP8 would be “H.264 Baseline Profile with a better entropy coder..." I won't pretend to know what all of those terms mean off the top of my head, but if the lead x264 developer sees absolutely no technical advantage in WebM vs. h264, that sounds pretty damning to me. As for why x264 encodes h264 so much faster than ffmpeg or whatever encodes WebM, the simplest explanation would be that there is much greater demand for an optimized h264 encoder. WebM has some admittedly large users (Wikimedia Foundation, Youtube for browsers with no h264 support and no flash player, and now 4chan), but they're still vanishingly small in comparison to the users of h264…

I agree with everything but your last point: > Apple should have just completely created their own format instead of extending ePub, and we wouldn't be having this discussion. Now Apple has become the convenient scapegoat for ePub's failure, whilst others, most notably Amazon, get a free pass for doing exactly the same. That seems like a terrible reason to head off on your own and create a totally new format. Case in point, look at audio/video CODECs. Under the hood, many of them have startling similarities. The consumer has no idea they're similar, because they're named differently. More directly, what is wrong with extending ePub, provided you don't represent it as ePub. Gruber's best points are made on a technical basis, not a business basis. Apple isn't calling the iBooks Author output ePub. That would be egregious. As is typical in the technology space, everyone is conflating arguments. They're pissed about Apple's business tactics, and so the argument spreads like fire in a field of dry grass. From a black box perspective, iBooks files are proprietary. Whether their "based on" ePub is no more relevant than the fact that VP8 bears similarities to H.264 and many other video CODECs [1]. 1 - http://x264dev.multimedia.cx/archives/377

> However, I was merely a contractor, and not making the business decisions. I worked with multiple contractors that thought similar things to yourself Do you know you're not talking to some contractor, you're talking to the guy behind x264? If you don't know him, read his analysis of VP8 (note copyright footer): http://x264dev.multimedia.cx/archives/377 Having been in the streaming industry since the mid 90's, with heavy work in live and on demand encoding, I've seen dozens of these companies make similar claims, usually in pursuit of investment dollars. For years running, regardless of open source versus closed source, none stack up well against x264 when measured for the way people see video and for the resources taken to produce the encoded content. You really don't have to read past the first graph: http://compression.ru/video/codec_comparison/h264_2011/ This is an informative study, and has been for the last seven years. If you have a better compression that would let me, as a CDN, offer clients movie delivery to users with enough less bandwidth and storage it's worth retooling for, we're all ears. Again, I've talked to dozens upon dozens. None really had it. So far, given their original source, I could personally produce an even smaller x264 file that end users prefer.

Your interpretation doesn't match those of people who have analyzed it. The problem with WebM is that it may infringe on H264 patents! And its not clear they have done enough to avoid the problems. H264 has come together to form a very strong patent pool. None of the patents can be asserted against H264 (nor can the partner companies in general), but any of them can be asserted against WebM. Read JGG's account, which is still the best account to date at: http://x264dev.multimedia.cx/archives/377 Some quotes: "But as noted in my previous post, merely being published by Google doesn’t guarantee that it is. Microsoft did similar a few years ago with the release of VC-1, which was claimed to be patent-free but within mere months after release, a whole bunch of companies claimed patents on it and soon enough a patent pool was formed." "VP8 is simply way too similar to H.264: a pithy, if slightly inaccurate, description of VP8 would be "H.264 Baseline Profile with a better entropy coder". Even VC-1 differed more from H.264 than VP8 does, and even VC-1 didn’t manage to escape the clutches of software patents." "Most importantly, Google has not released any justifications for why the various parts of VP8 do not violate patents, as Sun did with their OMS standard: such information would certainly cut down on speculation and make it more clear what their position actually is." And not about patents, but the quality of the spec: "The spec consists largely of C code…

Re: Third, nobody knows if anything is immune from unknown-by-definition patents, and MPEG-LA does not protect against them either: > Q: Are all AVC essential patents included? > A: No assurance is or can be made that the License includes every essential patent. http://www.mpegla.com/main/programs/AVC/Pages/FAQ.aspx However, as far as known patents go, VP8 is clean. I'm pretty sure Google put incredible effort in analysing the risk before making themselves troll target. The codec is very close to violating many patents, but with "strange omissions" (http://x264dev.multimedia.cx/archives/377) that make it non-infringing. IANAL, but that sounds like great strategy — there are unlikely to be two different 99% overlapping patents (that would survive re-examination and apply to VP8 only), and you can be sure you're not infringing the known one. If you look at list of supporters: http://www.webmproject.org/about/supporters/ That's patent troll dream. Everyone to sue, from small fish to test claims on to biggest pockets to drain. And yet, no trolls appeared.

There's something in all this that I suspect demonstrates a possible flaw in Google's traditional data-orientation, as opposed to human judgement. What the colour gradients (1Mcolors) example in the referred to article (http://englishhard.com/2010/10/01/real-world-analysis-of-googles-webp-… shows is that WebP, at the limits of quality, prefers to blur. The problem is that most real-world scenes are of objects that have texture, and an artificial texture is psychologically more pleasing than blurring away all these high-frequency small detail patterns. Dark Shikari’s article (http://x264dev.multimedia.cx/?p=541) shows this really clearly: in a real-world landscape, WebP loses badly to both jpg and x264 with too much blurring. Thing is, if you just do the "obvious" quality metric and do a subtraction of the encoded picture from the original, and sum up all the differences, that doesn't encode the psychological aspect. A blurred image might indeed do well here, because the blur colour will be somewhere in the middle between the highs and lows of the high-frequency pattern, so the error won't be too large. But a psychologically-oriented encoding may have encoded a pattern that resembles the original frequencies, but not precisely aligned, so the highs and lows may not coincide exactly between encoded image and original. It may thus simultaneously give a worse difference on a subtraction, but please the eye more.

I'm really excited about Daala, in ways that VP8 and VP9 never excited me—VP8 seemed like a damaged version of MPEG4. Although it looks like MPEG4 will be "good enough" for the web for a long time to come, it seems like HEVC development is weighed down by some perverse incentives. See this post from "Diary Of An x264 Developer" in 2010: http://x264dev.multimedia.cx/archives/360 Meanwhile, Opus came along and outperformed HE-AAC, which makes me think that more good things are going to come from open standards.

I don't know Dan's work/thinking well enough to anticipate what he might talk about. But if someone wanted me to speak on this prompt, I would talk about modern examples where optimizing compilers significantly underperform hand-written assembly: - LuaJIT's interpreter (here is Mike Pall's famous post giving a detailed explanation of why this is, incidentally prompted by a question I asked: http://lua-users.org/lists/lua-l/2011-02/msg00742.html) - x264, which has extensive acceleration implemented in assembly for a/v codecs (they even built their own assembly abstraction layer: http://x264dev.multimedia.cx/archives/191) - GMP, which has extensive acceleration implemented in assembly for arbitrary-precision arithmetic. I wouldn't go so far as to argue that any of this implies the death of optimizing compilers, so I will be very interested to see what Dan has to say.

The x265 encoder might not have Psy-RDO enabled --http://x264dev.multimedia.cx/archives/37 It penalizes "loss of detail" like that, biasing towards (inaccurate) noise over (accurate on average) absolute pixel values (PSNR).