John Regehr

Ah, Csmith! A well known and much read paper which inspired a lot of work. Differential testing of compilers via random generation of correct programs goes back a ways before this paper. For C, it's made more difficult by the need to avoid undefined behaviors in the generated code (which C is loaded with). So much of the work here was how to generate sufficiently diverse programs that still have defined behavior. A follow on paper (from 2012) described creduce, the failing test input reducer than can be used with csmith. It's also useful for other purposes, and for languages other that C. Reducing the inputs requires preserving the condition that they are valid C without undefined behaviors. https://users.cs.utah.edu/~regehr/papers/pldi12-preprint.pdf

Well, low skill floor, sure, but there's a very high skill ceiling. John Regehr and his students have done impressively deep work in finding compiler bugs. • https://blog.regehr.org/archives/category/compilers • https://blog.regehr.org/ (some of his compiler-related posts aren't properly tagged) • [PDF] https://users.cs.utah.edu/~regehr/papers/pldi11-preprint.pdf Finding and Understanding Bugs in C Compilers (edit I see someone else already posted this paper) • A (very) little discussion: https://news.ycombinator.com/item?id=7728035

There's already tooling available for using SMT to validate LLVM-IR transformations [1]. It's designed for zero false positives however, so some things might slip through the cracks. Additionally, this work focuses on phase ordering, which produces correct code regardless of what the LLM puts out, assuming there aren't any bugs in the passes being used (which could crop up as random orderings aren't as well tested as the standard orderings in the commonly used pipelines). 1. https://users.cs.utah.edu/~regehr/alive2-pldi21.pdf

If I understand correctly, the authors feared that C-Reduce could introduce undefined behaviors: > Unfortunately, if given the opportunity by a lax predicate, C-Reduce will write programs that have undefined behavior I don't think this is true, in the introduction of [1]: > Our reducers do this automatically and without introducing undefined behavior into the reduced test cases [1] https://users.cs.utah.edu/~regehr/papers/pldi12-preprint.pdf

The article touches on basic testing strategies that apply to general software as well as compilers. I have been involved with compiler testing projects on production LLVM and GCC in my past life. One thing that makes compiler testing specifically more difficult than general software is the Oracle Problem: how do you verify the output is in fact correct? Crashes are relatively easy to find by random fuzzing, but in the general case, proving that the output program is not miscompiled is non-trivial. There are a couple effective techniques in the literature that might be useful here: - Differential testing[1]: generate a bunch of random, correct, deterministic programs; run them under different compilers or under different compilation flags and check if the output of the program is identical - Equivalence Modulo Inputs[2]: a class of techniques that can be used transform a program to a distinct program that is supposed to be equivalent to the original for a specific input. (shameless plug) [1]: https://users.cs.utah.edu/~regehr/papers/pldi11-preprint.pdf [2]: https://web.cs.ucdavis.edu/~su/publications/emi.pdf

This is actually an incredibly deep and difficult question to answer. I would expect no less from cperciva. :-) As I mention in the talk, you get very bad tactical performance from taking a uniform random distribution and piping it into the emulator. The fuzzer is exponentially unlikely to hold the jump button for many successive frames without a break. In the fully general case, I think instead of maximum entropy, you want something more like Marcus Hutter's AIXI where you spend energy on inputs inversely proportional to their Kolmogorov complexity. Unfortunately, that's uncomputable, but it turns out that just switching to toggling bits with low probability does a lot better than pure randomness. The approach is analogous to swarm testing (https://users.cs.utah.edu/~regehr/papers/swarm12.pdf). All of which is to say, the result that we show here is vastly less jumpy than our first tries. The reason it's still more jumpy than a human player is that our platform has no idea where it is in the game, or even that it's playing a game. So if a jump doesn't harm it in the exploration process, there's some chance the first input getting somewhere new will involve a jump, and that will then get locked in. We do have the capability to do optimization on inputs (what conventional PBT calls "shrinking"), and indeed if you apply this to Mario you can get it to jump a lot less and complete levels a lot faster. That capability didn't exist yet when this video was recorded. We should…

You're making a very strong statement, so we might as well look at extreme cases, namely the formally verified middle-end of CompCert: https://users.cs.utah.edu/~regehr/papers/pldi11-preprint.pdf I believe this would count as empirical evidence of very strong static typing reducing bugs. I think this points to what many other comments have been getting at, which is that typing is not a binary yes/no question, but a large, multi-axis (static/dynamic, strong/weak) spectrum with lots difference between type systems, as well as big differences in how people apply those type systems to their problem. You could still work in a static, strongly typed language and represent everything as strings, converting back and forth as necessary, but that's essentially working in a dynamically typed language. You could also take advantage of the tools the type system gives you in order to reap the benefits, creating classes representing the legal values and asserting important invariants.