John Regehr

Their PLDI paper, "Finding and Understanding Bugs in C Compilers", is an amazing read: http://www.cs.utah.edu/~regehr/papers/pldi11-preprint.pdf

Relevant for those thinking that "volatile" will save the day: http://www.cs.utah.edu/~regehr/papers/emsoft08-preprint.pdf It shows that a) volatile does not have to do _anything_ (if that is documented by the compiler) and b) you cannot trust your compiler to be bug-free. The latter is one reason to follow cert.org's advice to read the disassembly output of your compiler.

But, if you don't even try to prove part of a process correct, it's almost certainly wrong. See the excellent work that John Regehr's group is doing related to compiler and optimization verification/testing (http://www.cs.utah.edu/~regehr/research/ ), much of which has been talked about here. There are other folks who have successfully done formal verification of operating systems, device drivers, hardware design, etc. It's a big, hard problem, but we'll get to the "whole stack" eventually.

Also worth reading: the paper that found bugs in the proven-correct compiler: http://www.cs.utah.edu/~regehr/papers/pldi11-preprint.pdf (okay, so only a portion of the compiler is proven correct, and they didn't find any bugs in that portion). Note also that a compiler with a proof isn't necessarily correct - it just means that you've pushed all your bugs into your description of the rules of C and of the target architecture. Since neither C nor any common CPU architecture have a formal semantics, someone has to read the (english) description and convert it to a description in their theorem proving software, and there's no reason to believe this will be easy.

First of all: fantastic article. In-depth, insightful, and the examples are absolutely top-notch. On the razor's edge between accessible and profound. Hats off to the author. I will say that the problems seem to lie in a few interesting interlanguage quirks, and not so much on language specs. For example, LLVM and C have different definitions of "undefined behavior"[1] -- this is pointed out when looking at the `poison` identifier. In LLVM this might just be a weird side-effect we don't care about, but actually having an integer overflow is a big deal in C that introduces UB. Given the introduction, LLVM (incorrectly) introduces (C) UB at times. The second section, on pointer provenance, is much trickier, as there's no "obvious" UB introduced there. Further, I would argue that pointer provenance is a meta-language (or meta-computational) concept (which the Rust docs hint at), and logically speaking, might be intractable in the general case (I'd have to think about this more). Pointer arithmetic can fiddle with meta-state, whereas IRs like LLVM tend to focus purely on symbol manipulation. As a side note, I'd be curious what happens when LLVM sees something like: `void *p = &p` which is a self-referential pointer. What does it deduce about it? Does it optimize it away? Cool thought-provoking article. [1] https://www.cs.utah.edu/~regehr/llvm-ub.pdf

Do note I'm not being adversarial when I say I don't have to give examples since it uses a known-good technique. The status quo for compilers is they have plenty of bugs. The number of bugs go up like most software with the complexity (eg lines of code or paths). Highly-optimizing compilers should be expected to have more bugs than barely optimizing ones. Evidence on application of formal methods in industry showed the programs had way fewer bugs. Cleanroom and Praxis Correct by Construction w/ SPARK Ada regularly hit some of lowest defect rates in existence. http://infohost.nmt.edu/~al/cseet-paper.html http://www.anthonyhall.org/c_by_c_secure_system.pdf So, we have one trend saying complicated software will have lots of errors with another saying verified software will have fewer errors. The hypothesis we should have at that point is a verified compiler will have either few errors in general (optimism) or much fewer than non-formally-verified compilers (pessimism). The empirical evidence confirms this with bug trackers on most compilers but especially the Csmith testing pitting CompCert against other C compilers: http://www.cs.utah.edu/~regehr/papers/pldi11-preprint.pdf https://embed.cs.utah.edu/csmith/ It was the first time (IIRC) a program was custom designed to toss computer-years worth of input against compilers to find problems. Found hundreds in common compilers despite them being analyzed and used so much that many bugs were already found. When they applied it…

Also, this has been studied empirically, and CompCert holds up well: • https://www.cs.utah.edu/~regehr/papers/pldi11-preprint.pdf • https://compcert.org/motivations.html I think things may have improved further since the 2011 paper, where they found bugs in parts of CompCert which were not, at the time at least, formally verified. The formally verified parts of CompCert withstood torture testing admirably.

The trophy case is here: https://www.cs.utah.edu/~regehr/souper/ Example program 6: https://www.cs.utah.edu/~regehr/souper/output_6.html int a, b; void fn1() { b = a * ~1 == 1; } Here we multiply an int by -2. LLVM emitted the multiply and compare, but Souper proved that no int multiplied by -2 can equal 1, so this could be optimized to 0. By its nature Souper becomes less effective over time as its findings are incorporated into LLVM, so most of the trophies are old!

Yeah, there were some bugs in the frontend and I think one in the backend. But there were zero bugs in the optimization pipeline, which is where typically the most subtle bucks lurk. For more details, see https://www.cs.utah.edu/~regehr/papers/pldi11-preprint.pdf

http://www.cs.utah.edu/~regehr/papers/overflow12.pdf > For undefined behavior checking using precondition checks, slowdown relative to the baseline ranged from −0.5%–191%. In other words, from a tiny accidental speedup to a 3X increase in runtime.

When it comes to high-level stuff especially in C++, like the exact rules for template deduction, I think the specs are reasonably clear and do add value. But for the low-level memory model and undefined behavior, the specs are indeed extremely ambiguous. In fact, they're often outright 'wrong', in the sense that they have too little undefined behavior to support common compiler optimizations (e.g. [1] [2])... while making other things undefined for no benefit [3]. [1] https://fzn.fr/readings/c11comp.pdf [2] https://www.cs.utah.edu/~regehr/oopsla18.pdf [3] https://www.imperialviolet.org/2016/06/26/nonnull.html

I'm of the general opinion that what people think volatile does vs. what the compiler really does is quite different. Perhaps the situation has improved since this paper appeared. https://www.cs.utah.edu/~regehr/papers/emsoft08-preprint.pdf

At least a lot of it was due to having to specify around the weird quirks of existing compiler implementations. C has far more undefined behavior than it needs for optimization, and modern programming language theory and compiler research has shown that most undefined behavior isn't really necessary at all. [0] For example, safe Rust has extremely close (and sometimes superior) performance to C, and has almost* no undefined behavior at all. [1] [0] https://www.cs.utah.edu/~regehr/papers/undef-pldi17.pdf [1] https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/gc… * Some is inherited from LLVM, some is bugs, some is dumb corner cases like using kernel syscalls to overwrite the program's own memory with random bits.

Hm, good point about the bitfields. The paper I cite [1] actually talks specifically about bitfields as their precise semantics in the presence of "poison"-style uninitialized memory is not entirely clear yet. [1]: http://www.cs.utah.edu/~regehr/papers/undef-pldi17.pdf

>Without this it is incredibly difficult to do full verification. You can verify the optimizations on their own. Example: https://www.cs.utah.edu/~regehr/papers/pldi15.pdf

As others have said, yes, that's mostly the case. Even parsing C and elaborating it to a correctly typed AST is not quite as simple as others are making it out to be, though. Getting all the implicit type conversions correct is not completely trivial, and is a popular source of bugs (see some examples in http://www.cs.utah.edu/~regehr/papers/pldi11-preprint.pdf for instance). There are also some annoying ambiguities in the grammar and in name resolution rules that mean that sometimes it's not easy to tell whether something is supposed to be a type name or a variable name: https://jhjourdan.mketjh.fr/pdf/jourdan2017simple.pdf C is a "simple" language in many senses of the word, but it has a lot of complex details. It's fine to gloss over them when writing a compiler for a language very similar to C for learning purposes, but getting everything just right for actual C is tough.

If you look at the actual papers [1, 2] they don't mention security even once. I believe the security angle was just a PR spin added to the blog post. This is a tool for compiler writers to reason about the correctness of optimizations, and as such I believe it is useful to unlock more and more sophisticated optimization techniques. I agree that this has no impact on secure code. [1] https://www.cs.utah.edu/~regehr/papers/pldi15.pdf [2] https://www.microsoft.com/en-us/research/wp-content/uploads/2017/06/un…

Answering my own question, the paper [1] referenced by the post goes into some detail on this in section 9. [1] http://www.cs.utah.edu/~regehr/papers/undef-pldi17.pdf

Actually the bugs they found in CompCert were not in the proved sections. The CSmith paper says "As of early 2011, the under-development version of CompCert is the only compiler we have tested for which Csmith cannot find wrong-code errors. This is not for lack of trying: we have devoted about six CPU-years to the task. The apparent unbreakability of CompCert supports a strong argument that developing compiler optimizations within a proof framework, where safety checks are explicit and machine-checked, has tangible benefits for compiler users." https://www.cs.utah.edu/~regehr/papers/pldi11-preprint.pdf

Check out "Finding and Understanding Bugs in C Compilers": http://www.cs.utah.edu/~regehr/papers/pldi11-preprint.pdf. The paper explains what-is-bug/what-causes-it/what-is-impact for selected bugs. Complete list of 282 compiler bugs found (79 GCC, 203 LLVM) is also available online: http://embed.cs.utah.edu/csmith/.

> I'm not sure how C is specifically to blame for something like int overflow C is not a good choice when safety is paramount. "The low-level nature of C and C++ means that bit- and byte-level manipulation of objects is commonplace; the line between mathematical and bit-level operations can often be quite blurry. Wraparound behavior using unsigned integers is legal and well-defined, and there are code idioms that deliberately use it. On the other hand, C and C++ have undefined semantics for signed overflow and shift past bitwidth: operations that are perfectly well-defined in other languages such as Java." http://www.cs.utah.edu/~regehr/papers/overflow12.pdf

Although there seem to have been parts of it that hadn't been formally verified: "Using Csmith, we found previously unknown bugs in unproved parts of CompCert—bugs that cause this compiler to silently produce incorrect code." (http://www.cs.utah.edu/~regehr/papers/pldi11-preprint.pdf) The CompCert homepage tells us that the same authors did a study in the same year (2011) without finding any wrong-code errors. But even if they hadn't; I'd still be excited about CompCert demonstrating that this level of verification is indeed possible. Because then verifying the unverified parts would probably be just as feasible, given enough resources.

I'm the author of this piece. Everyone should go try out Alive instead of reading the blogpost: https://github.com/nunoplopes/alive Alive automates all of the stuff described in the post. Here's some info: http://blog.regehr.org/archives/1170 Also we gave a talk at the LLVM Dev meeting a few weeks ago, the talk doesn't seem to be online (yet?) but here are David's slides: http://www.cs.utah.edu/~regehr/devconf14.pdf

Here is one of the papers relating to the project: http://www.cs.utah.edu/~regehr/papers/pldi12-preprint.pdf

Here is one paper that describes it: http://www.cs.utah.edu/~regehr/papers/pldi11-preprint.pdf It is the CompCert compiler, which is formally verified. It looks as if the errors found were in the unverified portion. Those errors have been found and the verified surface has been expanded.

Interesting that there was only one compiler that was immune to a wide class of bugs they found everywhere else: "The striking thing about our CompCert results is that the middleend bugs we found in all other compilers are absent. As of early 2011, the under-development version of CompCert is the only compiler we have tested for which Csmith cannot find wrong-code errors. This is not for lack of trying: we have devoted about six CPU-years to the task. The apparent unbreakability of CompCert supports a strong argument that developing compiler optimizations within a proof framework, where safety checks are explicit and machine-checked, has tangible benefits for compiler users." http://www.cs.utah.edu/~regehr/papers/pldi11-preprint.pdf

Compilers that are verified/validated (in some sense): http://compcert.inria.fr/compcert-C.html http://www.cs.utah.edu/~regehr/papers/ http://lambda-the-ultimate.org/node/4571 ------------- The "Related Work" section in this UVa dissert probably most closely answere your question http://www.cs.virginia.edu/~jck/publications/Xiang.Yin.dissertation.pd…

I don't think this one is lacking interest, but an obvious fix. It looks like Russ Cox did commit a fix for some portion of the cases. But a better fix would either need to significantly modify the way things are laid out to reduce the odds of false positives, or else move to a precise GC. (Precise GCs are possible in C-like languages, but trickier to implement. Here's a recent paper on one: http://www.cs.utah.edu/~regehr/papers/ismm15-rafkind.pdf)

I wonder what he's doing that csmith[1] doesn't do. I quickly skimmed the paper[2], but nothing jumped out at me. In fact it looks like some of the reducers are implemented as plugins on top of csmith. I guess I'll have to read the whole thing later. 1. http://embed.cs.utah.edu/csmith/ 2. http://www.cs.utah.edu/~regehr/papers/pldi12-preprint.pdf

Yes, this is funny. The existence of random paper generators for a discipline doesn't say much about the rigor of that discipline. You might be able to make some kind of argument based on how long it takes for someone trained in the discipline to detect it as randomly generated. Of course, you could imagine building a program that creates perfectly compileable, yet randomly generated programs. These guys actually did it to test gcc's handling of the volatile keyword: http://www.cs.utah.edu/~regehr/papers/emsoft08-preprint.pdf