R
Robert O'Callahan
robert.ocallahan.org · Personal blog · on HN since 2011
Robert O'Callahan, a longtime Mozilla engineer and creator of rr, writes about browsers, debugging tools, and C++ and Rust safety.
What HN saysCommenters cite his posts on Firefox's history, his antivirus advice, and his Rust versus C++ optimization writing.
149HN people shared it
68linked it in comments
148stories submitted
11Kstory points
12 months agolast shared
2011people per year2026
Most-shared posts
- 1Introducing rr Chaos Mode12 people · 8 points · first shared 2016 · discussion on HN
- 2Choose Firefox Now, or Later You Won't Get a Choice (2014)5 people · 873 points · first shared 2014 · 329 comments on HN
- 3Avoid Non-Microsoft Antivirus Software5 people · 859 points · first shared 2017 · 374 comments on HN
- 4I cannot consistently write safe C/C++ code5 people · 195 points · first shared 2017 · 118 comments on HN
- 5Browser-Wars History: MD5-Hashed Posts Declassified5 people · 163 points · first shared 2018 · 71 comments on HN
- 6Maintaining an Independent Browser Is Expensive4 people · 441 points · first shared 2017 · 300 comments on HN
- 7Leaving Mozilla4 people · 262 points · first shared 2016 · 103 comments on HN
- 8So you want to build a browser engine4 people · 225 points · first shared 2024 · 129 comments on HN
- 9“Safe C++ Subset” Is Vapourware4 people · 159 points · first shared 2016 · 116 comments on HN
- 10Forking HTML into a static language doesn't make sense4 people · 83 points · first shared 2020 · 90 comments on HN
- 11Pernosco Demo4 people · 15 points · first shared 2019 · discussion on HN
- 12Rust Optimizations That C++ Can't Do (Version 2)4 people · 5 points · first shared 2017 · 1 comments on HN
- 13WebAssembly: Mozilla Won3 people · 628 points · first shared 2017 · 361 comments on HN
- 14The Fight for Patent-Unencumbered Media Codecs Is Nearly Won3 people · 507 points · first shared 2018 · 170 comments on HN
- 15Mozilla's low-overhead open source replay debugger3 people · 272 points · first shared 2014 · 43 comments on HN
What HN people wrote when they linked it
sfink · 2025-02-03
Yes. That is what "...unless the base rate is set externally somehow" means.
It is different initially, when only one company is offering the "discount" and they have not yet adjusted their base price upward. In fact, the people who want the discount will presumably flock to their service, which may even mean they won't raise the base price all the way up if it makes their costs lower. But if that works, the other companies will follow suit.
In short: there's a period of time when there's a difference, and you have a real choice. If the difference is real, it will get locked in to the entire industry. It's a positive economic profit, and those go away.
cf https://robert.ocallahan.org/2014/08/choose-firefox-now-or-later-you-w…
cxr · 2020-07-11
Funny, before your comment showed up, I was actually going to edit in an addendum to link to
https://robert.ocallahan.org/2018/10/the-costs-of-programming-language…
... to pair with the mbeddr link.
You seem to be at a place where you think of Rust as already entrenched, but I'm at a place where I still see Rust as a proposition—one that could lead to that place, or not if we think better and nix it (following the reasoning above).
jcranmer · 2019-08-29
I'll add to this:
In 2007 (!) Robert O'Callahan, one of the major developers of Gecko, was advocating internally to drop Gecko and replace it with WebKit in Firefox. Instead of exposing a big rift publicly, he posted hashes of his thoughts. Last year, he finally revealed what he had written and accompanied the reveal with some commentary that he now believes that to follow that path would have been a mistake: https://robert.ocallahan.org/2018/01/ancient-browser-wars-history-md5-…
ken · 2018-12-01
Previously, from a former Mozilla developer [1]:
> At best, there is negligible evidence that major non-MS AV products give a net improvement in security. More likely, they hurt security significantly; for example, see bugs in AV products listed in Google's Project Zero. These bugs indicate that not only do these products open many attack vectors, but in general their developers do not follow standard security practices. (Microsoft, on the other hand, is generally competent.)
In the linked Project Zero issue tracker, all 3 of these "reputable sources" have exploits in their anti-virus software.
[1]: https://robert.ocallahan.org/2017/01/disable-your-antivirus-software-e…
demarq · 2017-04-05
> I find it funny and think it's pretty telling that there are bold claims made that are only crossed out after the author is corrected
crossed out? have you seen the follow up post... http://robert.ocallahan.org/2017/04/rust-optimizations-that-c-cant-do_…
The error was in his example not his assertion.
ferongr · 2013-03-08
I'm not really sure I could tell them apart in an ABX test. But it's not like the game is demanding in any way. The amount of sprites is way too low and they're all bitmaps. Furthermore, you don't need the webaudio api to playback an audio track and few effects concurrently, like this game does. Standard <audio> elements with the preload attribute work fine [1].
What I'd like to see is a comparison between 2D vector graphic performance of Flash compared to Web technologies as implemented in browsers. In my experience browsers are still very slow at that (without WebGL) while Flash had excellent performance for as long as I remember.
[1]http://robert.ocallahan.org/2011/11/latency-of-html5-sounds.html
doublec · 2013-01-20
As far as I know you are correct. Mozilla released the Audio Data API and produced a number of demos based on that. A W3C group was formed to work on Web Audio standardisation and in that group Google announced the work on their API.
Mozilla then followed with a "Media Processing" API: http://robert.ocallahan.org/2011/06/media-processing_15.html
It's possible that person you are responding to is thinking of that.
capitainenemo · 2024-02-20
Huh. I thought PNaCL had cross-platform endian issues from what I'd read at the time.
However: https://www.tomshardware.com/news/chrome-deprecates-pnacl-embraces-web…
Just mentions API concerns and: https://robert.ocallahan.org/2013/05/blink-pnacl-and-standards.html?ga… " PNaCl and Pepper are not open standards, and there are not even any proposals on the table to standardize them in any forum. They have documentation, but for the details one must defer to the large bundle of Chrome code that implements them. "
and
"1. NaCl is CPU-specific. That's not suitable for the web.
2. PNaCl is in theory portable, so it is potentially worth considering.
3. However there is no spec for PNaCl, and standardizing it would take an extreme amount of effort: It means standardizing a subset of LLVM IR, which is quite rich and complex, as well as the Pepper plugin API. This is an enormous amount of work that has not even been begun by Google AFAIK."
But you're right, I don't see anyone mentioning any hardware complications.
Oh well. Moot point these days, and I think we're all better off that everyone else stood their ground, in terms of security, integration with JS APIs, and having a variety of WASM implementations.
db48x · 2021-04-27
That’s an excellent idea. Debuggers certainly need better documentation, and guided tours are a good way to teach skills.
Some of the Mozilla folks did livestreams of their development work on Firefox, including debugging sessions. I haven’t watched any lately, but you might search them out. If they kept it up then they might be using RR now, instead of just GDB.
Here’s a demo video for Pernosco: https://robert.ocallahan.org/2019/10/pernosco-demo-video.html
Pernosco is based on RR, but this probably won’t teach you anything about using RR, since the UI is so different. But it was a memorable one because Brendan Gregg produced a really good tutorial for GDB, where he debugged a real problem in a real program (instead of a made–up problem in a program written just to have a made–up problem inserted into it). The Pernosco demo debugs the same problem in the same program, so comparing the two can be quite easy. I would like to see someone the same type of tutorial for RR, and for other debuggers that people like.
BrendanEich · 2019-10-19
"Logic" suggests Bayesian analysis but you're working with too few bits. When you make an absolute-sounding posterior claim based on thin or non-existent priors, you risk losing sound inference in favor of pushing a preferred conclusion.
I'm disputing your absolute-sounding "Mozilla’s employees are fiercely liberal". Even if you define "liberal" broadly, are they all? Really, no moderates or righties? As I happen to know some, and so do other people, you seem to be using hyperbole at best, if not making things up.
Also your absolute, unhedged "If you have a conservative bone in your body, you are not welcome on the Mozilla campus." Again, counterexamples are known personally to me and others on HN such as @roca. So instead of persuading, you're exaggerating for some reason (to divide and conquer, to get sympathy, it doesn't matter -- the exaggeration is the point and it blows back, assuming you are actually trying to persuade anyone; perhaps you are writing not to persuade?).
Mozilla would have trouble with state law if they actively fired or demoted anyone over past political acts, per California Labor Law 1101 and 1102. Let's agree that is not the case.
That leaves subtler biases. You could be right, especially after I left, that anyone who is known to be "on the right" (conservative is almost useless as a clear political term) would face subtler kinds of hostility, skepticism, pressure. You are probably right about executives, but that's true at a lot of places.…
moyix · 2017-06-15
I think many commenters here are viewing this only in terms of die cost. But the blog post is not about that: it's about the other costs that this complexity introduces! Each new feature interacts with all the others in complex ways, and each new feature requires significant new engineering efforts by anyone who has to interact directly with x86 assembly (I sketched a few of these use cases below [1]).
The same author has also written [2] about Intel's CET (their attempt to add control flow integrity (CFI) support at the CPU level):
> The tail end of Intel's document is rather terrifying; it tries to enumerate the interactions of their CFI feature with all the various execution modes that Intel currently supports, and leaves me with the impression that they're generally heading over the complexity event horizon.
[1] https://news.ycombinator.com/item?id=14560267
[2] http://robert.ocallahan.org/2016/06/is-control-flow-integrity-worth.ht…
bzbarsky · 2015-11-30
Yes, this is why people are talking about having "installable" web apps. It sounds like you clearly agree that some sort of explicit user opt-in is needed to get out of the sandbox, which is good; I've seen some people argue that this is too high a bar.
Now it might happen that this will effectively happen as a result of convergence and standardization of browser extension APIs. It'll be interesting to see what happens there.
That said, the manifest of permissions idea is fundamentally broken in at least two related ways:
1) Most users have no rational basis on which to decide whether to grant the permission or not. In fact, I would argue this is all users, due to issue #2:
2) The permissions are either overbroad, such that pretty much any app that wants to do anything interesting asks for them (this is the "all my apps on want to be able to read my contacts and browsing history" syndrome) or there are so many of them that the list becomes way longer than any user would ever read at install time.
You example of a permission for opening arbitrary TCP sockets is an excellent case in point. As a standalone permission, what fraction of users do you think would understand what is actually being granted? Even if you restrict to computer professionals, I suspect that most would not understand the implications without a somewhat lengthy explanation of what the permission actually allows. And even then, in practice it would be rolled into some other more broad permission…
TimorousBestie · 2025-09-29
> The article isn't claiming there is a "one true modern subset" of C++ that they use.
They’re arguing that they’ve found a sufficiently safe subset, which if true would be the first.
It would be a waste of time to relitigate the many programmer-hostile, unsafe decisions the committee has made over the years. I think your programmers who are supposed to “understand and have control over” C++’s arsenal of footguns are more or less fictional and a language designer shouldn’t take them as intended audience.
https://robert.ocallahan.org/2017/07/confession-of-cc-programmer.html (2017)
> Also, I believe the C++ standards committee does care about this, which is why Profiles [1] are being considered.
It was discussed at length why this proposal is insufficient: https://news.ycombinator.com/item?id=45234460
roca · 2025-06-25
You are correct. I worked on this for years at Mozilla. See https://robert.ocallahan.org/2007/02/units-patch-landed_07.html and https://robert.ocallahan.org/2014/11/relax-scaling-user-interfaces-by-… for example. Some of the problems were pretty hard but the Web ended up in a pretty good place --- Web developers pretty much don't think about whether scaling factors are fractional or not, and things just work... well enough that some people don't even know the Web is "free-scaling UI"!
aoli-al · 2025-02-22
AFAIK, Antithesis uses a hypervisor to achieve deterministic execution. This can be less effective because the hypervisor does not have language semantics and faces a larger search space. You may check Figures 5 and 6 in our technical report[1], where we compare Fray against RR, a record and replay tool that can also be used for concurrency testing at OS level[2].
[1]: https://arxiv.org/pdf/2501.12618
[2]: https://robert.ocallahan.org/2016/02/introducing-rr-chaos-mode.html
abhinavk · 2024-08-11
https://robert.ocallahan.org/2024/06/browser-engine.html
P.S.: This specific post was written in response to announcement of Ladybird.
khuey · 2024-07-21
From the perspective of an rr maintainer, Sid's work was good and we were supportive of it. The main issues with migrating to that as the "blessed" version are that 1) rr has accumulated a decade of very hairy fixes for crazy kernel/process behavior that we feared could be lost during a port and 2) there's a closed source project (remix[0]) built on top of rr that would have needed to be ported too.
[0] https://robert.ocallahan.org/2020/12/rr-remix-efficient-replay-only-bi…
Veserv · 2024-07-21
rr was introduced in 2014 [1].
gdb reverse debugging was introduced in 2009 [2].
You can see a fairly comprehensive history of time travel debugging here [3].
Not to say the built-in gdb reverse debugging was any good. It had (has?) like 1,000,000% overhead which is basically unusable. At least some implementations in the history that were introduced earlier only had ~1,000% overhead or less in general. Yes, a literal 1,000x overhead difference.
[1] https://robert.ocallahan.org/2014/03/introducing-rr.html
[2] https://www.sourceware.org/gdb/wiki/ReverseDebug
[3] https://jakob.engbloms.se/archives/1564
pm215 · 2024-07-21
This is true, but for at least some race conditions the rr "chaos mode" can help in tracking them down. Chaos mode basically randomly adjusts its thread scheduling so that it sometimes doesn't run a thread at all for a few seconds, so you run it a bunch of times until it hits the timing window for the race condition; and then when it does you debug the recording. IME it works at least sometimes but it does require that the thing you're testing doesn't take too long to fall over, since you need to do multiple recordings of it.
https://robert.ocallahan.org/2016/02/introducing-rr-chaos-mode.html
Lockal · 2024-06-24
"animated fonts" - not really; all meaningful applications not only calculate shaping once, they also aggressively cache the result (mentioned in https://robert.ocallahan.org/2024/06/browser-engine.html)
But things like this might be possible (for now): https://gwern.net/dropcap
oskarkk · 2024-06-21
One commenter who worked on Firefox made a nice blog post in response, about what it takes to make a web browser: https://robert.ocallahan.org/2024/06/browser-engine.html
khangaroo · 2024-06-03
The SSE float reciprocal instructions have slightly different results between Intel and AMD, which can be a source of headaches for those expecting deterministic results between PCs. (see https://robert.ocallahan.org/2021/09/rr-trace-portability-diverging-be…
soraminazuki · 2024-04-24
Antivirus being not only inherently ineffective, but actively undermining the security of the system through negligence and poor practices. Often tampering with other software and negating actual security hardening measures along the way. Has anything changed this past decade or two?
https://ia801200.us.archive.org/1/items/SyScanArchiveInfocon/SyScan%20…
https://robert.ocallahan.org/2017/01/disable-your-antivirus-software-e…
sanxiyn · 2023-06-27
I am somewhat pessimistic because LLVM is significantly behind GCC in debug information quality (for example, see https://robert.ocallahan.org/2018/11/comparing-quality-of-debug-inform… and there is limit to what Rust can do about it short of fixing LLVM.
rrnewton · 2022-11-22
Yes indeed.
That concurrency testing capability is a pretty well-studied area and we implement a couple existing algorithms. The first is our adaptation of the PCT algorithm (ASPLOS'10 https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/as… That's what you get by default with `--chaos`.
But we also have variations on straight up randomized scheduler (random thread selection at each time step).
rr chaos mode has its own take on this: https://robert.ocallahan.org/2016/02/introducing-rr-chaos-mode.html
This study compares a few approaches - http://www.doc.ic.ac.uk/~afd/homepages/papers/pdfs/2016/TOPC.pdf.
nyanpasu64 · 2021-12-03
https://github.com/tokio-rs/loom perhaps? It also models weak memory reordering, but takes some work to integrate into existing apps.
For triggering race conditions in compiled binaries, you could try https://robert.ocallahan.org/2016/02/introducing-rr-chaos-mode.html.
arkj · 2021-07-17
Adding a few more that helped me to learn few things,
Evan Klitzke - https://eklitzke.org
Eli Bendersky - https://eli.thegreenplace.net
Bartosz Milewski - https://bartoszmilewski.com
Jeff Preshing - https://preshing.com
Robert O'Callahan - https://robert.ocallahan.org
Marek's idea of the day - https://idea.popcount.org
Chris Wellons - https://nullprogram.com
Andrej Karpathy - https://karpathy.github.io/
gaul · 2021-03-29
Robert O'Callahan hashed some thoughts about Firefox that he revealed many years later:
https://robert.ocallahan.org/2018/01/ancient-browser-wars-history-md5-…
hnlauncher · 2020-10-02
By guaranteed optimisations talk, are you referring to this blog post by Robert O'Callahan: https://robert.ocallahan.org/2020/08/what-is-minimal-set-of-optimizati…
the_why_of_y · 2020-05-24
I think you'd have to do the same thing that you'd do in non-C++ languages: write an abstraction layer that encapsulates the low-level OS access (or hardware access, in embedded software) and whose implementation is unsafe, but whose interface is "safe", whatever that means.
Once you have that to build on top of, such a compiler flag could make sense... if it were possible in C++, which I'm not sure about.
See for example this criticism of one such effort: https://robert.ocallahan.org/2016/06/safe-c-subset-is-vapourware.html