Pernosco Blog

Totally shameless plug: if you like rr's technical capabilities but not it's UI, you might love Pernosco (https://pernos.co/)

I am not aware of any zero-payment solutions. These tools cost money to invent and develop so most of them are commercial. By GHS, developers of TimeMachine (TimeMachine created in ~2003), for C/C++: https://www.ghs.com/products/MULTI_IDE.html see “History viewer” By the developers of rr (rr created in ~2013) for C/C++: https://pernos.co/ By former GHS employees for Python: https://functiontrace.com/ Usable FOSS time travel debugging is new enough that no follow on FOSS visualization has been developed as far as I know.

Someone needs to design a programming language and an IDE (and possibly a new OS too) with great debugging as the primary goal. Debugger and IDE support is always just thrown on later in every new language these days. "Omniscient debugging" as seen in https://pernos.co/ is the holy grail. Time-travel debugging would also be great. People are far too obsessed with static type checking. A lot of this time would be better spent invested in live debugging tools. When I'm editing my code I want to see exactly what _value_ each variable contains, the type really doesn't matter so much. Wallaby.js/Console.ninja is a great example of this. Good debugging, especially deterministic record/replay is usually complicated by the OS. I often wonder what an OS would look like if designed with debugging as a top priority.

https://pernos.co does something similar, except that instead of searching for watchpoints/breakpoints we're building a database of everything that happened. That involves replaying different segments of the execution in parallel and also applying different kinds of instrumentation to the same segments, also run in parallel. It's great, we can saturate the biggest boxes AWS has :-).

For execution, a stack is a continuation. For debugging, we pretend like it's a historical record, and mostly get away with it. Various things break the correspondence slightly. TCO breaks it a lot more. Debugging is important. It doesn't get enough respect. Stacks are a pretty critical component of debugging, for better or worse. It would be great if we didn't depend on this fiction quite so much. With native code, there are definitely alternative options now, such as rr[1] and Pernosco[2] where if you want to look back in time—well, you just go back in time. For JavaScript, that's becoming more and more possible with things like Replay[3]. Perhaps before long, the debugging argument will just go away. [1] https://rr-project.org/ [2] https://pernos.co/ [3] https://www.replay.io/

(I have already replied to another comment with the same suggestion). Pernosco offers the best of both worlds ( debugger, print), along with a few magical features. https://www.pernos.co With it you can print anything present in your recording and step, and do anything you'd do in a regular debugging.

I literally (as in 30 minutes ago) just fixed a bug in 10 minutes that I thought was going to take me 3 days to fix using https://pernos.co/ (a "timeless" debugging tool that wraps Mozilla's `rr`). This is a factor of speedup on the order of 100x+, not 10x. I highly recommend checking this tool out if you value your time.

A third alternative is to use a better debugger. I recommend Pernosco (https://pernos.co/). Pernsoco lets you record your program once, and then builds a database out of the recording. You can quickly get a list of all the times your program hit a certain line, and you can then print local variables from each of those execution points. To make this more concrete, you can click on line number 442 to see a list of all the times your program was at line number 442. If you then type the expression "foo" into the query window, the list will update to show the value of the variable foo at every one of those occurrences. There's no need to edit the printfs, recompile, and rerun the program.

"Time travel debugging" seems to be taking off as the term - "reverse debugger" was around for a long time but doesn't seem to have taken root. It has fixed the conversation of "We make a reversible debugger", "Ah so you put the bugs in", though! > rr is amazing but it doesn't work well in cloud environments where it needs you to configure settings in the virtual machine (which you can't). So this limits it to desktop Linux machines. Hopefully we'll see cloud environments find a way to expose more of the hardware performance counter stuff by default. The alternative is to use an instrumentation-based system that doesn't need access to performance counters. That said, as with machine-level virtualisation, it'd be nice to see the CPU vendors provide more explicit hardware support over time to make implementation easier / higher performance than current approaches. > Valuable, but not as valuable as some other options. Plus, using GDB is. Ugh. Looks like there's some support / guidance for using an IDE interface to rr's GDB - https://github.com/rr-debugger/rr/wiki/Using-rr-in-an-IDE There's also Pernosco https://pernos.co/ which is a cloud service (hosted versions possible) from the original developers of rr. It post-processes uploaded rr traces into a database. This post-processing is compute-intensive but happens on their beefy worker machines. Once it's done, they can retrieve program state across time almost instantly - their website is worth checking out for…

Author here: I honestly expected it to be much worse. The tracing integration in particular is fantastic - being able to just slap a `#[instrument(...)]` attribute on any function and then be able to filter which spans you want to see is kind of a superpower. That said I think much could be improved, still. I only breezed through the perf/nperf stuff because I've used them before, and through the self-profiling stuff because rustc devs helped me with it. I would've killed for a REPL while I was working on this (or a server architecture so I could write my _own_ rustc queries), and step through them, etc. I like Kate's approach to this[1] - I'd just like something a little more... interactive. What do _you_ think it should look like? I feel like a ton of good ideas come from folks who "simply didn't know it was impossible" (more accurately: haven't been trained to accept to work with subpar tools). I'm excited to try out Pernosco[2] for example, it seems like a much-needed rethink of debuggers. [1]: https://twitter.com/thingskatedid/status/1386077675211526148 [2]: https://pernos.co/

I haven't personally used Replay, but from my experience using rr (a native debugger that also provides time-traveling features) being able to replay execution both backwards and forwards in time on a whim is amazing. These tools excel at diagnosing bugs that are hard to reproduce, because you only have to reproduce the bug once under the debugger and then you can endlessly replay that execution until you figure it out. As Jason said above, you can retroactively add print statements in places that would be useful, without having to waste time trying to reproduce the bug again! roc (the original author of rr) founded a company to build an even more compelling product on top of rr called Pernosco. They have some mind-blowing demos I'd recommend you check out: https://pernos.co/ . Being able to easily answer questions like "where did the value in this variable come from, and when did it get set?" makes debugging a wildly different experience.

See also https://pernos.co/ which is based on rr but adds a queryable database of the whole program execution, which allows you to do things like this: > [...] just click on the incorrect value. With full program history Pernosco can immediately explain where this value came from. The value is tracked backwards through events such as memcpys or moves in and out of registers until we reach a point where the value "originated" from, and each step in this process is displayed in the "Dataflow" panel that opens automatically. There is no need to read or understand the code and think about what might have happened, we can simply ask the debugger what did happen. https://pernos.co/examples/race-condition

To me this is very reminiscent of time travel debugging tools like the one used for Firefox’s C++ code, rr / Pernosco: https://pernos.co/

Almost as good as https://pernos.co (a commercial product built on top of rr produced by the developers of rr)

Time-travel decision like rr and Pernosco. I’ve heard firsthand stories from developers that found and fixed rare race condition bugs in minutes instead of hours or days. https://pernos.co/

This is why I am so hopeful for Pernosco, a better debugger with better interface. https://pernos.co/

The rr debugger [1]. I desperately missed it after switching to a mac laptop -- to the point that I now have an older linux system which exists primarily to run rr. (I've heard that Pernosco [2] - partly built on rr, AFAIK - is even more revolutionary, but haven't yet tried it myself) [1] https://rr-project.org/ [2] https://pernos.co/

Having recently had the pleasure of having to debug JIT-compiled code with an ABI mismatch, I can't overstate how useful `rr` (https://github.com/mozilla/rr) can be to debug assembly. The ability to `rsi` e.g. reverse step instruction is very powerful. One tool that I started exploring is https://pernos.co/, the ability to do dataflow analysis is super cool. Let's you easily answer the question "How did this value get into this register".

https://pernos.co/ is making a very similar omniscient debugger based on rr for AOT compiled languages with DWARF info.

There's a great series of posts going into more detail about Pernosco's design and advantages that is linked to in the final paragraph: https://pernos.co/about/

FYI none of the images on https://pernos.co/about are working for me. > static.pernos.co has a security policy called HTTP Strict Transport Security (HSTS), which means that Firefox can only connect to it securely. You can’t add an exception to visit this site. > The issue is most likely with the website, and there is nothing you can do to resolve it.