Reproducible Builds

Also the wider reproducible builds website/community https://reproducible-builds.org/ Also live-bootstrap, stage0, mrustc, mes, and so many amazing projects whose combined efforts all helped finally make probably trustworthy toolchains a thing.

The Reproducible Builds project is working on trustworthy builds: https://reproducible-builds.org/

Historically, major causes for non-determinism was embedded timestamps and unsorted file listings created by the build tools. I have not followed the progress recently, but https://reproducible-builds.org/ is a starting point if you are interested. There is a sane path forward for reproducibility on bare metal, no custom emulation is needed.

> most compilers of today don’t actually produce exactly the same code if you run them twice. By default, but https://reproducible-builds.org/ has made excellent headway on allowing that to be fixed if you care.

> The maintainers can be compromised though. Is every single version of every single "vetted" package / maintainer also vetted? Pretty much, packaging is not a brainless process. One of the effort that specifically target this is the Reproducible builds project [0], along with many other security measures set by each distro. There are also usually multiple testing and updates rolling stages. The best evidence of how effective these measures is its actual reputation and record on the ground. [0] https://reproducible-builds.org/

The term 'reproducible' in 'reproducible build' (https://reproducible-builds.org/) conveys "bit-for-bit identical output". -- The benefit of that is security; others can compile the same sources and check that they get the same binaries. But more generally, 'reproduce' just means 'create the same thing again'. Nix uses "reproducible" in a more general sense: you provide the same inputs, and you'll get a program which behaves the same way, regardless of whether you compiled from source or downloaded binaries from a cache. IMO, I think "reproducible build" is enough to unambiguously refer to the former. But, if you can think of a nicer word for "you get the same behavior from the same inputs" than "reproducible", it would be worth suggesting.

I love Ansible, but I'm waiting for the day that reproducible builds are part of most distributions. Nix is paving the way, and it seems like the rest are following. Then maybe the need for these configuration management tools will lessen. Look here for more info: https://reproducible-builds.org/ One of the useful parts of Ansible was its windows modules. I know that Microsoft has made DSC (Desired State Configuration) to handle config management in a declarative way, but haven't tried it out.

Reproducible build [0] is the modern solution to this problem, it generates the same binary output on everyone's computer by carefully controlling the compiler version and input data to the build system, thus allowing users to independently verify that an official binary is a faithful build from its source code. Although it's not a silver bullet (compiler bootstrapping is still vulnerable), but still greatly increases the level of confidence. Signal adopted reproducible build since 2016 [1]. [0] https://reproducible-builds.org/ [1] https://signal.org/blog/reproducible-android/

So happy someone are spending time on this issue, it's like a breath of fresh air and intelligence in the midst of all the usual software (security/privacy/etc. take your pick) mayhem. It's worth reading https://reproducible-builds.org/ for a brief (re-)reminder on why this project is important. Outtake: "With reproducible builds, multiple parties can redo this process independently and ensure they all get exactly the same result. We can thus gain confidence that a distributed binary code is indeed coming from a given source code."

I appreciate that predictability is a big priority with Cargo. I feel like Rust and Cargo are in a great position to deterministically (per compiler version, per set of build flags) build libraries. That would be an amazing step forward in security if it could be enforced. Does anyone know if this is planned? https://reproducible-builds.org

>If WhatsApp is so evil that they've backdoored their product, it is ... stupid to leave that backdoor in their source code. Reproducible builds[1] are the solution to that problem. Everyone doesn't have to build the source if a) building the source produces the exact same binaries every time, and b) source can be built by a trustworthy party to verify that it matches the build being distributed. This way, you only need one good build cop to warn the rest of the population about bad binaries. Signal for Android supports reproducible builds[2] so it seems entirely possible an open source WhatsApp client could as well. [1]https://reproducible-builds.org/ [2]https://github.com/WhisperSystems/Signal-Android/wiki/Reproducible-Bui…

Congrats on the launch. There is both a large need for improvements in npm supply chain security and a market willing to pay for them. Concerns: 1) npm Open-Source Terms condition four[1] states. You may access and use data about the security of Packages, such as vulnerability reports, audit status reports, and supplementary security documentation, only for your own personal or internal business purposes. You may not provide others access to, copies of, or use of npm data about the security of Packages, directly or as part of other products or services. This statement seems vague enough to potentially include your use case. It also seems to include what snyk, jfrog xray, sonatype, and white-source do, so maybe this is not an issue. 2) It appears that this will be an open-core business. What capabilities are you willing to provide in the free/community edition and under which licenses? 3) The website doesn't show pricing. Can you provide details on this? Questions: 1) What are your thoughts on using reproducible builds[2] plus Diverse Double-Compiling (DDC)[3] on the dependency graph to ensure build artifacts originate from known git repositories? Disclosure, I've been working on this for a few months now. 2) Where do you run your analysis? AWS and DigitalOcean have terms that prevent running high risk code. 3) Do you have examples of previous attacks and how your tooling would handle them? Best of luck. [1]…

Great article. I see this trend too: the systems powering the modern world are increasingly featureful, complicated, centralised into a few hands, and this will likely continue. I'm a keen developer and user in this world, and recognize the vast users this world provides for. At the same time, I appreciate a back-to-basics approach that emphasizes systems that can be understood and controlled by individuals and small communities: * Hardware that has open specs: Pinephone, Pinebook, Raspberry Pi, ... * Open source OSs: Linux desktop, Linux mobile, Lineage OS. * https://reproducible-builds.org/ and https://www.bootstrappable.org/, allowing trust to be distributed. * Monorepo based OSs: NixOS, Guix. * Gemini, rather than the web: web browsers are now beyond reasonable understanding/control for small communities. * Mastodon rather than Twitter. * Projects that are community driven. * XMPP/Matrix over Signal/Discord/FB. I choose to live within this world for my personal world where possible. It's not as featureful, and that's fine.

I disagree that rust, go or python inherently do not work with linux package managers, instead there's a culture within subsections of those communities who do not value stability (I'm not sure about nodejs community though), which cause these disagreements. C isn't immune to this (and never has been)—science codes are infamous around their lack of care around stability (and IMHO one of the reasons for scientific python's success over alternatives has been this stability). I'd also disagree with the idea that the alternative is reliable—try building and modifing a project that hasn't been touched in 6 months, and see how reliable that is. With your tree of dependencies (absent specific projects which abstract over a set of unstable dependencies and hence implicitly stabilise them, e.g. SDL) the stability of your project (whether it is a library or application or framework) is set by the least stable dependency you have. Increasing you dependencies increases the risk, but if you have an ecosystem which values stability, larger dependency trees should not see a significant increase in risk (personally, I'd love to see the scientific rust ecosystem achieve similar stability to python). That's not to say linux distros are perfect (change can be slow ;)), but there's lots of little things they do get right (how many projects handle updating configuration files correctly—that kind of thing is built into distro tooling), and they enable ecosystem-wide changes more that the…

It's probably exactly your concern about the ability for apps with network access to upload user data that led the Android team to introduce the locations permissions for BLE identifier scanning. Without a location requirement, an app could claim to use only Wi-Fi and BLE permissions, yet it could combine beacon scanning and data access to de-anonymize user locations. This permission requirement was introduced in Android 6.0 (see the release notes[1]) in October 2015, so it's been around for a while. If you have some suggestions around how to improve the tracking of permission usage (static analysis? run-time requests (preferably avoiding times when users are under duress and likely to click 'OK' by default)?) then you may wish to file some requests with them and/or contribute to other projects that you feel are taking a better approach. Your concerns around trust in app developers -- regardless of whether they are a government or any other entity -- are best handled by two means: - Open sourcing the code (which NHSX have done) - Enabling reproducible builds[2] so that users can confirm they have an authentic binary build of the source code [1] - https://developer.android.com/about/versions/marshmallow/android-6.0-c… [2] - https://reproducible-builds.org/

- Even assuming all this, a hybrid approach of HTTP + IPFS (or DAT) is still better than what we have now, since IPFS is essentially a worldwide CDN for static files.(Sorry: an inter-planetary one.) - The content-addressing aspect makes it perfect for distributing commonly used libraries. - We already cache all this content locally. What a waste! Why do I have to fetch jQuery from fricking California when it's sitting on my girlfriend's phone in the other room? - This extends beyond the web: think about the benefits (both in security, practicality, and performance) of content addressing introduced into package managers (take it one step further even: combine this idea with the new move towards reproducible builds (https://reproducible-builds.org) and package managers like guix and nix and things get really interesting). - It's actually easier to use for the average person. If you don't think this is the case I propose a simple experiment: download the beaker browser and set up a simple static site. I recently did this. It really is one-click hosting! Considering how complicated web hosting is to the average person (ever try to walk a friend through setting up a website? not. fun.) -- people would love to be able to set up personal websites this easily... and for free? - As others have mentioned, there are many solutions being worked on for the mirroring of data (Filecoin etc). - For websites that are visited regularly, this is not an issue -- all content is cached…

The hard part -- sometimes really hard -- is the "bit-for-bit identical" requirement. Lots of builds are recreatable but not reproducible (there is probably a better term of art here). You can go back to a point in time and build the version of the software as it was, but you are not guaranteed to get a bit-for-bit clone. (See https://reproducible-builds.org for a thorough discussion) The problem is that there are lots of uncontrolled inputs to a build that are due to sourcecode or compiler changes. Most famously there are timestamps and random numbers, which mess up all sorts of hashing-based approaches. These can even be non-obvious. Just the other day I and a colleague were investigating the (small but unsettling) possibility that an old buildpack had been replaced maliciously. We compared the historical hash to the file: different. We rebuilt the historical buildpack with trusted inputs: still different. Then we unzipped both versions and diff'd the directories: identical. What had thrown our hashes off was that zipfiles, by default, include timestamps. We have a build that is recreatable but not reproducible. Speaking of builds, we are able to reproducibly build some binaries but not others. Off the top of my head our most high-profile non-reproducible build is NodeJS. Some other binaries (Ruby and Python, in my not-at-all-complete recollection) are fully reproducible. This difficulty with fully reproducing makes it hard to provide a fully trustworthy chain of…

” If you are wondering why we are doing this at all, then hopefully the Reproducible Builds website will explain why this is useful.” https://reproducible-builds.org/ Could you perhaps respond to the argumentation here?

If your builds are not deterministic for the same set of inputs, you are doing something wrong - or victim of supply chain attack. https://reproducible-builds.org/

Debian has an ongoing project (https://wiki.debian.org/ReproducibleBuilds) to have 100% reproducible builds and are getting pretty close (numbers wise). Other Linux distros are working toward it as well. Many people care about this ... https://reproducible-builds.org/.

Reproducible builds ensure that you can build the same binaries with the same source code. Nothing like the current date for instance gets in the way of getting a different build. This allows independent people to check that provided binaries don't contain malicious stuff for instance. Ultimately, it lets you download binaries instead of rebuilding everything yourself locally if the binaries have been independently reproduced. You might be interested in checking: https://reproducible-builds.org/ https://en.wikipedia.org/wiki/Reproducible_builds

I would say it's considered a bad idea, since git comes from an OpenSource world and its implied understanding, where you don't depend on a specific binary release to reproduce your results, but your results can always be reproduced purely from source - binaries are just considered a product of "the plaintext truth recorded in your repository". Sure, there are nuances to it as the need for ReproducibleBuilds [1] project demonstrated or people proving this concept of "bootstrapped reproducibility" to the extremes like Guix' full-source bootstrap [2], but I believe the fundamental understanding is the same. [1] https://reproducible-builds.org/ [2] https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-building-f…

> there is nothing deterministic https://reproducible-builds.org/ > nothing [...] and predictable Even if the build isn't bit-for-bit-reproducible, any reasonable toolchain will, at least, give predictably equivalent results.

Ok, but my understanding is that that they are mostly deterministic. And that there are initiatives like Reproducible Builds (https://reproducible-builds.org) that try to move even more in that direction.

"In this way, consumers need only trust a small number of secure build platforms rather than the many thousands of developers with upload permissions across various packages."[1] Bad idea. We should instead have 1000's of people around the world continuously reproducing builds of software, and continuously verifying amongst each other that everyone is building and then using the same peer reviewed version of software.[2] In this way, an attacker is forced to reveal their attack to everyone and can't just compromise the "secure build platforms" and conduct selective attacks. [1] https://slsa.dev/spec/v1.0/use-cases#open-source [2] https://reproducible-builds.org/

https://martinfowler.com/bliki/ReproducibleBuild.html and https://wiki.debian.org/ReproducibleBuilds and https://reproducible-builds.org/ says just because something is "repeatable", doesn't mean it is "reproducible".

> That’s true of everything you haven’t compiled yourself Except if your software is reproducible: https://reproducible-builds.org/

clang+llvm releases do a three stage build and compare the output of the third stage with the second. This is fairly effective at rooting out many sources of indeterminate behavior. > In broad strokes they do but often they’ll have randomness creep in, such as different build hashes or iteration order of associative containers. In order to find defects related to codegen being altered by incidental ordering of objects in containers, a build option called LLVM_ENABLE_REVERSE_ITERATION was created. Builders periodically run with this mode to verify that the regression test suite still passes when containers iterate in reverse. That said, it's true that there are probably remaining sources of variation among builds. There is a significant effort [1] to find these and avoid them. [1] https://reproducible-builds.org/

In practice, chasing down the build variations has been proven workable with a good track record. See eg https://reproducible-builds.org/

This was posted to the the reproducible builds mailing list as "Argument about reproducible builds on usenet in 1999". A bit of a historical curiosity. https://reproducible-builds.org/ https://lists.reproducible-builds.org/listinfo/rb-general