Lockdown Sceptics

The conclusions from the first critical code review cited: All papers based on this code should be retracted immediately. Imperial’s modelling efforts should be reset with a new team that isn’t under Professor Ferguson, and which has a commitment to replicable results with published code from day one. On a personal level, I’d go further and suggest that all academic epidemiology be defunded. This sort of work is best done by the insurance sector. Insurers employ modellers and data scientists, but also employ managers whose job is to decide whether a model is accurate enough for real world usage and professional software engineers to ensure model software is properly tested, understandable and so on. Academic efforts don’t have these people, and the results speak for themselves. https://lockdownsceptics.org/code-review-of-fergusons-model/

There's 3 tweets in that thread: > It is disappointing, but not really unexpected, to see this take on the epidemic simulation code release: https://lockdownsceptics.org/code-review-of-fergusons-model/ If we accept it at face value, we have a retired software engineer making the case that non-determinism in a simulation shows incompetence. I am all \ > for deterministic-everything, but issues with random seeds and parallel floating point reductions are extremely widespread, and a great deal of science gets done with non-deterministic simulations. Heck, professional software engineering struggles mightily with just making \ > completely reproducable builds. I take no position on the mechanics of the simulation or the conclusions drawn from the runs, but implicitly encouraging scientists to keep their code secret if it isn't "perfect" is damaging.

I am referring to this blog post: https://lockdownsceptics.org/code-review-of-fergusons-model/ It says, word-by-word: > Clearly, the documentation wants us to think that, given a starting seed, the model will always produce the same results. > >Investigation reveals the truth: the code produces critically different results, even for identical starting seeds and parameters. > I’ll illustrate with a few bugs. In issue 116 a UK “red team” at Edinburgh University reports that they tried to use a mode that stores data tables in a more efficient format for faster loading, and discovered – to their surprise – that the resulting predictions varied by around 80,000 deaths after 80 days: ... The bugs which the blog post implies here are such ones as described by Jens Regehr: https://blog.regehr.org/archives/213 Not that I do not endorse these statements in the blog - I am rather skeptical whether they are true at all. What the authors of the blob post mean is clearly "undefined behaviour" in the sense of non-deterministic program execution of a program that is not well-formed. It is clear that many non-experts could confuse that with the pseudo-randomness implicit in Monte-Carlo simulations, but this is a very different thing. The first is basically a broken, invalid, and untrustworthy program. The second is the established method to produce a computational result by introducing stochastic behavior, which is for example how modern weather models work. These are wildly…

Really? > Imperial are trying to have their cake and eat it. Reports of random results are dismissed with responses like “that’s not a problem, just run it a lot of times and take the average”, but at the same time, they’re fixing such bugs when they find them. They know their code can’t withstand scrutiny, so they hid it until professionals had a chance to fix it, but the damage from over a decade of amateur hobby programming is so extensive that even Microsoft were unable to make it run right. That's from the first analysis[1]. There's a follow up[2]: > Sadly it shows that Imperial have been making some false statements. and > It’s clear that the changes made over the past month and a half have radically altered the predictions of the model. It will probably never be possible to replicate the numbers in Report 9. [1] https://lockdownsceptics.org/code-review-of-fergusons-model/ [2] https://lockdownsceptics.org/second-analysis-of-fergusons-model/

John Carmack replies https://twitter.com/ID_AA_Carmack/status/1258192136744456199: > It is disappointing, but not really unexpected, to see this take on the epidemic simulation code release: https://lockdownsceptics.org/code-review-of-fergusons-model/ If we accept it at face value, we have a retired software engineer making the case that non-determinism in a simulation shows incompetence. I am all for deterministic-everything, but issues with random seeds and parallel floating point reductions are extremely widespread, and a great deal of science gets done with non-deterministic simulations. Heck, professional software engineering struggles mightily with just making completely reproducable builds. I take no position on the mechanics of the simulation or the conclusions drawn from the runs, but implicitly encouraging scientists to keep their code secret if it isn't "perfect" is damaging. Also later: > There was a continuous integration regression test running a small model in single threaded mode on every check in, comparing for exact accuracy, at least when I was working on it -- I haven't pulled the public release. For background, he was pulled in to do a bit of work on the simulation code a month or two ago.

But you're overlooking the most likely outcome - that nobody is qualified in this field. However software developers can admit this and epidemiologists can't. Bearing in mind for a moment that epidemiologists are amateur programmers, their work output is literally papers+code, they have a major handicap that they can't stop and say "maybe this doesn't work". What else would they do? I've found the comments by Caswell Bligh very insightful on this topic. Bligh is one of the awful "men in tech" programmers the article laments, who has been writing his own models based on epidemiological papers. Perhaps even from an armchair. Everything he's concluded is worth reading but I'll quote some extracts: https://lockdownsceptics.org/2020/04/26/latest-news-12/#comment-935 Do ‘R0’ or the other ‘R’ values even mean anything when ‘infection’ versus ‘exposure’ can’t even be defined? If social distancing reduces the level of exposures but not the number, does this mean the ‘R’ number is the same, or reducing? Maybe the lockdown means that the population is being quietly, in the background, inoculated to a certain level of resistance (which is what I think is probably happening) but that it doesn’t show up in antibody tests. Many of these things will never be known. They are probably not even predictable with viruses that are not ‘novel’. Virologists and immunologists may know some of it, and they also know what they don’t know. The modellers don’t know any of it. As such, models will…

It also sounds exactly like Ferguson's Imperial College epidemiology model that apparently compelled politicians into imposing hard lockdowns (and was likely wrong by at least an order of magnitude): - "a single 15k line C file that had been worked on for a decade" [0] - code review of the model: [1] - corresponding HN discussion: [2] (including sad appeals to authority: you're not an epidemiologist) - other HN discussion [3] (including ridiculously blaming programmers for making C++ available to non-programmers) [0] https://twitter.com/ID_AA_Carmack/status/1254872369556074496 [1] https://lockdownsceptics.org/code-review-of-fergusons-model/ [2] https://news.ycombinator.com/item?id=23093944 [3] https://news.ycombinator.com/item?id=23222338

So, I looked at the article they are refuting (or one of them?) and expected a bunch of superficial criticism as suggested in this refutation, but it does not look to me at all that "superficial" criticism is being defended against: https://lockdownsceptics.org/code-review-of-fergusons-model/ ^^^Swallow your politics if you can for a moment and read. At a minimum it seems that the rebuttal here was being argued in very bad faith. Given the stage set, I was expecting bikeshedding about the Law if Demeter or JSON vs other format, etc...

John Carmack who cleaned up the codes take. > it turned out that it fared a lot better going through the gauntlet of code analysis tools I hit it with than a lot of more modern code. There is something to be said for straightforward C code. Bugs were found and fixed, but generally in paths that weren't enabled or hit. https://twitter.com/ID_AA_Carmack/status/1254872370260652033 Also his opinion on the deniers efforts. > It is disappointing, but not really unexpected, to see this take on the epidemic simulation code release: https://lockdownsceptics.org/code-review-of-fergusons-model/ If we accept it at face value, we have a retired software engineer making the case that non-determinism in a simulation shows incompetence. https://twitter.com/ID_AA_Carmack/status/1258192134752145412

Has anyone ever reviewed the code and analyzed the software engineering practices behind the climate models? I used to take these things for granted and just believe articles and studies like this one but after humongous public failure of Imperial College Coronavirus model I'm not so sure anymore. We need code and tech reviews in addition to peer review. https://lockdownsceptics.org/code-review-of-fergusons-model/ http://www.rationaloptimist.com/blog/lockdown-and-mathematical-guesswo…

They're all just winging it. The software model being used is garbage: https://lockdownsceptics.org/code-review-of-fergusons-model/ The "Ferguson Model" is described here: https://www.nature.com/articles/d41586-020-01003-6

There's a code review, of sorts, (from an ex-Googler) here: https://lockdownsceptics.org/code-review-of-fergusons-model/

Review by someone claiming to be a ex-Googler here: https://lockdownsceptics.org/code-review-of-fergusons-model/ If the original code was a single file and you’ve ‘fixed bugs’ then what is on GitHub can in NO way be ‘largely what was written’ This code was used to lockdown an entire country - defending the code now based on who it was written by and under what constraints is disingenuous. It must have been known that there were these issues and they ought to have been fixed before using the output for something that had such huge societal impact.

Intermittently down, but points to https://lockdownsceptics.org/code-review-of-fergusons-model/

Some people have raised issues with the model[1][2], or rather, the software implementation of it. There's also a (flagged) submission on HN discussing this[3] referencing [1]. [1] (warning: possibly partisan link) https://lockdownsceptics.org/code-review-of-fergusons-model/ [2] https://github.com/mrc-ide/covid-sim/issues/165 [3] https://news.ycombinator.com/item?id=23099212

This article brings up scientific code from 10 years ago, but how about code from .. right now? Scientists really need to publish their code artifacts, and we can no longer just say "Well they're scientists or mathematicians" and allow that as an excuse for terrible code with no testing specs. Take this for example: https://github.com/mrc-ide/covid-sim/blob/e8f7864ad150f4002279cff6cf86… This was used by the Imperial College for COVID-19 predictions. It has race conditions, seeds the model multiple times, and therefore has totally non-deterministic results[0]. Also, this is the cleaned up repo. The original is not available[1]. A lot of my homework from over 10 years ago still runs (Some require the right Docker container: https://github.com/sumdog/assignments/). If journals really care about the reproducibility crisis, artifact reviews need to be part of the editorial process. Scientific code needs to have tests, a minimal amount of test coverage, and code/data used really need to be published and run by volunteers/editors in the same way papers are reviewed, even for non-computer science journals. [0] https://lockdownsceptics.org/code-review-of-fergusons-model/ [1] https://github.com/mrc-ide/covid-sim/issues/179

>And idiots will continue parroting this exact line as billions of people are dying. Exactly the same way idiots are protesting social distancing, lockdowns, and mask wearing in US right now There is no purpose to have a discussion when your thinking is so narrowly black and white. Your worldview doesn't seem to allow for the shades of grey necessitated by uncertainty. If all skepticism was met with the same belittling as climate science, we'd still be stuck in the dark ages. Don't think it's possible for an institution of sciences to be very wrong for a very long time about a very important topic? Look at the forces that caused us to fruitlessly pursue the amyloid plaque hypothesis for decades. Consensus has a strong normalizing effect, but that does not necessarily mean that a given consensus is correct. Anyway I'm being rate limited by sanctimonious downvoters so this conversation is effectively over. Edit: by the way, with respect to the lockdown controversy, did you hear about the atrocious academic modeling code that informed the lockdown policy? A single 15k line C file which, among other things, does not seed random number generation and cannot be reproduced. So bad that I just found there's an entire website dedicated to a teardown[0]! But I'm an "idiot" for showing skepticism. Do you think climate modeling code is any better? Have you worked with academic code? I have. It's universally bad. But, again, I'm an "idiot" for expressing any degree of…