John Dalbey

It's probably worth considering a return to a fallback comms system. In the 1990s, telcos were the pinacle of reliability. Sure, AT&T had local issues from time to time, and that cascading failure incident [1] in 1990, but things were on a path towards even more reliability. Now, I expect a several hour, multistate outage of 911 about every other year. Nationwide single carrier cell phone outages aren't uncommon either. I'm just going to assume a hostile first strike makes satellite comms difficult. [1] http://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collapse

You should read this 1994 SciAm article (Software's Chronic Crisis, SciAm, September 1994). There are many copies of it on the web. It's old but it's still the best summary of the problem as it puts the software issue into context with other fields of engineering. These links were the first I found and they've tested OK. The first is an html version, the others are PDFs with the last being the best quality (in color). If they don't work just search for the title and year. http://users.csc.calpoly.edu/~jdalbey/SWE/Papers/SciAmGibbs/SciAmGibbs… http://sarec.nd.edu/courses/SE2017/Week1-TheSoftwareCrisis.pdf https://www.researchgate.net/profile/Wayt-Gibbs/publication/247573088_…

History doesn't repeat, but it rhymes .... There was a huge AT&T outage in 1990 that cut off most US long distance telephony (which was, at the time, mostly "everything not within the same area code"). It was a bug. It wasn't a reconvergence event, but it was a distant cousin: Something would cause a crash; exchanges would offload that something to other exchanges, causing them to crash -- but with enough time for the original exchange to come back up, receive the crashy event back, and crash again. The whole network was full of nodes crashing, causing their peers to crash, ad infinitum. In order to bring the network back up, they needed to either take everything down at the same time (and make sure all the queues are emptied), but even that wouldn't have made it stable, because a similar "patient 0" event would have brought the whole network down. Once the problem was understood, they reverted to an earlier version which didn't have the bug, and the network re-stabilized. The lore I grew up on is that this specific event was very significant in pushing and funding research into robust distributed systems, of which the best known result is Erlang and its ecosystem - originally built, and still mostly used, to make sure that phone exchanges don't break. [0] https://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collapse

> The problem is that coverage tools report whether that line of code executed and not whether its logic is correct. This can easily give you a false sense of security. This is true. But isn't that true of any test? A suite of naive or badly written tests can also give us a false sense of security, so why write any at all (I don't mean that literally)? I think that a greater level of confidence in our code can be achieved by a combination of - Judicious choices of unit and integration tests - Static and dynamic analysis (if the language supports it), and - Property-based testing (the canonical example of which is QuickCheck[1]). Property-based testing is a great way to help us hit those edge cases. As for 100% code coverage, I think it is worth striving for, not just for the sake of having all lines of code tested. It can expose design and testability flaws, for example. If 100% coverage cannot be achieved, we need to ask ourselves: - Did we really need that piece of code we couldn't test? Is it actually called anywhere, or is it one of those YAGNI things? - Did we write code that is not very testable? Are there functions or methods, for example, that are so dependent on external state that they can't be mocked or tested some other way? Should we refactor it? - If it's a trivial line of code like a getter or setter that cannot possibly be wrong, then it should be fairly trivial to generate automatically a test case for it. More severe defects have probably been…

Yeh I can't find anything about this, only the outage in 1990: https://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collapse

Something like a major telco going out, for example the AT&T 1990 outage of long distance calling: > The standard procedures the managers tried first failed to bring the network back up to speed and for nine hours, while engineers raced to stabilize the network, almost 50% of the calls placed through AT&T failed to go through. > Until 11:30pm, when network loads were low enough to allow the system to stabilize, AT&T alone lost more than $60 million in unconnected calls. > Still unknown is the amount of business lost by airline reservations systems, hotels, rental car agencies and other businesses that relied on the telephone network. https://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collapse

Here is one with a negative impact https://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collapse

Software that killed people: http://users.csc.calpoly.edu/~jdalbey/SWE/Papers/THERAC25.html https://spectrum.ieee.org/how-the-boeing-737-max-disaster-looks-to-a-s…

This comment tickled something in my memory: All Circuits are Busy Now: The 1990 AT&T Long Distance Network Collapse http://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collapse

"I did." That was CEO Robert Allen's response when the AT&T network collapsed [1] on January 15, 1990 He was asked who made the mistake. I can't imagine any CEO now a days making a similar statement. [1] http://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collapse.html

A small error, which, if I remember correctly, although it's not mentioned in this account, was in a tiny change that was not tested as fully as it should have been, took down AT&T's long distance network in 1990, back when it probably still carried most of the nation's long distance calls: http://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collapse.html

Sorta related: "All Circuits are Busy Now: The 1990 AT&T Long Distance Network Collapse": http://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collapse.html

Oh yeah. Here is a REAL MEDICAL DEVICE, BUILT TO HIGHEST STANDARDS failing radiation dosing: http://users.csc.calpoly.edu/~jdalbey/SWE/Papers/THERAC25.html And you think that the lowest-bid contractor who wrote the software for the porno scanners is of higher quality. Your optimism is cute.

Doesn't matter, things go wrong. http://users.csc.calpoly.edu/~jdalbey/SWE/Papers/THERAC25.html

> Between June 1995 and January 1987, six patients were seriously injured or killed by unsafe administration of radiation from the Therac-25 medical linear accelerator. > The Therac-25 software errors that cause radiation overexposures can be reduced down to interface errors. The first of these errors involved the entering of treatment data by the machine operator. Once an operator enters treatment information at the terminal outside of treatment room, the magnets used to filter and control radiation levels are set. There are several magnets, and the process takes about 8 seconds. If the operator makes a very, very quick change of the treatment information, within 1 second, the change is registered. Or, if the operator is rather slow about it, takes more than 8 seconds, the change is also registered. However, if the change occurs within the eight seconds it takes to set the magnets, the change is not detected and the magnets continue to be set up improperly, and thus the level of radiation is set up improperly. > The last of the accidents occurred at the Yakima Valley Memorial Hospital. On January 17, 1987 an operator placed a patient on the turntable in the field-light position for small position verification doses. After attempting to administer the treatment dose, the machine shut down with a quick malfunction message and a treatment pause. The operator pushed the "P" button, and the machine paused again. The machine indicated that the patient had received his…