Safety Research & Strategies

Apparently Toyota followed their own internal standards, the code met them, and no bug that would cause unintended acceleration has ever been found. I still wouldn't advocate for the problems that Barr and Koopman savaged as expert witnesses in the lawsuit, but Toyota was held to an extremely high standard in court. https://users.ece.cmu.edu/~koopman/pubs/koopman14_toyota_ua_slides.pdf https://www.safetyresearch.net/Library/BarrSlides_FINAL_SCRUBBED.pdf

http://www.safetyresearch.net/Library/NHTSA_Autosteer_Safety_Claim.pdf "As a consequence, the overall 40 per-cent reduction in the crash rates reported by NHTSA following the installation of Autosteer is an artifact of the Agency’s treatment of mileage information that is actually missing in the underlying dataset."

Whenever I see articles like this, I like to remind people that a great production system doesn't make a company more honest or accountable. Unintended acceleration flaw in Toyota models around 2005 to 2011, due to firmware bugs: http://www.safetyresearch.net/blog/articles/toyota-unintended-accelera… Great for USA customers. They were made whole. I personally was driving a 2011 Toyota minivan in Thailand in 2013 when I suffered this firmware bug. Toyota refused to consider it was anything other than driver fault. I was only going 5kph when it happened so no damage, but damn scary as hell. No recall in that country, and unless dealers did a covert firmware upgrade during maintenance, the flaw still exists in similar model years.

I think the Bookout v. Toyota case is a pretty good example of how culpability alone is an insufficient motivator and how an add process (e.g. external audit) could have prevented a tragedy. Michael Barr's review of Toyota's ECU code (http://www.safetyresearch.net/Library/BarrSlides_FINAL_SCRUBBED.pdf) showed numerous compliance issues with established industry best practices (80,000 violations of MISRA-C) and failure to even follow Toyota's much laxer internal coding standards (32% rule violation). Toyota shipped uncertified versions of their code and the design and behavior of that code prevented defect detection.

Nice article, thanks for posting it. It's ironic that this article mentions the Toyota Production System as an example of a safe and defect-free system. Another article about Toyota was posted on HN today: «Toyotas Unintended Acceleration and the Big Bowl of "Spaghetti" Code (2013)» http://www.safetyresearch.net/blog/articles/toyota-unintended-accelera… Apparently Toyota's software development doesn't follow the Toyota Production System. OK, that was a flip comment; it's pretty clear that TPS isn't suited to software development. However, it does seem clear that Toyota's software development practices are deficient.

Reading through the crash test procedure, it is astounding how little attention is paid to pedestrians. 1. Front crash test. Procedure: Crash car into stationary barrier at 35 mph. Is also applicable to face-to-face crash with car of same size, going at same speed. 2. Side crash test. Procedure: Slam concrete block into side of stationary car at 38.5 mph. 3. Side pole test. Procedure: Drag car sideways towards a pole. 4. Rollover resistance. Procedure: Compare the cars footprint to the height of the center of gravity. The biggest thing to notice is that not one of these metrics involves pedestrians. Metrics 1-3 can be easily improved by making a bigger car, elevating the passengers and providing more crumple room. Metric 4 is unaffected, as the track width is increased to compensate. If a low sedan hits a pedestrian, the pedestrian rolls over the car, having a lower impulse given over a longer period of time. If a high SUV hits a pedestrian, the pedestrian is knocked back, having a higher impulse given over a shorter period of time. Safety ratings need to account for the danger cars pose to others. Source: https://www.nhtsa.gov/ratings Source (SSF): https://www.safetyresearch.net/rollover-stability

That's a good question. The first thing that came to mind for me was the bad, namely Toyota's unintended acceleration debacle some years back.[0] If anything this illustrated (at the time) a woeful lack of any proper software engineering practices at Toyota that was to some degree indicative of a larger problem across their entire industry. No doubt this was due in large part to Japan's corporate culture and their admirable-yet-deleterious work ethic. As far as the good, the first place I thought to look was Hideo Kojima's games, namely their engine technology. The history of the Fox Engine[1] and in particular the Decima[2] engine appears very interesting. Kojima games are almost always bleeding-edge with respect to rendering technology. There's of course Nintendo. I had a friend who worked in software QA there, and there wasn't much good to say about it. That said, it was some years ago and one person's experience in one location. Japan used to be a leader in robotics, though I'm not sure how true that is anymore what with Boston Dynamics in the spotlight in recent years. To answer your question however, my guess would be no, probably not—Japan is likely playing catch up right now in terms of software practices. The Japanese development teams and researchers that do output quality, I'd imagine they just use common sense good practices recognized internationally. Namely: hire high-caliber talent, pay them well and don't overwork them. Quality usually follows regardless…

> It's not just money. It's paperwork, bureaucracy, and ultimately not effective. Practically, the only way to ensure safety is to take down the internet. https://logfall.wordpress.com/all-or-nothing-fallacy/ We can and should be making incremental improvements to technology. Continuing to use methods that are known to be faulty is gross negligence in the legal sense: https://en.wikipedia.org/wiki/Gross_negligence We do not see enough lawsuits around software negligence today because companies are under no obligation to release source code - there is no way to tell how or why or even whether software is defective. This was a major issue in the investigation of the Toyota unintended acceleration scandal. The investigation uncovered major, obvious code quality problems, ultimately concluded the fault was caused by software (something Toyota denied, and without the investigation, consumers never would have learned), and I hope becomes a precedent: http://www.safetyresearch.net/blog/articles/toyota-unintended-accelera… IMO if poor coding practice is gross negligence, it follows that failing to reveal the source code constitutes intent: poorly developed closed-source software is criminal negligence.

for another example, see the detailed (scrubbed?) study https://www.safetyresearch.net/Library/BarrSlides_FINAL_SCRUBBED.pdf Since then we have extended automotive software from engine control to driving.

>It can't be significant 1.3 Billion-Dollar settlement. Seems significant. https://safetyresearch.net/hyundai-kias-billion-dollar-engine-problem-…

I know for the fact simulink generates spaghetti and spaghetti code was partially blamed for Toyotas problems. Hence the inference See https://www.safetyresearch.net/toyota-unintended-acceleration-and-the-…

Some information about the Toyota cases: https://users.ece.cmu.edu/~koopman/pubs/koopman14_toyota_ua_slides.pdf page 14 is especially interesting. And more technical information: https://www.safetyresearch.net/Library/BarrSlides_FINAL_SCRUBBED.pdf

An analysis by expert witnesses in the trial found that a small amount of memory corruption could trigger task death and unintended acceleration. The report did not find the cause of the memory corruption, but many software errors can corrupt memory. https://www.safetyresearch.net/Library/BarrSlides_FINAL_SCRUBBED.pdf

I also nominate the "Toyota Unintended Acceleration" issue. A lecturer friend of mine likes to reference these slides from the trial: https://www.safetyresearch.net/wp-content/uploads/2013/11/BarrSlides_F…

This? From almost 20 years ago. https://www.safetyresearch.net/Library/BarrSlides_FINAL_SCRUBBED.pdf

> I'd expect that if any ECU software was to be released that we'd finally realize how bad things really are and that there would be a massive amount of work done on making sure these pieces of critical software would be as safe as they could possibly be. toyota/denso michael barr testimony cough edit: oh, there's even slides now, you don't even have to read the court transcript https://www.safetyresearch.net/Library/BarrSlides_FINAL_SCRUBBED.pdf

Lots of discussion online about the investigation. Search for “Toyota ECU spaghetti code” One result: https://www.safetyresearch.net/toyota-unintended-acceleration-and-the-…

Have you seen the report on the Toyota source code? http://www.safetyresearch.net/Library/BarrSlides_FINAL_SCRUBBED.pdf https://news.ycombinator.com/item?id=9449559 "In a nutshell, the team led by Barr Group found: “a systematic software malfunction in the Main CPU that opens the throttle without operator action and continues to properly control fuel injection and ignition” that is not reliably detected by any fail-safe." And the black box could fail to record any braking input in such cases. That's not good! People died! And you would have a hard time convincing a jury that the driver was not at fault, if not for the thorough independent code review. And that's the safety critical stuff, now imagine all the code for the infotainment systems.

There have been difficulties in independently verifying at least some of these claims. I do not know whether they have been cleared up. http://www.safetyresearch.net/blog/articles/quality-control-systems-co… https://jalopnik.com/feds-cant-say-why-they-claim-teslas-autosteer-cut…

I agree; it's like all the rollaway accidents that happen now because of electronic shifters: https://www.safetyresearch.net/the-persistence-of-rollaway/ Design matters, and bad designs can literally be deadly. Why auto mfgs feel the need to "innovate" with shifter designs is beyond me. The worst part is how every mfg seems to be implementing a different design of bad electronic shifters, from wheels and touchscreens to one-click-at-a-time joysticks to single-function pushbuttons for some gears with others on a scroll wheel. They've taken something and made it worse with no benefit to the user. At least with those auto-flushing toilets, the intention was good, even if the implementation is still somehow so awful decades later.

The investigations carried out for unintended acceleration in Toyotas didn't paint a good picture. https://www.safetyresearch.net/blog/articles/toyota-unintended-acceler… https://users.ece.cmu.edu/~koopman/pubs/koopman14_toyota_ua_slides.pdf

I don't know if they ever proved beyond possible doubt that any particular incident was caused by a software bug, but they did show that there were myriad ways in which the software in question could cause the acceleration problems. (In particular the process which handled reading the accelerator pedal position could crash and leave the throttle value fixed without tripping the watchdog timer, which iirc was doubly bad because in push-to-start vehicles there was no key to turn off.) Source: http://www.safetyresearch.net/blog/articles/toyota-unintended-accelera… More: https://embeddedgurus.com/barr-code/2013/10/an-update-on-toyota-and-un…

This is in a footnote: http://www.safetyresearch.net/Library/2018-11-26%2520Redacted%2520PE16…

> Braking systems are controlled by a dedicated embedded software system that's extremely well-understood and well-tested (probably even formally verified). Are you sure? That code is closed source, and the peek behind the curtain we saw with Toyota's investigation did not inspire confidence: http://www.safetyresearch.net/blog/articles/toyota-unintended-accelera…

A rather stark rebuttal to Gladwell's analysis was posted elsewhere in this discussion: http://www.safetyresearch.net/blog/articles/tipping-point Edit: I see this is addressed in updates to the Revisionist History posting, in PDF form.

I'm disappointed in Malcom Gladwell's analysis of this situation. His advice is dangerous and should be corrected. Here is a response to his piece that is well laid out. http://www.safetyresearch.net/blog/articles/tipping-point

Perhaps have a look at Koopman https://users.ece.cmu.edu/~koopman/pubs/koopman14_toyota_ua_slides.pdf and Barr http://www.safetyresearch.net/Library/BarrSlides_FINAL_SCRUBBED.pdf

Check out this, and the linked court transcripts linked therein: http://www.safetyresearch.net/blog/articles/toyota-unintended-accelera… The security researchers who were expert witnesses in the Toyota case and that analyzed Toyota's source code testified in court that they managed to reproduce the unintended acceleration with a real car on a dynamometer.

Ironic that Tesla touts the value of being transparent. It freely cites the NHTSA report that Autopilot seems to reduce crashes by 40%. But the NHTSA is currently being sued because it won't release the data behind that finding: http://www.safetyresearch.net/blog/articles/quality-control-systems-co…

Yup, there's a lawsuit trying to get the raw data behind the 40 percent number [1]. The original report [2] gives two averages, the crash rate per million miles driven before/after autosteer, 1.3 and 0.8. It doesn't say how many miles were driven, how many crashes, or when any of these occurred, so it's impossible to verify that result. Further, the same NHTSA report notes that AEB reduces rear-end collisions by 40 percent. Supposedly, AEB and autosteer were enabled around the same time. [1] http://www.safetyresearch.net/blog/articles/quality-control-systems-co… [2] https://static.nhtsa.gov/odi/inv/2016/INCLA-PE16007-7876.PDF