Embedded Gurus

great rant, but then he submits code like if (mpa && mpa->locale) freelocale(mpa->locale); which are known to cause bugs fairly easily (see gotofail [0]) [0] https://embeddedgurus.com/barr-code/2014/03/apples-gotofail-ssl-securi…

Making a PoC is usually only the beginning. Firmware upgrades for microcontrollers may be difficult to do in the field, without UART and JTAG debugging. If clients expect that pretty demos will continue working, they might be quite disappointed. It's better to follow good design patterns (Active Objects, state machines [1]) from the start when possible. dmitrygr is a legendary embedded hacker [2]: his Transcend WiFi SD card reverse-engineering is what led to me having a PQI Air Card in my pocket, and he's also written a bit-banged Bluetooth Low Energy driver. Great for a demo or when there's nothing else available, but far from a finished product. Writing a MISRA-compliant BLE driver would make me MISRA BLE. [1] https://embeddedgurus.com/state-space/2016/04/beyond-the-rtos-a-better… [2] http://dmitry.gr/?r=05.Projects

Top 10 Causes of Nasty Firmware Bugs https://embeddedgurus.com/blog/2010/12/top-10-causes-of-nasty-firmware… it would be great if all those would be easy issues, or solved.

To back up your point, this was a great blog post on the fatal consequences of failure in mission-critical car software: http://embeddedgurus.com/state-space/2014/02/are-we-shooting-ourselves… After reading one may think: Why the fuck is there any dynamic memory allocation (stack) going on there? Why isn't the car software implemented as one big pre-allocated hierarchical state machine? You wouldn't expect such execution dynamism in hard circuits or even FPGAs. But in any case I think non-deterministic systems (especially complex CPUs+programs) should be outlawed for such mission critical signal paths.

Thanks for posting this, I was faced with that exact same problem this week. But with the following extra constraints : - Not everybody uses GCC, especially in the embedded world where CLZ is most useful - It must be portable to a Cortex-M0+ that doesn't have a CLZ instruction Like me you probably end up here : http://embeddedgurus.com/state-space/2014/09/fast-deterministic-and-po…

Choosing between bitfields and masks for register twiddling was something I looked into deeply. Like the article, I wanted the readability of bitfields, but after reading about all the pitfalls and how loosely bitfields specs were defined in the C89 standard, I decided to go with masks instead. Here was a great summary I found from a comment explaining the dangers of bitfields. Seeing it spelled out like this made it clear which to choose, lest you're a fan of nasal demons :). "The royal mistake is to use bit fields in the first place. The following is not specified by the standard: – whether bit field int is treated as signed or unsigned int – the bit order (lsb is where?) – whether the bit field can reach past the memory alignment of the CPU or not – the alignment of non-bit field members of the struct – the memory alignment of bit fields (to the left or to the right?) – the endianess of bit fields larger than one byte – whether plain int values assigned to them are interpreted as signed or unsigned – how bit fields are promoted implicitly by the integer promotions – whether signed bit fields are one’s compliment or two’s compliment – padding bytes – padding bits – values of padding bits. – and so on." Source: https://embeddedgurus.com/stack-overflow/2009/10/effective-c-tip-6-cre…

All the time. If you want to be horrified, read about the ludicrous bullshit that is a breathalyzer. The fundamental problem seems to be three fold: first, as anyone not an idiot would expect, animals process alcohol differently so at best you have a model relating alcohol intake with alcohol present in the lungs. Second, the programming appears to be ludicrously bad [1]. Pigs and manufacturers have fought and fought to avoid having to share code behind these devices or present any proof that they measure what they purport to measure besides, "Trust me." When competent engineers have finally gotten their hands on code, you'll be unsurprised to discover it's full of bugs. Third, even accepting the bac to gas model arguendo, it's an open question if these devices measure what they purport to measure. Just a few representative issues when competent software engineers have got their hands on the code: - Of the available 12-bits of A/D precision, just 4-bits (most-significant) are used in the actual calculation. This sorts each raw blood-alcohol reading into one of 16 buckets. (I wonder how they biased the rounding on that.) - Out of range A/D readings are forced to the high or low limit. This must happen with at least 32 consecutive readings before any flags are raised. - There is no feedback mechanism for the software to ensure that actuated devices, such as an air pump and infrared sensor, are actually on or off when they are supposed to be. …

You could always ask Apple. Braces are explicit, whitespace implicit. https://embeddedgurus.com/barr-code/2014/03/apples-gotofail-ssl-securi…

Easy; a subclass struct embeds the superclass struct as its first member. Simple example here: https://embeddedgurus.com/state-space/2008/01/object-based-programming… Lots more here: https://stackoverflow.com/questions/351733/how-would-one-write-object-…

I'm not sure I buy that explanation fully. one of the guys driving was a CHP officer and the acceleration was stuck wide open... the mat could have been a contributing factor but there were software issues that could have caused it too. The witnesses said the car was surging in acceleration immediately prior to the accident. there were also some components changed in the accelerator assembly too. https://www.youtube.com/watch?v=Bx2diBKXN_Y https://embeddedgurus.com/barr-code/2013/10/an-update-on-toyota-and-un…

Apples famous "gotofail" SSL Security Bug was due to this https://embeddedgurus.com/barr-code/2014/03/apples-gotofail-ssl-securi…

Uhhh. https://abcnews.go.com/Blotter/toyota-pay-12b-hiding-deadly-unintended… https://mashable.com/2014/03/19/toyota-lied-aceleration-recall/ https://www.rs-law.com/unintended-acceleration/ And here's an article by one of the experts who reviewed Toyota's code and found the smoking gun: https://embeddedgurus.com/barr-code/2013/10/an-update-on-toyota-and-un…

> Taking 60-90s completely out of protection to reboot a critical defensive system when someone might, at any moment, toss a Mach 5 projectile at you from a couple hundred miles away is a far-from-ideal fix, even if it had been communicated properly to the end users (which it wasn't.) https://embeddedgurus.com/barr-code/2014/03/lethal-software-defects-pa…

Good article. I think this is more relevant for Model based development and auto code generators. In auto-generated code there usually exist may unreachable paths and un-resolved conditions which only surface up when code is run over certain period of time or some environmental conditions are met. Visualising Level 3 as explained in article is easy in model based development but it's consequences on Level 1 and Level 2 are difficult to predict. These things are of topmost priority in safety critical systems. E.g. [1] Toyota's unintended acceleration case [1] https://embeddedgurus.com/barr-code/2013/10/an-update-on-toyota-and-un…

It was my understanding that C++ was being abandoned in the embedded world in favor of C due to the failure of UML. See for example, the intro of this (possibly outdated) article: http://embeddedgurus.com/state-space/2008/01/object-based-programming-… Is this changing now?

I think that the unreliability has entered the public consciousness fairly recently, so we'll see how that goes. Polygraphs offer a more positive example, as well as breathalyzers. [0] [0] http://embeddedgurus.com/barr-code/2009/11/breathalyzer-source-code-an…

Yes, and the PIC's return point stack is tiny. 2 (!) on the smallest machines, 8 on most, 31 on the "high end" PICs.[1] [1] http://embeddedgurus.com/stack-overflow/2009/04/pic-stack-overflow/

The NASA audit wasn't nearly as comprehensive as you may have expected: http://embeddedgurus.com/barr-code/2013/10/an-update-on-toyota-and-uni… NASA didn't find a software defect, this is not the same as NASA finding that there is no software defect. Toyota absolutely built and sold dangerous vehicles with software and hardware defects. This was not only a matter of gas pedals sticking to floor mats. Importantly in this context Toyota did not put customer safety first and decided to suppress the truth of the problem rather than own the solution. This is cause for concern as we move toward self-driving systems which have to be perceived to work in order for the product to remain economically viable.

Well, I'd say it's just pride to be so confident in the ability of human engineers to make machines outperform humans consistently (at least in the next few years that vendors want to start selling auto-drivers.) I don't think anyone's doubting whether radar+computer can move faster than a human, but we may doubt whether it will make the "right" reaction and if it might make unwanted decisions in some other cases we don't think of until they happen. And that's just the "AI" level, not stupid things like your electronic throttle's stack overflowing and locking in the open position[1]. [1] http://embeddedgurus.com/state-space/2014/02/are-we-shooting-ourselves…

And the coding style doesn't exactly avoid errors as well. "Do not unnecessarily use braces around a single statement: if (condition) action(); and if (condition) do_this(); else do_that(); " Didn't people learn from the goto fail bug ? http://embeddedgurus.com/barr-code/2014/03/apples-gotofail-ssl-securit…

Suggestion: Move your business logic into .c files. Headers is there for other reasons. Take a look at, http://embeddedgurus.com/barr-code/2010/11/what-belongs-in-a-c-h-heade…

If Toyota had anything to do with that analysis then it wasn't independent. My recollection of the Barr analysis is that a particular failure case was in fact described which occurs during certain conditions of memory exhaustion: http://embeddedgurus.com/barr-code/2013/10/an-update-on-toyota-and-uni… "the team led by Barr Group found what the NASA team sought but couldn’t find: 'a systematic software malfunction in the Main CPU that opens the throttle without operator action and continues to properly control fuel injection and ignition' that is not reliably detected by any fail-safe."

You mean this one? http://embeddedgurus.com/state-space/2014/02/are-we-shooting-ourselves…

And CARS too: http://embeddedgurus.com/state-space/2014/02/are-we-shooting-ourselves…

You'd think, right? Luckily the ARM dudes came up with Thumb-2, so most (all but 6) of the Cortex-M0+ instructions are actually 16 bit, not 32. (Nice chart at http://en.wikipedia.org/wiki/ARM_Cortex-M .) But what's even more interesting is to actually compile apps and compare code size. Miro Samek did this for a couple of RTOS-like framesworks and compared across a mix of 8, 16 and 32 bit micro architectures (PIC, 8051, M8C, 68HC08, AVR, MSP430, M16C, ARM7 Thumb, and ARM Cortex-M3 Thumb2). For his test, the MSP430 was the most dense, but Thumb2 wasn't far behind. (http://embeddedgurus.com/state-space/2009/03/insects-of-the-computer-w… Of course, there are 4 bit micros out there...

Also relevant (and on HN newest as of now) - Toyota accelerator issue: http://embeddedgurus.com/state-space/2014/02/are-we-shooting-ourselves…

> What makes you think that the engineering teams in companies like Siemens, Rockwell Automation, Mitsubishi, GE etc. are "a long way behind in terms of 'software engineering'"? data. http://embeddedgurus.com/barr-code/2010/09/the-sad-state-of-embedded-s… I was in a class taught by this guy and I seem to recall him saying that a large (20-40%) of embedded developers he surveyed reported using no version control software. and look at some of the responses in that thread! "I think that gcc -Wall -Wextra can be subtituted by lint tool (if you use gcc)."

"v later found out with more debugging that it happened because the module in question cast that struct back from memory, and then added its own stuff to it." That's your problem right there: "cast". In C, you can cast things, but doing so breaks the type system, which can no longer help you. That's not the fault of C, that's your fault. You literally asked for it. C is the useful idioms of assembly language made first-class. For fundamental reasons of the universe, C is the programming language. See also: https://embeddedgurus.com/barr-code/2018/02/c-the-immortal-programming… "This year a portion of my analysis is focused on multi-year trends. One trend that really stands out to me is that the C programming language refuses to give up the ghost. "A longitudinal study of survey data spanning almost a decade and a half shows that C remains the primary programming language of embedded software. Remarkably, in that time C has actually gained market share from 50% to about 70%–at the expense of assembly, C++, and Java."

I think it means that this was the first conflict that it had in against Scuds. There was a software update made a few days before this incident that was in response to another Patriot/Scud engagement. The big issue with this time was that the missile didn't even fire. http://embeddedgurus.com/barr-code/2014/03/lethal-software-defects-pat… It's interesting that there were 6 software updates to the Patriot missile batteries during Desert Storm, several of them were specifically to fix the clock errors. It would be interesting to see the data for the misses and the hits compared to how long each system had been online for and what version of the software it was using. Today's patriots are better (PAC-3, PAC-4 in testing) but still have several issues. I remember reading somewhere where a pilot got so mad that he was constantly being locked on by friendly Patriot batteries that he shot a missile at a friendly battery. http://books.google.com/books?id=7cU8Kpzu9osC&pg=PA339&lpg=PA3… These seem to be mostly RADAR issues, I can't imagine how difficult it would be to program a missile to intercept another missile, there's hundreds of variables involved. I'd be willing to bet that these days they can run them through a computer simulator and detect most of these issues. Unrelated to the missile but relating to the original topic: http://www.wired.com/2012/11/google-spanner-time/

"Toyota’s developers used MISRA-C and the OSEK operating system, both good choices for a safety-critical real-time system. But then they ignored, sidestepped, or circumvented many of the very safety features they are designed to enforce." "On a cyclomatic-complexity scale, a rating of 10 is considered workable code, with 15 being the upper limit for some exceptional cases. Toyota’s code had dozens upon dozens of functions that rated higher than 50. Tellingly, the throttle-angle sensor function scored more than 100, making it completely and utterly untestable." http://www.eejournal.com/archives/articles/20131127-toyota/ Michael Barr was the engineer that was able to sit down with the Toyota engine code during the "unintended acceleration" lawsuit and determine what NASA could not: that the code was a wreck: http://embeddedgurus.com/barr-code/2013/10/an-update-on-toyota-and-uni… Like I said. Bubble gum and duct tape.