Toyota unintended acceleration and the big bowl of 'spaghetti' code (2013)
Source: Hacker News
Background
Last month, Toyota settled an unintended‑acceleration (UA) lawsuit just hours after an Oklahoma jury found the automaker acted with “reckless disregard” and returned a $3 million verdict for the plaintiffs. The jury had not yet ruled on punitive damages.
The jury’s decision was heavily influenced by testimony from two plaintiff experts in software design, who examined Toyota’s engineering process and the source code of a 2005 Toyota Camry. Both concluded that the system was defective, dangerous, and riddled with bugs and gaps in its failsafes.
The Bookout v. Toyota case
The case stemmed from a September 2007 UA event that caused a fatal crash. Jean Bookout and her passenger Barbara Schwarz were exiting Interstate 69 in Oklahoma when the 2005 Camry lost throttle control.
- The service brakes failed to stop the speeding sedan.
- Bookout applied the parking brake, leaving a 150‑foot skid mark from the right rear tire and a 25‑foot skid mark from the left.
- The vehicle continued down the ramp, crossed the road, and crashed into an embankment.
Schwarz died; Bookout spent five months recovering from head and back injuries.
Attorney Graham Esdale (Beasley Allen) noted that the black skid marks were a key piece of evidence:
“Toyota just couldn’t explain those away. The skid marks showed that she was braking.”
After learning of the settlement, jurors asked Judge Patricia Parrish if they could stay and discuss the trial. A dozen jurors, the judge, and the plaintiffs’ lawyers met, and the conversation made it clear the jury was poised to punish Toyota for its conduct and alleged cover‑up.
Expert testimony
Two software experts—Phillip Koopman and Michael Barr—provided detailed insights into Toyota’s software development process and source code.
- Michael Barr, an embedded‑software specialist, spent more than 20 months reviewing Toyota’s code in a secure, guarded environment. He produced an 800‑page report and testified about specific code defects.
- Phillip Koopman, a Carnegie Mellon professor in computer engineering and author of Better Embedded System Software, testified about Toyota’s engineering safety process.
Both described the code as “spaghetti code”—badly written and poorly structured.
Barr’s observations
“There are a large number of functions that are overly complex. By standard industry metrics some of them are untestable… Some are unmaintainable, meaning fixing a bug is likely to create a new one. … The safety architecture is a house of cards. It is possible for a large percentage of the failsafes to be disabled at the same time that the throttle control is lost.”
Barr also cited an internal Toyota document in which a programmer called the engine‑control application “spaghetti‑like.”
Koopman’s criticisms
Koopman highlighted Toyota’s failure to follow the MISRA (Motor Industry Software Reliability Association) coding standards, first published in 1995. He explained that each rule violation statistically introduces bugs: roughly three minor bugs and one major bug per 30 violations.
- NASA engineers, reviewing parts of Toyota’s code for NHTSA in 2010, found 7,134 MISRA‑C violations in the sections they could access.
- Barr’s own check against the 2004 MISRA edition uncovered 81,514 violations.
Toyota used its own process, which overlapped little with industry standards, and often broke its own rules without adequate documentation of the departures.
“If safety is not baked into the recipe in the process of creating the product, it cannot be added later.” – Koopman
Major software deficiencies identified
| Deficiency | Description |
|---|---|
| Complex, untestable functions | Functions so intricate that reliable test suites are impossible; fixing bugs risks introducing new ones. |
| Single‑point failures | Critical hardware/software components that, if they fail, render the entire safety system ineffective. |
| Excessive global variables | Over 10,000 global variables (industry best practice: zero). |
| Lack of peer code review | No systematic review process for source code. |
| Unchecked secondary CPU code | Toyota did not review code from the second CPU supplied by Denso, despite assurances to Congress and NHTSA that engine software could not be the cause. |
| Task deaths | Unexpected termination of tasks within the CPU, disabling failsafes (e.g., the “Task X” or “kitchen‑sink” task that controlled throttle and cruise control). |
| Inadequate failsafe design | Failsafes contained defects and gaps; the overall safety architecture was fragile. |
Single‑point failure example
“If there is a single point of failure, by every safety standard I have ever seen, it is by definition unsafe, and no amount of countermeasures, no amount of failsafes will fix that.” – Koopman
Global variable overload
“And in practice, five, ten, okay, fine. 10,000, no, we’re done. It is not safe, and I don’t need to see all 10,000 global variables to know that that is a problem.” – Koopman
Conclusion
The testimony of Koopman and Barr revealed a cascade of software‑engineering failures in Toyota’s 2005 Camry: non‑compliance with established coding standards, excessive complexity, single‑point failures, and a lack of rigorous safety processes. These deficiencies likely contributed to the unintended‑acceleration event that resulted in a fatal crash and a $3 million jury verdict. The case underscores the critical importance of adhering to industry‑standard software practices—especially in safety‑critical automotive systems.