Normal accidents
June 3, 2015 — June 3, 2015
On the certainty of disaster.
Cars driving on a wheel and a prayer:
But on the whole, the safety architecture is a house of cards. It is possible for a large percentage of the failsafes to be disabled at the same time that the throttle control is lost. […]
Koopman was highly critical of Toyota’s computer engineering process. The accepted, albeit voluntary, industry coding standards were first set by Motor Industry Software Reliability Association (MISRA) in 1995. […] Toyota substituted its own process, which had little overlap with the industry standard. Even so, Toyota’s programmers often broke their own rules And they failed to keep adequate track of their departures from those rules — and the justification for doing so, which is also standard practice. […]
Barr testified to some of the vehicle behavior malfunctions caused by the death of tasks within the CPU, and concluded that [the accident] was more likely than not caused by the death of a redacted-name task, called Task X at trial. Barr dubbed it “the kitchen-sink” task, because it controlled a lot of the vehicle’s functions, including throttle control; the cruise control — turning it on, maintain the speed and turning it off — and many of the failsafes on the main CPU.
He was critical of Toyota watchdog supervisor — software to detect the death of a task — design. He testified that Toyota’s watchdog supervisor “is incapable of ever detecting the death of a major task. That’s its whole job. It doesn’t do it. It’s not designed to do it.”
Cool.
See also Eliezer Yudkowsky, Moore’s Law of Mad Science
Every eighteen months, the minimum IQ necessary to destroy the world drops by one point.