Jeremiah 29:11

I thought you guys might find this article interesting.

Monday, March 1, 2010

Toyota learns the tyranny of software complexity

Mar 1 2010 9:03AM | Permalink |Comments (22) |


As a former auto engineer, I feel sorry for Toyota, I really do. To me it seems like their primary error was just believing their stuff has to be superior and can never break. Unfortunately, as cars get more and more complex, horrid problems like these unintended acceleration events are sure to occur. The rocket scientists that designed the Space Shuttle thought they were infallible and hot stuff too, until the ridiculous complexity of the machine caught up to them and the inevitable disasters started occurring. Maybe Toyota did the same kind of oversimplification of analysis that NASA did.

And believe it, a modern car is a wickedly complex device. Dozens if not hundreds of microprocessors are all whirring away, and they must communicate of the CAN bus to sense and control critical vehicle functions. The government has been concentrating on the reliability of pollution control functions, since it never occurred to them a car would get so complex its safety would fail in these subtle ways. When I was an engineer at Ford Motor I would often hear engineers exclaim “That is a million to one!” when I pointed out some subtle failure mode. I would respond, “Right, and how many cars does Ford make every year?” After they hemmed and hawed and admitted we were shipping 6 million vehicles year, I would ask which 6 people they wanted to kill, and what would they say to that person’s mother and loved ones?

I do think the news media is piling on a bit, and the government is more than happy to oblige since they hold a major stake in General Motors. ABC News led the battalions of incompetent non-technical journalists acting like they had a big scoop when they accused Toyota of knowing about a “surge problem”. What idiots. Surge is slight speeding up and slowing down under constant speed—usually due to a fuel-injection or spark-retard mapping problem. It is not the same as unintended acceleration.

Of course, Toyota, just like the American car companies, cannot quite believe the alacrity with which our citizens can wrap cars around trees. There is no motor so powerful that you just can’t stomp on the brakes and bring the car to a stop. The proper procedure is to put the car in neutral and let the motor scream, bring the car to a halt and then turn off the motor. Turning of the motor while underway locks the steering wheel and now you have really got a problem.

Months before the Toyota scandal, I had the pleasure of talking to Mentor Graphics CEO Wally Rhines at a diner function. I mentioned being a former auto engineer and talk turned to the insane complexity facing vehicle engineers. Wally told me how the Airbus people were struggling with the determinism of data on the hundreds of serial communication buses in their aircraft. On top of that they had configuration management and change-control problems. Thing about how huge the task is. How to insure that a rev C module will properly work with a rev G controller and not clobber that rev B module you just added to the system? Mentor offers simulation environments and wiring tools like their Vesys package to help engineers deal with this brutal problem.

The importance of these simulation and verification tools became apparent when I was visiting a friend in Portland. We were waiting on his buddy to show up for a barbecue. We got a panicked call from the guy, a usually unflappable retired cop. His brand-new GMC pickup truck was acting really weird, with the door locks cycling and transmission display going crazy and the instrument panel flashing and acting broken. We all agreed he should park the truck and we should pick him up. Subsequent phone calls revealed he had just picked up the truck from the dealer where he had a failed rain sensor replaced. This sensor is a whiz-bang feature that turns on your wipers when it rains. I think stuff like this is a waste for the exact reason for what happened. We told the guy to turn off the switch for the automatic rain sensing. The problem went away and he drove the truck the rest of the way. Now what kind of goofy CAN bus disaster could make a bad module trigger door locks and many other functions is beyond me. I have a hard time believing that some noise on the line could case so many false actuations. Let us hope that GM kept the windshield wiper CAN buss away from the ABS brake or engine control CAN-bus, but what kind of buffoons could design any system that has such an egregious failure mode? Maybe GM should stick to bending tin and let us electrical engineers design the electronic systems, the mechanical engineers that are lapdogs to the MBA finance types who are destroying the US auto industry are doing a pretty poor job.

The system level error that Toyota made is not letting a brake signal override a throttle signal. I designed speed control systems at Ford, and everything was dependent on having a tap on the brake cancel any speed control function. A throttle-by-wire car like Toyota makes is almost free to add speed control, you just have to have a button to tell the ECU (engine control module) to hold speed and a brake signal, and that is probably already sent to the module. So it was just software, a couple lines of typing, that means that once a car accelerates a brake input will send the throttle angle to zero. I have to assume that Toyota engineers talked themselves into thinking there are times when you want to hit the throttle and brake at the same time. Motorcycle racers do this to put torque loads on the frame so when they do let off the brake coming out of a corner, the bike is already “bent” by the chain loads and then handles more predictably. I can’t think of a reason a car needs to have brake and throttle on at the same time, but somebody must have dreamed it some sports-car-dork reason to not have the brake single cancel the throttle signal.

I don’t want to let Toyota off the hook, it does look like they could not bring themselves to admit they had a problem and then covered it up once they did. We can forgive them for not being perfect, but we have a harder time forgiving sleazy wickedness.

You might want to put this Toyota recall into context however. Look at the Ford Explorer rollover problems. A tire blowout would cause the vehicle to spin out of control and overturn. Car and Driver magazine could not replicate the problems on a test track; their drivers were always able to bring the car to a safe stop. But the reality of the situation is that an amateur driver not expecting a blowout would jerk the wheel and the SUV would end up flipping. According to my auto industry pals the reasons were straightforward. One, the roll center of the Explorer was too high, making a rollover possible. Two, Ford under-specified and under-inflated the tires to give that Ford boulevard ride they have sought for 4 mush-filled decades. OK, you want your truck to ride like limo? Fine, you don’t do that by under inflating the tires. And if you do, you put high-temp rated tires on the vehicle so they don’t fail prematurely. And since when is a tire failure supposed to trigger a rollover? These are not military Jeeps, these are million-selling passenger vehicles. The answer is what Ford is all about and why I quit. Cheap cheap cheap. Putting in high-temp tires costs money. Using mushy shocks means you need better sway bars and that costs money. Designing the roll center to be lower costs money. So for some filthy lucre and that boulevard ride, Ford killed dozens of people, many of them completely innocent people who where hit by that overturned Explorer. Now I don’t know about your moral compass, but I think Ford has a lot more culpability over this than Toyota does for any unintended acceleration.

Want another Ford recall? The ignition switch would catch fire. The cause for one of these recalls was that the Ford told the vendor, I believe it was ITT, to lower the height of a creepage fin between the switch terminals. The plastic was not 3 cents a pound polyethylene, like Ford likes to make every component out of. I think the plastic was polyamide or an expensive engineering thermoplastic that cost 80 cents a pound. So to save maybe a tenth of a cent on the cost of the switch, they cost-reduced a working design into something that killed the baby of the mother who trusted enough to leave her child in the car while she took the groceries into the house. If I remember right, the vehicle was not even running.

Ford had a design philosophy of starting with something so cheap and crappy it would never work, and then they added pennies until the thing seemed to work. GM would tend to design the thing to work solidly and then take the cost out. Unfortunately they often took too much cost out and were as bad as Ford. All this is 30 years ago, so I don’t know if Ford and GM are still so incompetent, but the linked website above that accuses MBAs of trying to design cars at Ford was sure right. I had to spend days explaining to those Ford MBAs why we could not make the wiper motor out of plastic or use aluminum battery cables instead of copper.

I can tell you this; Toyota still builds the most reliable car on earth. When I was at Ford in 1980, our goal, which we thought was impossible, was to be only twice as bad as a Toyota mini-pickup. I drive Honda since I think Soichiro is a real engineering hero, but if I ever give up on my Accord, I might buy a Nissan, since my buddy Bruce works there, or a Toyota for the best reliability. I would no sooner drive a Ford than I would play Hacky Sack with nitroglycerine.

tdub2112

I think that that last paragraph is the problem with american manufacturers today. Chrysler, GM, and somewhat Ford just don't get it. You don't need a car to fit everyones liking. You need a car that functions well, has High dependability and gets good mileage. Not a flashy, car that has aligator leather heated seats and a TV screen in every back rest. We need to get back to basics with our cars.

MaybeChallenger

I think NASA predicted that 1 in 50 shuttle missions would be lost.