How can a software company, with one small mistake, cripple computers worldwide?
McAfee security users left the office on Tuesday night with a perfectly good Windows XP computer. On Wednesday morning, they were staring at a useless pile of plastic and computer chips. Without so much as the stroke of an enter key or the push of a mouse, their PCs had been changed. The error was simple: McAfee’s software erroneously decided that an essential file used by the Windows operating system was really an 18-month old Trojan horse. That sent many PCs into an infinite re-booting loop that couldn’t be stopped without skilled, manual intervention.
The root of the problem lies in a critical decision made a decade ago by security professionals. But the result — perhaps millions of PCs rendered useless, each one requiring manual repair — is just the latest sign that bad guys seem to be winning in cyberspace.
Back before the turn of the new millennium, the computer world was wrestling with a dramatic new concept — giving large software firms like Microsoft the right to access consumers' computers and update their software automatically. To many purists, who might be considered technological libertarians, the idea of letting an outside company reach inside their hard drive and change things was pure lunacy.
A five-year-long onslaught of global virus outbreaks quickly changed hearts and minds. Consumer behavior consistently revealed that the vast majority of PC users wouldn't bother manually installing software patches and antivirus protection. That made them easy prey for Code Red, The Love Bug, Nimda, and dozens of other malicious programs. And as the rate of new malicious programs began to grow exponentially, it became physically impossible for even full-time experts to manually update their systems.
Today, most computer users don't think twice about letting Microsoft update their machines, or about downloading patches or new protection files from security firms like McAfee, Symantec, Kaspersky, and others. The system has worked; it's been 10 years since an outbreak like the Love Bug. But Thursday's Mcafee disaster – which affected corporate and not consumer users — brought into focus the downside of giving control to an outside software firm. Doctors, students, and office workers worldwide were left disconnected from the outside world, feeling very much the way they used to when a virus outbreak crippled their organizations.
"Automatic updates are still kind of a thorny issue, but when you look at threats today and the number of products that can be impacted, automatic updates are really the only viable means to insure someone's system is kept up to date," said Mary Landesman, senior security researcher at ScanSafe. “But they still carry the same risk they always did. If something is wrong with the update it's going to impact a great number of users.”
At the same time, security firms are dealing with the overwhelming capacity of malicious programmers to churn out devious new code.
Ten years ago, according to Symantec Corp, there were 10 to 15 new computer viruses each week. Today, the firm must find ways to protect customers against up to 20,000 new software threats every day. While defusing all those bombs, the possibility of accidentally disabling a good file — a so-called false positive — is increasingly likely.
McAfee’s problem was created by just such a false positive. While the firm has yet to release additional details about the breakdown, it's hard to imagine the breakneck speed didn't play a major factor.
"It is an issue every malware company has to deal with, we have to adjudicate every file on someone’s system and decide if it’s good or bad," said Gerry Eagan, a security expect at Symantec. "In this game of cat and mouse, as we try to be more aggressive and catching malicious programs…if we fail, something like this can occur."
Eagan said the file that McAfee erroneously identified was a relatively old threat. He believes that McAfee was expanding a virus definition to cover a new variant of that old threat– a normal practice that helps the software run more efficiently – and that McAfee engineers probably “wrote too generic a definition and caught a good file.”
But Landesman said she thinks the flaw was related to McAfee’s ability to detect viruses based on how they behave – by observing keylogging activity, for example — rather than the old-fashioned black-list method of identifying a known piece of computer code inside a malicious program. Behavioral protection is a bit less scientific, she said, and more prone to false positives.
“As we go to behavioral methods were going to see an increase,” she said. “There has to be a certain acceptance that we're going to have them. But there will be some really ugly incidents like this one.”
Even those who weren't hit by the McAfee bug might have had other run-ins with automatic updates. Microsoft's Windows can be very insistent about installing updates — again, a sound security practice — but it often leads to unintentional system restarts and lost files. It's not uncommon that third-party software updates cause system instability. In fact, in 2007, thousands of users of Symantec's Norton antivirus software reported persistent crashes after that company issued an update.
So is it time to reconsider the practice of surrendering control of your PC to large software company?
"I've been reading forums and there are a number of people chiming in saying, 'This is why I don't use antivirus software,' or 'This is why I don't apply patches,'" Landesman said. "There are people saying having antivirus software is worse than having none at all. That's just not true…but it is a real risk that people on the fringe about having antivirus software will point to this as a poster child for why they shouldn't. "
Eagan said consumers and companies must make a logical decisions weighing the risks of unprotected surfing with the risks of a software goof. Simply installing the software but reject installation of updates is not an option, he said.
"If you delay your downloads, then you aren't protected,' he said. The firm added 2.7 million virus detection fingerprints to its software automatic updates last year.
Sometimes those updates come several times a day, Landesman said, and that has overwhelmed even the most heavily staffed corporate security teams. Once upon a time, firms would test all updates in a lab before releasing them to their employees. In most cases today, Landsman said, there's just no time.
"So they almost have to take a leap of faith that it will work," she said. "That's the only practical avenue."
McAfee customers who made that leap on Wednesday weren’t rewarded. Instead, a bit like a rattled offense facing an overwhelming full-court press in basketball, McAfee goofed. By overwhelming the system with volume, by forcing security firms to rush and implement imperfect technologies, by robbing companies of proper time to test, malicious software writers have gained the advantage. Even this incident, while ultimately harmless for victims (outside of lost time), created a big opening for the bad guys. Consumers affected by the bug who went to Google looking for answers last night found fake Web pages offering help that were loaded with booby traps.
“This is already an industry struggling to keep up,” she said. For some time, McAfee will struggle to restore lost faith from customers.
Become a Red Tape Chronicles Facebook fan or follow me at http://twitter.com/RedTapeChron
