Vista Validation Woes Caused by Preproduction Code

Comments

By Peter Galli Human error was responsible for the Windows Genuine Advantage system shutdown that denied validation to Vista systems, Microsoft says.
The system shutdown that prevented Microsoft from validating some 12,000 Vista systems under its Windows Genuine Advantage program was the result of preproduction code being sent to production servers, the company said. While activations, which were also affected by the preproduction code, were fixed in less than 30 minutes by rolling back the changes, the rollback did not solve the problem with the validation service, which was down for some 20 hours. Microsoft, based in Redmond, Wash., confirmed Aug, 27 that the problems with processing validations started at about 3:30 p.m. PDT on Friday, Aug. 24, and continued until around 11:15 a.m. Pacific Time on Saturday. "Nothing more than human error started it all. Preproduction code was sent to production servers. The production servers had not yet been upgraded with a recent change to enable stronger encryption/decryption of product keys during the activation and validation processes," Alex Kochis, senior product manager for Microsoft's Windows Genuine Advantage team, said in a blog post on Aug. 28. "The result of this is that the production servers declined activation and validation requests that should have passed," he said. While Microsoft expected the rollback to fix both the activation and validation issues at the same time, "We now realize that we didn't have the right monitoring in place to be sure the fixes had the intended effect," he said. With regard to why its validation system does not assume that systems are genuine when its servers are down, Kochis said that is the default. "It's important to clarify that this event was not an outage. Our system is designed to default to genuine if the service is disrupted or unavailable," he said. Microsoft designed Windows Genuine Advantage to give customers the benefit of the doubt, he said. "But this event was not the same as an outage because in this case the trusted source of validations itself responded incorrectly," Kochis said. Microsoft is improving its monitoring capabilities so that it will be alerted more quickly should something like this happen again. Even though the validation system stopped working at about 3:30 p.m., Microsoft only discovered the issue that evening as a result of calls to its customer service lines and posts to its forum. The software maker is also making other changes, such as increasing the speed of escalations and adding checkpoints before changes can be made to production servers.

Adding to Microsoft's woes and those of its customers was the fact that customer support agents incorrectly told people whose Vista systems were not being validated that the servers might be down until Tuesday, Aug. 28, when, in fact, they had already been fixed as of late morning Pacific Time on Aug. 25.

"We're reviewing our procedures on that score as well—communicating clearly and accurately are super-important when things like this happen," Kochis said.

Without validated systems, customers are not able to access a number of Vista's features. These include the new Aero user interface; ReadyBoost, which expands virtual memory; Windows Defender, which still scans and identifies all threats but cleans only the severe ones; and optional updates via Windows Update, which still makes security and other critical updates available.

Adding insult to injury for customers unable to validate their systems was the message that appeared in the lower right-hand corner of the desktop area, saying "This copy of Windows is not genuine." The message remains until a successful validation is completed.

One customer angrily posted to the Windows Genuine Advantage blog, "I paid well in excess of $200 for my copy of Vista Ultimate and getting told that 'Your copy of Vista appears to be counterfeit' and having functionality removed and not being able to install updates from the Microsoft site doesn't make for a very good experience.

"I was and still am quite mad at Microsoft. I'm pretty sure I'm going to switch to a Mac because of it. I certainly don't feel like I've been rewarded for being a good citizen and actually buying my copy of Windows. All those pirates out there probably had no problems at all," the post said.

For his part, Kochis once again expressed his regret about the situation. "I also want everyone to know that I am personally very disappointed that this event occurred. As an organization we've come a long way since this program began and it's difficult knowing that this event confused, inconvenienced and upset our customers," he said.

But one customer lauded Kochis for being open and upfront about the problem in an MSDN blog.

"I know that takes a lot of guts to do, even though once it's done, it seemed so simple," the customer wrote. "I highly encourage more transparency in both the WGA and Activation efforts at Microsoft. I'm pretty sure it will do you guys a lot of good in the long run. There are still many people with grave concerns as I'm sure you know!"