Benchmarks are Wiggedy Wiggedy Whack...

I wrote this article for the latest edition of CPU Magazine. Remember, if you haven't had the opportunity to read this magazine you should go pick up a copy soon.

There has been a lot of speculation about AMD’s new Phenom processor, and up until now the chip’s performance benchmark scores vs. the competition’s were predicted to be somewhat underwhelming. I’d like to address that this month, as well as the validity of benchmarking as it pertains to the customer experience in general.

I like performance benchmarks, but I’m starting to think people are putting too much weight on them and not enough on the overall customer experience. This is roughly as silly as directly equating one’s physical strength to how much he can bench, or comparing drag racing to Formula One racing—there are some profound differences here.

When you really boil it down, no one cares about benchmarks when they are playing a game, churning out a report, or reading a medical image. Let’s face it, on the high end 300 points in 3DMark06 isn’t going to affect gameplay even at the highest resolution with maximum detail. Gamers care that their PCs display the best possible image while delivering the most compelling and most stable gaming experience. Users care that when they turn on their PC it boots reasonably quickly and works with all of their devices. If one of their components fails, they are looking for a simple way of replacing the component with the best access to customer support. Thus there are undeniable differences between overall experience and performance benchmarks. I like to refer to this internally as the “experience benchmark,” and it’s a tough one to measure because it’s so subjective.

Yes, there was a point when gamers thought the only things that mattered were frame rates, but I think that the tide is shifting. Ask any pro gamer to compare two similarly equipped PCs side by side running same game at the same resolution, and I guarantee you that he will have no clue which delivers the highest frame rate. He’ll most likely be inclined to choose the one that he thinks delivers the best image quality.

These changes are occurring because current hardware delivers incredibly high levels of performance and the software has not quite caught up yet, although some games are starting to deliver, like MS Flight Simulator X, Company of Heroes, and upcoming titles such as Hellgate London. Vista introduced new levels of complexity and experience to gamers and enthusiasts alike. Overall the initial experience of Vista sucked, but it has turned a corner and things are beginning to look up thanks to DirectX 10.

So again, I believe that performance benchmarks are not the true measure of a compelling experience. There are many more factors to consider when building and/or buying your next PC. You’ll want to consider operating noise, image quality, ease of access, ease of upgradeability, ease of replacing components, how “quick” it feels when you’re booting it up, storage space, stability, operating system usability, style and design, and, if you are a gamer, how well it delivers in the area of gameplay.

So, AMD decided to unveil Phenom running at 3.0GHz without showing actual benchmarks. What it showed was a game running smoothly with all details enabled, which makes perfect sense to me. And for the record, if you were to benchmark Phenom at 3GHz you would see that it kicks the living crap out of any current AMD or Intel processor—it is a stone cold killer (at 3GHz, now imagine how it would perform if they could squeek some more juice out of it?).

I’m guessing that AMD will be able to launch some parts at higher clocks than it is currently showing in its roadmaps, and if the company can get these chips on shelves in a timely fashion, I think it could be a major coup and could even be the impetus for the turnaround the company so desperately needs. Of course, Intel probably won’t get caught flat-footed, but AMD has to start somewhere.

It’s interesting to note that AMD isn’t showing benchmarks on a part that delivers the goods—perhaps it too is seeing that performance benchmarks are only a small piece of the overall experience puzzle. That said, I suspect there will be some more shaking up at AMD before the sun starts to shine green again.