I've written previously comparing ATI to NVIDIA, but that article was written purely from the perspective of a gamer concerned with performance. Today I have to write again from a different perspective, namely, from that of a philosopher disgusted with NVIDIA's unethical, and potentially illegal, behavior. The problem, speaking generally for a moment, is pretty simple: hardware vendors sometimes "cook the books". That is, they sometimes deliberately structure their hardware and software so that the scores obtained from industry-standard benchmarking programs do not reflect real-world product performance. The point, of course, is to convince the customer that the product is more capable than it really is.
There is a line between good marketing and unethical, and potentially illegal, behavior. The former is a matter of highlighting a product's strengths, while glossing over any weaknesses it might have. The latter is a matter of outright lying about the product and what it can do. What NVIDIA has done, of late, falls clearly into the latter camp, and they should be punished for their behavior, if not by the courts, then at least by consumers. The purpose of this essay is to explain the situation at hand, so that others might be convinced to help bring NVIDIA to justice.
As former President Clinton couldn't whine fast enough when caught with his pants down, "everybody does it!" Sadly, this is true, more or less, when you're dealing with human persons. To borrow from the Bible, no temptation has seized NVIDIA except what is common to man, or, if you prefer, there is no one righteous, not even one. In other words, the shoe was on the other foot not so long ago. Others may not remember the Quake/Quack benchmarking flap with ATI some years ago, but I do. That whole sorry business came about when it was discovered that merely by renaming the Quake III Arena (Q3A) executable from "Quake.exe" to "Quack.exe", the benchmark scores turned in with ATI's Radeon 8500 card changed significantly. Those interested in that saga can read the relevant articles at The Tech Report or FiringSquad web sites.
At that time, ATI was chastised—and rightly so—for rigging the game, for cheating, in effect, with their drivers. What they did, in brief, was rig their drivers to look for the Q3A executable running, and, when it was detected, change the way their drivers rendered the images, dropping the overall quality in order to run faster, thus making their video card look more capable than it really was. To be clear, this is not just "good marketing". The reason is that good marketing doesn't stray into deliberate deception and false statements, which is exactly what such fiddling about is, at its core. What it amounted to was false advertising, which is a crime for good reason. Prior to buying a product, the customer is almost completely dependent upon advertising materials, reviews, and so forth to make his decision. It is because of that dependence that such shifty tactics are illegal. Yes, caveat emptor still reigns, but no amount of buyer wariness will help when the whole game is rigged.
To their credit, ATI has come a long way since then. Yes, they too have been brought into the current scandal, but their hands are clean, though explaining why that is the case requires that we consider what separates optimization from cheating. For any given frame of any specified scene that a graphics card is supposed to render, there is only one correct result. That one correct result is the set of all the pixels to be rendered, which are defined ultimately by their red, green, and blue (RGB) values. There may be any number of different algorithms that are employed in rendering that scene, mind you, for contemporary graphics cards can do all sorts of nifty things. Further, full-scene anti-aliasing (FSAA), anisotropic filtering (AF), and other such features can be implemented in different ways, the differences lying typically in the delicate balancing act between image quality and rendering speed.
But none of this changes the fact that there is, nevertheless, one and only one correct result for any given frame of any specified scene. Whatever algorithm the manufacturer has decided upon for FSAA, for example, will have exactly one determinate resulting output, given all the relevant inputs. The same is true of any other such feature in video cards; i.e., to the best of my knowledge, all such features, on today's video cards at least, are wholly deterministic in their algorithms and do not involve any kind of randomization. This is why it is possible to say that there is one and only one correct result for any given frame of any specified scene. For those more mathematically inclined, one might say instead that there is a one-to-one mapping between the input domain and the output range.
With this in mind, it is possible to distinguish optimization from cheating. Anything that the vendor does to speed up the process, without deviating from the one correct result in any way whatsoever, is an optimization. Anything that the vendor does to speed up the process, which unavoidably changes the result, no matter how subtly, is cheating. Again for those more mathematically inclined, the function that provides the one-to-one mapping is irrelevant, as long as the proper mapping is still performed. In other words, optimization involves trying to get the correct result as quickly as possible, while cheating involves trying to get an incorrect result that's "close enough" to fool any observers. What ATI did in the whole Quake/Quack business was clearly cheating. What NVIDIA has done more recently is also clearly cheating. ATI's role in the current controversy is entirely benign—at least at present—as we will soon see.
Today's mess has been brewing for some time. It arguably began when NVIDIA disagreed with FutureMark over how their next generation of benchmarking software, 3DMark 2003, should be built. Since plenty of detail is available elsewhere, suffice it to say that there was a falling out between NVIDIA and FutureMark. NVIDIA found themselves, for the first time in quite a while, in the unenviable position of playing catch-up with the market leader, ATI. Worse, NVIDIA's new flagship offering was turning out to be far less robust than was previously hoped, and, as such, its benchmarking scores were pretty underwhelming. No doubt all of these factors contributed to the desire to cheat.
To cut right to the chase, NVIDIA tried to "fix" the performance issues with a new release of their drivers, Detonator FX. Unfortunately, those drivers were caught cheating, as detailed by an article at ExtremeTech. FutureMark subsequently investigated the matter and published their own white paper on the subject, explaining how NVIDIA's drivers were indeed cheating. At this same time, ATI's performance numbers were called into question as well. In their case, however, the correct results were being obtained; the differences in performance were the result of legitimate optimizations, namely, a re-shuffling of certain instructions, as reported elsewhere. Despite their role in the current controversy being entirely benign, thus far, ATI has publicly stated that they will remove that optimization. This seems a genuine effort to avoid even the appearance of impropriety.
It was but a couple of days ago, presumably after various ranks of lawyers fired a few volleys back and forth, that NVIDIA and FutureMark released a joint statement. In that statement, FutureMark backed away from their prior claim, explaining now that they have a "deeper understanding" of the situation, and that, in light of that "deeper understanding", NVIDIA's driver design is not a cheat, but rather an "application specific optimization". It is probably safe to say that this change of position came as a bit of a surprise to many onlookers. After all, there is only one correct result for any given frame; NVIDIA's drivers either produced that result, or they did not. We're not rolling dice here, people, we're calculating RGB values. Do you really want your video card drivers "deciding" when they should start cutting corners to make the video card seem faster than it is?
As if the situation weren't bad enough already, The Tech Report today takes the whole mess to the next level, explaining how NVIDIA has apparently taken other liberties that weren't caught during FutureMark's audit. To be more specific, their testing shows pretty clearly that when 8x anisotropic filtering is enabled—a setting many hardware review sites enable for benchmarking video cards these days—the NVIDIA drivers sacrifice image quality for a non-trivial performance boost. With a proper sense of historic irony, the testers verified this by renaming the 3DMark 2003 executable from "3DMark03.exe" to "3DMurk03.exe". Apparently, NVIDIA's Detonator FX drivers detect when any executable named "3DMark03.exe" is running and take steps to improve performance at the expense of image quality, at least when certain options are enabled. The article cited stops short of saying that NVIDIA is cheating, leaving the reader free to draw his own conclusions, but this strikes me as nothing more than moral cowardice. It's pretty obvious that NVIDIA is cheating—well, unless you're going to go all wobbly and grant that it's just another "application specific optimization".
Here's the bottom line, folks: NVIDIA has just been caught cheating, and all this talk of "application specific optimization" is absolutely shameful nonsense. A video card is a piece of hardware, nothing more. It requires drivers in order to interface with and be used by whatever operating system the user has installed on his computer. The job of those drivers is to receive a "description" of what needs to be rendered at a relatively "high level", then pass the proper commands and data to the video card so that it can do its whiz-bang super-fast stuff internally to arrive at a set of calculated RGB values for all of the visible pixels. Though that description is a horrible oversimplification, it's essentially how the process is supposed to work. The whole point of having a video driver stand between the operating system and the hardware is so all of the applications have a single, common point of access; i.e., the very reason for having a standard video driver interface is to prevent application specific nonsense!
Now, it doesn't matter to the user, or it shouldn't anyway, whether the driver code or the video card hardware resequences instructions, makes changes in the data precision, etc. in order to speed up the process, as long as the final output remains correct. If anything, vendors should be commended for adding code and/or hardware to speed up the rendering process while still arriving at a correct output. That's attention to detail. That's working hard to give the customer a better product. That's worthy of praise. As long as the correct frames end up being displayed on the user's monitor, it doesn't matter what the hardware or software is doing "under the hood". As long as the functional characteristics of the system remain constant (i.e., the same outputs for the same inputs), the vendor is free to improvise as he will.
But where does any vendor get off cutting corners with the final output?! If this doesn't piss you off, NVIDIA users, it should. Ask yourself this question: at what point did you agree that NVIDIA could change the images from some of your games and applications? When, precisely, did you cede control of the operating parameters of your system to the company that makes your video card? Granted, the differences are so subtle that they're difficult to spot—even in a side-by-side comparison—but that's not the point. The point is that NVIDIA is deliberately attempting to deceive reviewers and customers about its products by changing the way their hardware and software responds, based on what it "thinks" its running! What happens when your video drivers "decide" not to bother handling a game's shader properly because they "confuse" it with one of the shaders in 3DMark 2003? What happens when your video drivers "decide" to drop the image quality a few notches because the name of the game you're running happens to have the same name as some other, popular game? This is simply unacceptable.
Again, the differences are pretty subtle, so many will probably be inclined to say "so what?", just as many were inclined to say "so what?" when the aforementioned letch of a President was caught with his pants down. To those people I can say only this: if you let NVIDIA off the hook this time, then you are essentially forfeiting your right to expect your hardware or software to do what the vendor says it does. Seriously, there is no difference in principle between NVIDIA deliberately fudging benchmarks, and some other company selling you a piece of software to do X when it doesn't do X at all. The difference of deception is only of magnitude, not kind. If you won't uphold principle in the former case, then you have little justification for upholding it in the latter.
For those who still aren't convinced, think of it another way. When you're buying a particular game, for example, you're paying for a certain experience, one that you (hopefully) will enjoy. The game designer has assembled certain algorithms and data intended to convey that experience within certain parameters. When the game looks crappy on your system because some video driver "decided" to cut corners, is this a good thing? What about all the pain and suffering the user and developers endure when a game starts crashing left and right because there's a bug in an "application specific optimization" in the video drivers? Is that a good thing? Or what if, in a nod to concerns about violence, "family friendly" video drivers eventually start censoring game output? That last example is a bit of a stretch, of course, but again it differs only in magnitude, not kind, from what NVIDIA has done. The fundamental problem here is that NVIDIA is taking liberties in how their cards operate, violating the consumer's trust and specified settings, solely so that their cards look more powerful than they are.
To be sure, FutureMark's benchmarking software plays a big role in this whole mess. It wasn't that long ago that reviewers focused solely upon pseudo real-world performance in contemporary games. I say "pseudo" because many reviewers (e.g., Anandtech) often take their benchmark data without sound cards in their systems, which clearly isn't realistic in light of how most users' systems are built. In the last few years, however, FutureMark's 3DMark series of applications have become the industry standard. For better or worse, users seem to like being able to sum up the results of a synthetic benchmark with a single number. It makes it easier keeping up with the Jones family, after all, if you know exactly what number you need to beat to do so. Like it or not, folks, that's just human nature.
This current situation provides another opportunity, however, for game developers, hardware vendors, benchmark vendors, and so forth to reassess exactly what sorts of benchmarks are most meaningful to the consumer. Think about it: do you really buy a certain video card because you want to get a better score from 3DMark 2003? No doubt some people do, but that's just silly. The reason most consumers, and more precisely most gamers, buy the latest and greatest hardware is because of the way it runs their games and applications. When I'm thinking about buying a new card, I'm asking myself two questions: (1) Will it run better than my current card?, and (2) Can I afford it? In light of this, one might wonder: why is a purely synthetic benchmark so heavily relied upon? Some (e.g., the folks over at HardOCP) will even go so far as to argue that synthetic benchmarks are the wrong tools for the job.
But such claims simply do not survive scrutiny. In fact, if anything can be concluded from this mess, it's that both real-world and synthetic benchmarks are useful for different reasons. Were 3DMark 2003 ignored altogether as a merely synthetic benchmark, then vendors would simply tailor their cheats for real-world games. It's obviously happened in the past (e.g., Quake/Quack), and it would surely happen again without 3DMark 2003 on the scene. It's just silly to lay the blame for cheating on the wide acceptance of a synthetic benchmark. True, the very existence of synthetic benchmarks do force vendors to pay attention to how well their hardware works with those benchmarks, rather than with any specific game, but why is this a bad thing?
No one game or handful of games exercises every feature of a given video card in the same way. Anyone who's played more than a handful of games can attest to this, as there are often bizarre and obscure incompatibility issues with various drivers and hardware. Believe me, I've written dozens of essays in light of such problems. A synthetic benchmark, designed from the ground up to exercise all of a given graphics system's functions thoroughly, can expose a host of issues that shipping games do not. That's not to say that synthetic benchmarks are perfectly comprehensive, of course, but it's unavoidable that benchmark authors approach their job far differently from game developers, and therein lies the value.
Further, since shipping games frequently lag significantly behind the available hardware features—not one game on the market today takes full advantage of DirectX v9.0a, for example—synthetic benchmarks give consumers important data that real-world games do not. That is, they give the consumer a rough idea of how well a given video card will "hold up" over time, of how well it will play the next generation of games in addition to those sitting on store shelves at present. In short, both synthetic and real-world benchmarks have their uses, and this is unlikely to change, despite any bluster to the contrary ensuing from the current mess.
I wrote previously that I "couldn't care less" about whether ATI or NVIDIA had the fastest hardware. I said that I would buy whatever card gives me the best bang for my buck, considering all the relevant factors, regardless of who makes it. I'm not a "fanboy" for either ATI or NVIDIA. As a result of NVIDIA's deception, my overall stance hasn't changed; i.e., I'm still not a "fanboy", and I'll still try to buy the card that gives me the best bang for the buck. What has changed, however, is that I can no longer trust NVIDIA. When they come out with their next "ATI killer" card in a few months, why should I believe that any of the benchmark numbers I see are correct? Why should I believe that their drivers aren't "cooking the books" in their favor? Why should I risk my money on a company that's obviously more interested in making cards that are perceived to be good than cards that really are good?
Those questions alone will prevent me from taking NVIDIA seriously for a while, and I hope that my readers will do the same. As consumers in the hardware and software markets, we have so little power when it comes to defective products. The only might that we can wield is that provided by how we spend our hard-earned cash. I hope that NVIDIA takes it on the chin from this whole episode, and I hope all the more that they put their own house in order as a result of it. I would much rather see NVIDIA stick around and compete fairly against ATI, rather than trying to seize the performance crown by such disingenuous, unethical, and potentially illegal, means.
Your one option to redeem yourself, NVIDIA, is for you to come clean. If you admit your cheating and promise to (1) remove it from the current drivers, and (2) not do it again, consumers can respect that. If you continue on as you have thus far, however, trying to avoid the charge of cheating altogether by calling it "application specific optimizations", then you'll have lost this customer and plenty of others as well. ATI handled the Quake/Quack debacle relatively well. It forced them to focus on making great hardware and software, which is why their products are presently kicking NVIDIA's nether regions. Learn from their example, NVIDIA, and get back to the basics. You remember the basics, right? In case you've forgotten, that means making great hardware and software, not adding cheats to your drivers.
06/05/2003