Building a Better (Linux) GPU Benchmark

Why Average Frames Per Second Can Be Misleading

Timedemos make up the core backbone of our video card benchmarks. Timedemos are simply pre-recorded scenes that are re-rendered as the program plays them back. The example that we use here is the timedemo "radar" from the game Wolfenstien: Enemy Territory. Depending on the conditions of the testbed, radar might render more or less frames than another machine, but both machines will always take the same amount of time to play back. The program generally reports the number of frames rendered and we divide by the number of seconds that the timedemo lasted. This calculates our average frames per second.

Plotting more than just the average FPS is not a new concept. AMD, Intel, NVIDIA and ATI have all expressed their thoughts to us about plotting box-and-whisker or line graphs of our timedemos rather than just an average FPS. Of course, CPU manufacturers do not like timedemos at all, but that is an argument that we will leave for a different day. Let us claim during our timedemo that we record the instantaneous frames per second several times. "Instantaneous" may be a bit of a misnomer if you do not have an explicit math background, but for argument's sake, let's say that we are calculating how many frames we render in exactly one second and plotting those FPS over the course of a minute. The graph may look something like this.


This line graph demonstrates a 60-second clip from a timedemo with 45 average frames per second. When we look at an average FPS, we expect the timedemo's instantaneous frames per second to look something like this. In reality, the timedemo may look something more like this extreme example below.


Both graphs have an average ~45 frames per second. In fact, the lower graph actually averages to slightly higher. We make written notes of issues like this in our Windows GPU analyses, but a line graph puts it into a format that is less subjective.