You are here: Home > Fastcode project > Ideal Benchmark Documentation

The Fastcode Project

The Ideal Benchmark

We do not know anything about usage patterns of the function and must assume
that every value of the one and only parameter is used the same number of
times. The benchmark should therefore ideally measure the runtime of the
function with every possible value of the input parameter. Each run should
contribute the same to the overall benchmark.

The Practical Benchmark

If we call the function with every possible value of the Cardinal input
parameter, the 4G runs would take a very long time. We have to use subsets
of the parameter range. Even numbers are never prime and a given IsPrime
function will be fast when called with an even valued parameter. This way
they do not count much in the benchmark. Because it takes longer time to
verify whether a big number is prime or not compared to a small number, big
parameter values will contribute more to the benchmark if we use an equal
amount of small and big numbers. How the runtime grows with bigger parameter
values depends on the used algorithm in the function. Because many different
algorithms may be used there is no perfect distribution of parameter values.
We only know that there should be more small values than big values.

The Cardinal input range is divided into 3 subranges = 3 subbenchmarks. Each
subbenchmark weigth is adjusted such that the fastest function for each
subbench scores 300 point on my P4 1600. This way functions that perform
badly in a certain subbenchmark is penalized heavily.

The parameter is stepped by this function

Parameter := Parameter * Step + 1

It is coded sligthly different because I use Double for the Step. +1 is to
avoid selecting non primes all the time. If the parameter is a number times
Step then the number is divisible with Step and not prime. The formula
ensures that more small values than bigger values are benchmarked in each
subrange. The use of subranges makes the contribution better.

NumberFP := MINSUBBENCH1NUMBER;
Number := Round(NumberFP);
while Number < MAXSUBBENCH1NUMBER do
 begin
   Prime := IsPrimeFunction(Number);
   NumberFP := NumberFP * SUBBENCH1STEPSIZE;
   Number := Round(NumberFP) + 1;
 end;

The three subbenchmarks differ in range only.

Shortcomings of the Practical Benchmark
Not all possible values of the parameter is used in the benchmark and it is
possible to optimize the function for these values only and still win with a
function that performs bad on other values. If that happens I twist the
benchmark code a little to use other values and the "winner" drops back.

Conclusion

It is not possible to create the perfect benchmark and we must live with
something less. In this case things are pretty simple because the function
only has one input parameter and we can make a pretty good benchmark.