Performance Testing

Often in software development one needs to determine which algorithm is more performant. The easiest way to do this, in C#, is to use the Stopwatch class. To ensure fairly accurate results, it is best to run each algorithm N times and take the average. Even so, results are not entirely accurate as the first run will be cold. That is, currently, the JIT compiler may not have been enacted. My own test results show that that the first run tends to be slowest. Subsequent runs tend to be much faster (in the same session).

Regardless of the above, timings are useful enough to determine performant algorithms.

In this post...

• Goals
• Analysis
• Design
• Code

Goals

With any software project, no matter the size, some planning is always worthwhile. So, for this project I want the following...

• Easy test specification, e.g. test name and a function pointer to the test.
• Specify the number of times each test should be run.
• Obtain the total time for all test runs and the average run time.
• The ability to obtain test results as a string.

Top

Analysis

Given the above requirements the following can be stated.

• Each test is named and timed.
• Each test is run N times, from this one can deduce the total runtime and average runtime.
• One may run multiple tests and each test may be run N times. From this one can deduce that an array of sorts will be required.

Top

Design

Given the requirements and preliminary analysis we can proceed to perform some basic design.

Let's start with the idea of a test. A test simply consists of a name and a function pointer. Something like...

Test(string name, Func<object> func)

The function pointer expects the function to return an object. This leaves it open-ended and will prove useful to ensure that in release mode functions are not removed. Release mode in most languages will remove functions that appear useless.

Next we need a structure that holds the results of a single test run N times. We might call this TestGroupResult. The structure should maintain the test name, total elapsed time (for the N runs, and the average time per test run). Such a structure might be as follows...

TestGroupResult(string name, TimeSpan totalElapsedTime, TimeSpan averageElapsedTime)

For the next step we need a structure that maintains the entire result set. That is all the group test results. Again, for each test we run said test N times to obtain a total elapsed time and average times. The result is the TestGroupResult. Let's name this new structure BenchmarkResult...

BenchmarkResult(int runsPerTest, GroupResult[] results)

Finally, we need an entry point, where we specify the tests to run and the number of times each test should be run. It makes sense to call this entry point Benchmark.

Benchmark.Run(string name, Func<object> func)

To create an instance we might run the following code.

internal class Program
{
  static void Main(string[] args)
  {
    var result = Benchmark.Run(10,
      ("Test1", () => { Thread.Sleep(500); return 1; }),
      ("Test2", () => { Thread.Sleep(400); return 2; })
    );

    Console.WriteLine(result.GenerateReport());    
  }
}

Using the above code, my console output is as follows..

Top

Source code

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Text;

namespace Fun.Benchmarking
{
  /// <summary>
  /// Encapulates a group of tests.
  /// Maintains the total elapsed time and average elapsed time.
  /// Generate a report as a string.
  /// </summary>
  public class TestGroupResult
  {
    public string Name { get; }
    public TimeSpan TotalElapsedTime { get; }
    public TimeSpan AverageElapsedTime { get; }

    public TestGroupResult(
      string name,
      TimeSpan totalElapsedTime,
      TimeSpan averageElapsedTime)
    {
      this.Name = name;
      this.TotalElapsedTime = totalElapsedTime;
      this.AverageElapsedTime = averageElapsedTime;
    }

    public StringBuilderGenerateReport(StringBuilder sb)
    {
      return sb
        .AppendLine($"Test Name : {Name}")
        .AppendLine($"  Total Elapsed Time: {TotalElapsedTime}")
        .AppendLine($"  Total Average Time: {AverageElapsedTime}");
    }
  }

  /// <summary>
  /// Represents a benchmark result which includes...
  /// The number of runs for each supplied test.
  /// An array of test group results (a group is a single test run N times).
  /// </summary>
  public class BenchmarkResult
  {
    public int RunsPerTest { get; }
    public TestGroupResult[] GroupResults { get; }

    public BenchmarkResult(
      int runsPerTest,
      TestGroupResult[] groupResult)
    {
      this.RunsPerTest = runsPerTest;
      this.GroupResults = groupResult;
    }

    public string GenerateReport()
    {
      StringBuilder sb = new StringBuilder();
      sb.AppendLine($"Runs Per Test: {RunsPerTest}");
      foreach (TestGroupResult groupResult in GroupResults)
        groupResult.GenerateReport(sb);
      return sb.ToString();
    }
  }

  /// <summary>
  /// The Benchmark allows one to determine the performance of an algorithm.
  /// Multiple algorithms may be tested, great for determining which algorithm performs best.
  /// </summary>
  public class Benchmark
  {
    /// <summary>
    /// Run one or more tests and specify how many times each test should be run.
    /// The total and average elapsed time for each test will be calculated.
    /// </summary>
    /// <param name="runsPerTest">How many times each test should be run.</param>
    /// <param name="tests">An array of tests to be run.</param>
    /// <returns></returns>
    public static BenchmarkResult Run(
      int runsPerTest,
      params (string Name, Func<object> Code)[] tests)
    {
      List<TestGroupResult> groups = new List<TestGroupResult>();
      foreach (var test in tests)
      {
        TestGroupResult group = RunTestGroup(runsPerTest, test);
        groups.Add(group);
      }
      return new BenchmarkResult(runsPerTest, groups.ToArray());
    }

    /// <summary>
    /// Run a single test (N runsPerTest) times.
    /// </summary>
    /// <param name="runsPerTest"></param>
    /// <param name="test"></param>
    /// <returns></returns>
    public static TestGroupResult RunTestGroup(
      int runsPerTest,
      (string Name, Func<object> Code) test)
    {
      Stopwatch stopwatch = Stopwatch.StartNew();
      for (int testCount = 0; testCount < runsPerTest; testCount++)
      {
        test.Code();        
      }
      stopwatch.Stop();
      TimeSpan elapsedTime = stopwatch.Elapsed;
      return new TestGroupResult(test.Name, elapsedTime, new TimeSpan>(elapsedTime.Ticks/runsPerTest));
    }
  }
}