Normal Tests

If you’ve used the Objective-C unit testing framework (OCUnit, also know as SenTestKit) , or for that matter any other xUnit testing framework, you’ll be familiar with the basic way it goes:

Parameterised Tests

The normal tests work great if you want to run each test once, but what if you have a set of test data and you want to run each test multiple times, applying each item of test data to it in turn?

The naive approach is to define lots of test methods that just call onto another helper method supplying a different argument each time. Something like this:

- (void)testXWithDataA { [self helperX:@"A"]; } 
- (void)testXWithDataB { [self helperX:@"B"]; }

That gets tired quickly, and it doesn’t allow for a dynamic amount of test data determined at runtime.

What you really want in this case is to add the following abilities to SenTest:

SenTestKit is very flexible, but it’s also a bit obscure in the way it’s written, so it’s not immediately apparent how to achieve these goals. After a bit of investigation though it turns out to be pretty simple, and I’ve figured it out so that you don’t have to!

You can find the following classes as part of a small open source module I’ve created, called ECUnitTests. This module includes some other utilities too, but the thing I’m concentrating on in this blog post is the ECParameterisedTest class.

ECParameterisedTest: How To Use It

At runtime the data method will be called to obtain the data. The names of each key should be single words which describe the data. The values can be anything you like - whatever the test methods are expecting.

To simplify the amount of modification to SenTest, the test methods still take no parameters. Instead, to obtain the test data, each test method uses the parameterisedTestDataItem property.

ECParameterisedTest: How It Works

To make its test suites, SenTestKit calls a class method called defaultTestSuite on each SenTestCase subclass that it finds.

The default version of this makes a suite based on finding methods called testXYZ, but it’s easy enough to do something else.

Here’s our version:

+ (id) defaultTestSuite
{
    SenTestSuite* result = nil;
    NSDictionary* data = [self parameterisedTestData];
    if (data)
    {
        result = [[SenTestSuite alloc] initWithName:NSStringFromClass(self)];
        unsigned int methodCount;
        Method* methods = class_copyMethodList([self class], &methodCount);
        for (NSUInteger n = 0; n < methodCount; ++n)
        {
            SEL selector = method_getName(methods[n]);
            NSString* name = NSStringFromSelector(selector);
            if ([name rangeOfString:@"parameterisedTest"].location == 0)
            {
                SenTestSuite* subSuite = [[SenTestSuite alloc] initWithName:name];
                for (NSString* testName in data)
                {
                    NSDictionary* testData = [data objectForKey:testName];
                    [subSuite addTest:[self testCaseWithSelector:selector param:testData name:testName]];
                }
                [result addTest:subSuite];
                [subSuite release];
            }
        }
        
    }

    return [result autorelease];
}

Most of this is self explanatory, but some key things to note are:

To make things simple, we want to use the existing SenTestKit mechanism to invoke the test methods. Since SenTestKit expects test methods not to have any parameters, we need another way of passing the test data to each method. Each test invocation creates an instance of a our class, and we do this creation at the point we build the test suite, so the simple answer is just to add a property to the test class. We can set this property value when we make the test instance, and the test method can extract the data from the instance when it runs.

Obtaining Test Data

To obtain the test data, we’ve added a method parameterisedTestData that we expect the test class to implement.

This method returns a dictionary rather than an array, so that we can use the keys as test names, and the values as the actual data. Having names for the data is useful because of the way SenTestKit reports the results.

Typically it reports each test as [SuiteName testName], taking these names from the class and method. Since we’re going to use the name of the test method for each of our suites, we really need another name to use for each test. This is where the dictionary key comes in.

Where the test data comes from is of course up to you and the kind of tests you are trying to perform. There is a simple scenario though, which is that we want to load it from a plist that we provide along with the test class.

Since we need a default implementation of the method anyway, we can cater for this simple case automatically. We look for a plist with the same name as the test class. If we find it, we load it, and return the top level object from it (expecting it to be an NSDictionary).

+ (NSDictionary*) parameterisedTestData
{
    NSURL* plist = [[NSBundle bundleForClass:[self class]] URLForResource:NSStringFromClass([self class]) withExtension:@"plist"];
    NSDictionary* result = [NSDictionary dictionaryWithContentsOfURL:plist];
    
    return result;
}

An Example

Here’s a fully worked example.

###ExampleTests.m:

@interface ExampleTests : ECParameterisedTestCase
@end

@interface ExampleTests

- (void)parameterisedTestOne
{
	STFail(@"test one with data %@", self.parameterisedTestDataItem);
}

- (void)parameterisedTestTwo
{
	STFail(@"test two with data %@", self.parameterisedTestDataItem);
}

@end

###ExampleTests.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
	<dict>
		<key>NameA</key>
		<string>ValueA</string>

		<key>NameB</key>
		<string>ValueB</string>
	</dict>
</plist>

When run, this should produce something like the following log output (simplified for clarity here):

Run test suite ExampleTests
  Run test suite parameterisedTestOne
    test NameA failed: test one with data ValueA
    test NameB failed: test one with data ValueB
  Run test suite parameterisedTestTwo
    test NameA failed: test two with data ValueA
    test NameB failed: test two with data ValueB