Normal Tests
If you’ve used the Objective-C unit testing framework (OCUnit, also know as SenTestKit) , or for that matter any other xUnit testing framework, you’ll be familiar with the basic way it goes:
- define a class that inherits from SenTestCase
- add some test methods which take no arguments, return no results, and have names prefixed by “test”
- at runtime SenTest makes a test suite for each SenTestCase subclass it finds
- it adds each test method that it finds to the suite
- it then runs each suite in turn, running each test once, and reporting the results
Parameterised Tests
The normal tests work great if you want to run each test once, but what if you have a set of test data and you want to run each test multiple times, applying each item of test data to it in turn?
The naive approach is to define lots of test methods that just call onto another helper method supplying a different argument each time. Something like this:
- (void)testXWithDataA { [self helperX:@"A"]; }
- (void)testXWithDataB { [self helperX:@"B"]; }
That gets tired quickly, and it doesn’t allow for a dynamic amount of test data determined at runtime.
What you really want in this case is to add the following abilities to SenTest:
- the ability to define parameterised test methods using a similar naming convention to the normal ones
- the ability to define a class method which returns a dictionary of test data
- have SenTest make a sub-suite for each parameterised method we found
- have the sub-suite contain a test for each data item
- iterate the suites and tests as usual, applying the relevant data item to each test in turn
SenTestKit is very flexible, but it’s also a bit obscure in the way it’s written, so it’s not immediately apparent how to achieve these goals. After a bit of investigation though it turns out to be pretty simple, and I’ve figured it out so that you don’t have to!
You can find the following classes as part of a small open source module I’ve created, called ECUnitTests. This module includes some other utilities too, but the thing I’m concentrating on in this blog post is the ECParameterisedTest class.
ECParameterisedTest: How To Use It
- inherit from ECParameterisedTest instead of SenTestCase
- define test methods which are named parameterisedTestXYZ instead of testXYZ (they still take no parameters)
- either: define a class method called parameterizedTestData which returns a dictionary containing data
- or: create a plist with the name of your test class, which will contain the data
At runtime the data method will be called to obtain the data. The names of each key should be single words which describe the data. The values can be anything you like - whatever the test methods are expecting.
To simplify the amount of modification to SenTest, the test methods still take no parameters. Instead, to obtain the test data, each test method uses the parameterisedTestDataItem property.
ECParameterisedTest: How It Works
To make its test suites, SenTestKit calls a class method called defaultTestSuite on each SenTestCase subclass that it finds.
The default version of this makes a suite based on finding methods called testXYZ, but it’s easy enough to do something else.
Here’s our version:
+ (id) defaultTestSuite
{
SenTestSuite* result = nil;
NSDictionary* data = [self parameterisedTestData];
if (data)
{
result = [[SenTestSuite alloc] initWithName:NSStringFromClass(self)];
unsigned int methodCount;
Method* methods = class_copyMethodList([self class], &methodCount);
for (NSUInteger n = 0; n < methodCount; ++n)
{
SEL selector = method_getName(methods[n]);
NSString* name = NSStringFromSelector(selector);
if ([name rangeOfString:@"parameterisedTest"].location == 0)
{
SenTestSuite* subSuite = [[SenTestSuite alloc] initWithName:name];
for (NSString* testName in data)
{
NSDictionary* testData = [data objectForKey:testName];
[subSuite addTest:[self testCaseWithSelector:selector param:testData name:testName]];
}
[result addTest:subSuite];
[subSuite release];
}
}
}
return [result autorelease];
}
Most of this is self explanatory, but some key things to note are:
- parameterisedTestData is a class method which returns a dictionary containing the data
- the method as a whole returns one SenTestSuite object
- that SenTestSuite object contains more SenTestSuite objects, which in turn contain the actual tests
To make things simple, we want to use the existing SenTestKit mechanism to invoke the test methods. Since SenTestKit expects test methods not to have any parameters, we need another way of passing the test data to each method. Each test invocation creates an instance of a our class, and we do this creation at the point we build the test suite, so the simple answer is just to add a property to the test class. We can set this property value when we make the test instance, and the test method can extract the data from the instance when it runs.
Obtaining Test Data
To obtain the test data, we’ve added a method parameterisedTestData that we expect the test class to implement.
This method returns a dictionary rather than an array, so that we can use the keys as test names, and the values as the actual data. Having names for the data is useful because of the way SenTestKit reports the results.
Typically it reports each test as [SuiteName testName], taking these names from the class and method. Since we’re going to use the name of the test method for each of our suites, we really need another name to use for each test. This is where the dictionary key comes in.
Where the test data comes from is of course up to you and the kind of tests you are trying to perform. There is a simple scenario though, which is that we want to load it from a plist that we provide along with the test class.
Since we need a default implementation of the method anyway, we can cater for this simple case automatically. We look for a plist with the same name as the test class. If we find it, we load it, and return the top level object from it (expecting it to be an NSDictionary).
+ (NSDictionary*) parameterisedTestData
{
NSURL* plist = [[NSBundle bundleForClass:[self class]] URLForResource:NSStringFromClass([self class]) withExtension:@"plist"];
NSDictionary* result = [NSDictionary dictionaryWithContentsOfURL:plist];
return result;
}
An Example
Here’s a fully worked example.
###ExampleTests.m:
@interface ExampleTests : ECParameterisedTestCase
@end
@interface ExampleTests
- (void)parameterisedTestOne
{
STFail(@"test one with data %@", self.parameterisedTestDataItem);
}
- (void)parameterisedTestTwo
{
STFail(@"test two with data %@", self.parameterisedTestDataItem);
}
@end
###ExampleTests.plist:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>NameA</key>
<string>ValueA</string>
<key>NameB</key>
<string>ValueB</string>
</dict>
</plist>
When run, this should produce something like the following log output (simplified for clarity here):
Run test suite ExampleTests
Run test suite parameterisedTestOne
test NameA failed: test one with data ValueA
test NameB failed: test one with data ValueB
Run test suite parameterisedTestTwo
test NameA failed: test two with data ValueA
test NameB failed: test two with data ValueB