elegant chaos

Unit testing is great, but when you’re working on networking code, it can be a bit of a pain in the, erm, transport layer.

I’ve been doing quite a bit of network-related code for Karelia recently, and I wanted to be able to unit test, so this became an issue for me.

If there are two things that you definitely want from a suite of unit tests, it’s that they run fast, and that they are consistent. You want to be able to fire off the tests regularly, you want to be able to trust the results, and you want to be able to run them anywhere.

Throw a network server or two into your testing mix, and these bets are generally off. If the server is internal, the tests may not run when you find yourself on the wrong side of a firewall. If the server is public, you probably have no control over it. In any case, you have no control over the bit of soggy string between the test machine and the server, and any number of problems could cause your tests to fail spuriously.

This problem gets even worse when you consider that one of the best uses of unit tests is to check that your code copes when something goes wrong. Networking code is very good at going wrong in exciting and unexpected ways. Making real servers go wrong on demand in exactly the same way each time so that you can check that you handle it is challenging.

Typically the solution to this problem is some sort of “mocking” - i.e. finding some API or object that you can swap out or inject into the system in such a way that your tests can work as if they are talking to a real server, whilst actually talking to something else.

One approach is to somehow intercept the network request and, instead of passing it on to a real server, fake up the relevant response and return it directly. In the Cocoa world, a custom NSURLProtocol is a good way to achieve this, and it’s the approach taken by things like Nocilla. If you can work at such a high level, this is a nice solution as no actual networking need ever take place. You just intercept the URL request and deliver back an NSData or NSError object depending on what you’re trying to simulate.

The only problem for me was that this approach relies on the networking going via an NSURLConnection, and not all of the code that I wanted to test did this - I was testing code using libcurl, for example.

So I needed another approach, which would work with any networking code without me modifying it.

The second solution is essentially to implement a real (albeit very cut down) server. Bind a local port, listen on it for a connection, and respond to it as if you were a server.

Providing that you’re happy to require that your tests all connect to a custom port on localhost, rather than to real addresses, this solution is infinitely flexible.

There’s only one problem… the bit where I said “respond to it as if you were a server”. Isn’t writing servers a bit tricky?

Well, yes, if you want to write an actual server. But if you think about it, most interactions with servers are just a series of query, response, query, response. In a testing scenario you know what network operation you’re going to do, so you should be able to figure out what query the underlying code will actually send to the server. All you need to do then is to arrange for the right responses to come back.

Even better, in a testing scenario you generally only have to deal with one request happening at once (not always quite true, but near enough a lot of the time).

Subsequently I’ve discovered that there are a few solutions out there that do this sort of thing, but often they seem to just pretend to be a particular kind of server, most commonly a web server. I needed something that could pretend to be an FTP server, a web server, do WebDAV, maybe do SFTP, etc.

More to the point, being a reckless sort, I thought: how hard can it be?

I decided to find out, and thus MockServer was born.

MockServer

In a nutshell, the way MockServer works is this:

• at the start of your unit test, you make a KMSServer object, and give it a responder (more on these later)
• you then fire off the network operation that you want to test, but instead of pointing it at a known port like 80 on a real server you point it at localhost, at the port that the KMSServer object told you to use
• you now tell the KMSServer object to run until your operation is complete - this pauses your test
• once your operation has completed (or failed), your normal completion or error routines will be called
• in your completion routine you tell the KMSServer object to pause
• pausing the KMSServer object returns control to your test, and you can check for the right responses.

Internally, MockServer just opens up a port and listens for connections.

When it gets a connection, it opens up a socket and waits for input.

When it gets input, it passes it off to a responder, which matches it against things that it knows about, and sends back some commands. Mostly of these commands are just “send xyz back to the client”, but they can also include other things like “wait for x seconds”, or “close the connection”.

There is a bit more cleverness going on (for example, MockServer can fake the second connection that an FTP transfer requires), but essentially that’s it.

Responders

As I mentioned above, for the purposes of testing, most server interactions are just a series of query, response, query, response.

MockServer models this query/response behaviour using an abstract KMSResponder class - an instance of which you give to the server object at startup. So depending on the actual responder class, you can pretend to be any kind of server you like.

In real life some servers rely on a lot of state in order to give the correct response to a given query. In theory a responder could keep any state it liked, to deal with this problem. You could even genuinely implement an actual server protocol as a responder class if you were mad enough.

However, for testing this often isn’t necessary. MockServer supplies the KMSRegExResponder class, which is based on performing regular expression pattern matching on the input that it receives, and choosing from a series of stock responses.

It turns out that you can cover an awful lot of ground with this approach alone. Especially when you add in the ability to perform some limited substitution into the stock responses - things like inserting the current time in the correct format, or including part of the query you matched in your reply.

Responses

So now the meat of testing a particular network interaction comes down to identifying the queries that are going to arrive at the server, and arranging the right responses to send back.

This is where some low level knowledge of the underlying network protocol becomes useful, but even here it turns out that a lot of work can be done by trial and error.

You can configure MockServer so that it will log out everything it receives, so when it comes to figuring out the query, you can literally watch to see what arrives, and then write a pattern to match it.

Working out what to send back is slightly harder, as you can’t just watch MockServer, you have to actually do something yourself. The trick here though is to watch a real server, either by sniffing a network connection with it, or by using a tool such as Transmit and examining the log, or even by opening up a telnet session with the real server and pretending to be the client.

The good news is that a lot of this work only needs to be done once or twice for each kind of server.

You ask yourself “how does an FTP client send the password, and what does the server do?”.

Once you know that the client sends PASS thisismypass, and the server responds with 230 User user logged in. or 530 Login incorrect., you can just set up these a couple of alternate responses to match against a pattern like PASS (\w+). Normally you’ll want the first response, but if you want to test your error handling for the wrong password, you use the second one instead.

To support this sort of thing, MockServer provides the KMSResponseCollection class, which lets you define a series of patterns and responses in a JSON file, and then group them into higher level sets of responses.

So far in the course of my own tests, I’ve developed a couple of the JSON files - one for FTP and one for WebDAV, with quite a few useful responses in them. These files are included with MockServer, and can be used as the basis for your own tests. As time goes by I expect that I’ll add more response files for other protocols.

An Example Test

This is a simple example of how you might set up an FTP test, using the FTP responses supplied with MockServer.

#import "KMSTestCase.h"
#import "KMSServer.h"

@interface ExampleTest : KMSTestCase
@end

@implementation ExampleTest

- (void)testFTP
{
    // setup a KMSServer object, using the ftp: scheme and taking the "default" set of responses from the "ftp.json" file.
    if ([self setupServerWithScheme:@"ftp" responses:@"ftp"])
    {
        // set up the data that the server will return
        NSString* testData = @"This is some test data";
        self.server.data = [testData dataUsingEncoding:NSUTF8StringEncoding];

        // setup an ftp request for an imaginary file called "test.txt", on the local port that
		// our mock server is running on
        NSURL* url = [NSURL URLWithString:[NSString stringWithFormat:@"ftp://user:pass@127.0.0.1:%ld/test.txt", (long)self.server.port]];
        NSURLRequest* request = [NSURLRequest requestWithURL:url];

        // perform the request using NSURLConnection
		__block NSString* string = nil;
		[NSURLConnection sendAsynchronousRequest:request queue:[NSOperationQueue currentQueue] completionHandler:^(NSURLResponse* response, NSData* data, NSError* error)
		 {
		     if (error)
		     {
		         STFail(@"got unexpected error %@", error);
		     }
		     else
		     {
		         string = [[NSString alloc] initWithData:data encoding:NSUTF8StringEncoding];
		     }

		     [self pause]; // this call causes runUntilPaused to return
		 }];

		[self runUntilPaused]; // we hang here until the completion block above has executed

        // check that we got back what we were expecting
        STAssertEqualObjects(string, testData, @"got the wrong response: %@", string);
    }
}

Obviously not all tests will use FTP, and not all tests will use NSURLConnection. Some networking APIs may be callback based, or require delegates. As long as you call [self pause] in your callback or delegate method, the test should behave correctly - waiting at the runUntilPaused call until the callback or delegate method has fired.

For simplicity in this test we encoded the user name and password into the request URL. More complicated examples would probably rely on some sort of NSURLCredential callback or delegate method (such as implementing connection:willSendRequestForAuthenticationChallenge:), as a way of obtaining the details to send to the server. The KMSTestCase class has handy user and password properties that you can use to store the details that your authentication method can pass back. This lets you set this information on a per-test or per-request basis, as a way of exercising the different code paths for authentication success and failure.

For More Information

This post is just a quick overview, but there is also a fair bit of documentation which you can find over on github.

Also, of course, you can look at the code.

I wrote this code for Karelia as part of some other work. Many thanks (and much kudos) to them for allowing me to open source it.