Unit testing is great, but when you’re working on networking code, it can be a bit of a pain in the, erm, transport layer.

I’ve been doing quite a bit of network-related code for Karelia recently, and I wanted to be able to unit test, so this became an issue for me.

If there are two things that you definitely want from a suite of unit tests, it’s that they run fast, and that they are consistent. You want to be able to fire off the tests regularly, you want to be able to trust the results, and you want to be able to run them anywhere.

Throw a network server or two into your testing mix, and these bets are generally off. If the server is internal, the tests may not run when you find yourself on the wrong side of a firewall. If the server is public, you probably have no control over it. In any case, you have no control over the bit of soggy string between the test machine and the server, and any number of problems could cause your tests to fail spuriously.

This problem gets even worse when you consider that one of the best uses of unit tests is to check that your code copes when something goes wrong. Networking code is very good at going wrong in exciting and unexpected ways. Making real servers go wrong on demand in exactly the same way each time so that you can check that you handle it is challenging.

Typically the solution to this problem is some sort of “mocking” - i.e. finding some API or object that you can swap out or inject into the system in such a way that your tests can work as if they are talking to a real server, whilst actually talking to something else.

One approach is to somehow intercept the network request and, instead of passing it on to a real server, fake up the relevant response and return it directly. In the Cocoa world, a custom NSURLProtocol is a good way to achieve this, and it’s the approach taken by things like Nocilla. If you can work at such a high level, this is a nice solution as no actual networking need ever take place. You just intercept the URL request and deliver back an NSData or NSError object depending on what you’re trying to simulate.

The only problem for me was that this approach relies on the networking going via an NSURLConnection, and not all of the code that I wanted to test did this - I was testing code using libcurl, for example.

So I needed another approach, which would work with any networking code without me modifying it.

The second solution is essentially to implement a real (albeit very cut down) server. Bind a local port, listen on it for a connection, and respond to it as if you were a server.

Providing that you’re happy to require that your tests all connect to a custom port on localhost, rather than to real addresses, this solution is infinitely flexible.

There’s only one problem… the bit where I said “respond to it as if you were a server”. Isn’t writing servers a bit tricky?

Well, yes, if you want to write an actual server. But if you think about it, most interactions with servers are just a series of query, response, query, response. In a testing scenario you know what network operation you’re going to do, so you should be able to figure out what query the underlying code will actually send to the server. All you need to do then is to arrange for the right responses to come back.

Even better, in a testing scenario you generally only have to deal with one request happening at once (not always quite true, but near enough a lot of the time).

Subsequently I’ve discovered that there are a few solutions out there that do this sort of thing, but often they seem to just pretend to be a particular kind of server, most commonly a web server. I needed something that could pretend to be an FTP server, a web server, do WebDAV, maybe do SFTP, etc.

More to the point, being a reckless sort, I thought: how hard can it be?

I decided to find out, and thus MockServer was born.

MockServer

In a nutshell, the way MockServer works is this:

• at the start of your unit test, you make a KMSServer object, and give it a responder (more on these later)
• you then fire off the network operation that you want to test, but instead of pointing it at a known port like 80 on a real server you point it at localhost, at the port that the KMSServer object told you to use
• you now tell the KMSServer object to run until your operation is complete - this pauses your test
• once your operation has completed (or failed), your normal completion or error routines will be called
• in your completion routine you tell the KMSServer object to pause
• pausing the KMSServer object returns control to your test, and you can check for the right responses.

Internally, MockServer just opens up a port and listens for connections.

When it gets a connection, it opens up a socket and waits for input.

When it gets input, it passes it off to a responder, which matches it against things that it knows about, and sends back some commands. Mostly of these commands are just “send xyz back to the client”, but they can also include other things like “wait for x seconds”, or “close the connection”.

There is a bit more cleverness going on (for example, MockServer can fake the second connection that an FTP transfer requires), but essentially that’s it.

Responders

As I mentioned above, for the purposes of testing, most server interactions are just a series of query, response, query, response.

MockServer models this query/response behaviour using an abstract KMSResponder class - an instance of which you give to the server object at startup. So depending on the actual responder class, you can pretend to be any kind of server you like.

In real life some servers rely on a lot of state in order to give the correct response to a given query. In theory a responder could keep any state it liked, to deal with this problem. You could even genuinely implement an actual server protocol as a responder class if you were mad enough.

However, for testing this often isn’t necessary. MockServer supplies the KMSRegExResponder class, which is based on performing regular expression pattern matching on the input that it receives, and choosing from a series of stock responses.

It turns out that you can cover an awful lot of ground with this approach alone. Especially when you add in the ability to perform some limited substitution into the stock responses - things like inserting the current time in the correct format, or including part of the query you matched in your reply.

Responses

So now the meat of testing a particular network interaction comes down to identifying the queries that are going to arrive at the server, and arranging the right responses to send back.

This is where some low level knowledge of the underlying network protocol becomes useful, but even here it turns out that a lot of work can be done by trial and error.

You can configure MockServer so that it will log out everything it receives, so when it comes to figuring out the query, you can literally watch to see what arrives, and then write a pattern to match it.

Working out what to send back is slightly harder, as you can’t just watch MockServer, you have to actually do something yourself. The trick here though is to watch a real server, either by sniffing a network connection with it, or by using a tool such as Transmit and examining the log, or even by opening up a telnet session with the real server and pretending to be the client.

The good news is that a lot of this work only needs to be done once or twice for each kind of server.

You ask yourself “how does an FTP client send the password, and what does the server do?”.

Once you know that the client sends PASS thisismypass, and the server responds with 230 User user logged in. or 530 Login incorrect., you can just set up these a couple of alternate responses to match against a pattern like PASS (\w+). Normally you’ll want the first response, but if you want to test your error handling for the wrong password, you use the second one instead.

To support this sort of thing, MockServer provides the KMSResponseCollection class, which lets you define a series of patterns and responses in a JSON file, and then group them into higher level sets of responses.

So far in the course of my own tests, I’ve developed a couple of the JSON files - one for FTP and one for WebDAV, with quite a few useful responses in them. These files are included with MockServer, and can be used as the basis for your own tests. As time goes by I expect that I’ll add more response files for other protocols.

An Example Test

This is a simple example of how you might set up an FTP test, using the FTP responses supplied with MockServer.

#import "KMSTestCase.h"
#import "KMSServer.h"

@interface ExampleTest : KMSTestCase
@end

@implementation ExampleTest

- (void)testFTP
{
    // setup a KMSServer object, using the ftp: scheme and taking the "default" set of responses from the "ftp.json" file.
    if ([self setupServerWithScheme:@"ftp" responses:@"ftp"])
    {
        // set up the data that the server will return
        NSString* testData = @"This is some test data";
        self.server.data = [testData dataUsingEncoding:NSUTF8StringEncoding];

        // setup an ftp request for an imaginary file called "test.txt", on the local port that
		// our mock server is running on
        NSURL* url = [NSURL URLWithString:[NSString stringWithFormat:@"ftp://user:pass@127.0.0.1:%ld/test.txt", (long)self.server.port]];
        NSURLRequest* request = [NSURLRequest requestWithURL:url];

        // perform the request using NSURLConnection
		__block NSString* string = nil;
		[NSURLConnection sendAsynchronousRequest:request queue:[NSOperationQueue currentQueue] completionHandler:^(NSURLResponse* response, NSData* data, NSError* error)
		 {
		     if (error)
		     {
		         STFail(@"got unexpected error %@", error);
		     }
		     else
		     {
		         string = [[NSString alloc] initWithData:data encoding:NSUTF8StringEncoding];
		     }

		     [self pause]; // this call causes runUntilPaused to return
		 }];

		[self runUntilPaused]; // we hang here until the completion block above has executed

        // check that we got back what we were expecting
        STAssertEqualObjects(string, testData, @"got the wrong response: %@", string);
    }
}

Obviously not all tests will use FTP, and not all tests will use NSURLConnection. Some networking APIs may be callback based, or require delegates. As long as you call [self pause] in your callback or delegate method, the test should behave correctly - waiting at the runUntilPaused call until the callback or delegate method has fired.

For simplicity in this test we encoded the user name and password into the request URL. More complicated examples would probably rely on some sort of NSURLCredential callback or delegate method (such as implementing connection:willSendRequestForAuthenticationChallenge:), as a way of obtaining the details to send to the server. The KMSTestCase class has handy user and password properties that you can use to store the details that your authentication method can pass back. This lets you set this information on a per-test or per-request basis, as a way of exercising the different code paths for authentication success and failure.

For More Information

This post is just a quick overview, but there is also a fair bit of documentation which you can find over on github.

Also, of course, you can look at the code.

I wrote this code for Karelia as part of some other work. Many thanks (and much kudos) to them for allowing me to open source it.

In my Mac Pro Update - Think Different post a few weeks ago, I said that I thought we don’t really need a great big new Mac Pro from Apple.

What we need instead (and I admit that I don’t have an exact technical solution for this problem, but nevertheless) is a unit or units that can seemlessly enhance the processing power of a laptop when it’s on our home network.

You could argue that there’s still a need for a big box that can host all sorts of large expansion cards, but I’m not really convinced. Thunderbolt should deal with most of that.

Something else has occurred to me recently.

People have been talking about a potential switch away from Intel to ARM for the Mac range, and assuming that it would happen first with a Mac Mini or an iMac, since they are seen as “consumer”, and there’s a perception that consumer machines will be easier to switch over.

I actually think that the opposite will happen. I think that the first ARM Mac is likely to come in the guise of a new “Pro” desktop box. I also think that it’s likely to be loaded to the gills with cores, to the extent that it blows away anything out there in terms of raw processing power.

With GCD / blocks / NSOperation, Cocoa has a really nice clean model for making use of multiple cores. Better than anything I’ve used before (although admittedly I’m now a bit out of date with regard to the competition).

With the expansion of privelege separation and things like XPC, we’re also moving closer to a world where even a single app could be run seamlessly across multiple machines.

It seems to me that switching to a relatively low power, low cost, risc architecture makes perfect sense in this world.

Before you ask - I’ve absolutely no evidence for this. Just thinking aloud…

Simon Wolf wrote a blog post recently talking about passing NSError pointers into methods like this:

NSError* error = nil;
[something someMethod:@"blah" error:&error];

He talked about the fact that when you do this, you should always check the result of the method before using the error. In other words, you do this:

NSError* error = nil;
if (![something someMethod:@"blah" error:&error])
{
	NSLog(@"got error %@", error);
}

and not this:

NSError* error = nil;
[something someMethod:@"blah" error:&error];
if (error)
{
	NSLog(@"got error %@", error);
}

This is good advice. The method you’re calling isn’t guaranteed to set error to a non-nil value if nothing goes wrong. It might leave it alone, but it might set it to a temporary value, even though it eventually returns an “ok” result.

However, Simon then went on to talk about this line at the top:

NSError* error = nil;

His assertion was that by setting error to nil, you’re indicating that you intend to test it after the call - in other words that it’s an indicator that you don’t know what you’re doing.

He suggested that you should just do this:

NSError* error;

Strictly speaking, this is correct. If the method you’re calling follows the implicit contract for such functions, this should be fine.

However, I always set the error to nil before I pass it in, even though I never test it after the call. Why do I do this?

Consider the following method:

- (BOOL)someMethod:(NSString*)string error:(NSError**)error
{
	if (error)
	{
		if (something)
			*error = [self doSomeStuff];
		else (somethingelse)
			*error = [self doSomethingElse];
		//... lots of other cases here...
		
		//... later
		
		if ([(*error) code] == someCode)
		{
			[self doSomeExtraStuff];
		}
	}
	
	return YES;
}

This is a bit of a contrived example, but the point is that this method might be the one you’re calling, and it might be in a third party library that you have no control over.

Now lets imagine that some refactoring or other code change introduces a bug where *error doesn’t get set in the first block of if/else statements. Let’s also imagine that the developer has the relevant warnings turned off, and doesn’t noticed (shock horror - some people actually do this!).

What happens when the code starts accesses *error? Well, it depends what’s in it. If you did this:

NSError* error = nil;

you’ll be fine.

If you did this:

NSError* error;

you get undefined behaviour, based on what happened to be on the stack in the position that the error variable occupies. It’s a classic uninitialised variable bug, and potentially a bastard to track down.

At this point you may well be saying “if the quality of the library you’re using is that bad, you’ve only yourself to blame”. You might have a point, but bugs do creep in, even to good libraries.

Admittedly too, when you’re accessing an uninitialised Objective-C pointer, you’re way more likely to crash straight away than you would have been if you were just dereferencing a pointer to read a member in C/C++.

However, all of this is the reason why I still do:

NSError* error = nil;

even though I’m not going to test error afterwards.

You could call it paranoia, but don’t mistake it for misunderstanding the NSError** contract!

So, yesterday’s 1.1.2b5 release didn’t quite go according to plan, as it contained a rather serious bug which corrupted it’s store of cached information. The result of this was that when you relaunched it, Ambientweet encountered problems and generally ended up in a non functional state.

Luckily, I can now bring you 1.1.2b6, now with added not-so-many-bugs-honest.

Since it’s only been a day and people may not have read the previous post, it’s worth re-iterating the changes in these new versions, so here’s an edited version of yesterday’s message:

I’ve been working on some improvements to Ambientweet, for the next version, which I’m currently calling 1.1.2 (although it’s getting to the point where there are enough changes to warrant calling it 1.2).

Internally, the code that manages the communication with Twitter has been modified a fair bit. This was partly a case of tidying up some old code, and partly improving the design to make it easier to support multiple Twitter accounts at once.

I’ve not quite got as far as multiple Twitter accounts in this revision, but I have added support for multiple windows. This means that you can set up more than one Ambientweet window if you wish, and have them track different timelines. One could track your normal timeline, one could focus on tweets from a single user, and one could show the results of a search for some text or a hashtag.

Because you can now have different windows, I’ve removed the window related settings from the preferences, and instead added a “Window Options” menu to the “View” menu, so that you can set them individually for each window.

The other major change that has happened relates to the automatic update mechanism. Unfortunately this got broken in version 1.1.1, and as a result automatic updating may not work, particularly if you are running Mountain Lion. These problems should now be fixed, but if you’re running one of the versions where updating was broken, then unfortunately you’ll have to manually get yourself past this version by downloading the latest beta.

Other than that, there are a couple of minor fixes:

• The preview display which says what tweets are being downloaded (before there are any to display) should now make a bit more sense, and shouldn’t show an ugly looking internal description for a user any more.
• The preference for centring the window when it shows some sort of sheet (eg when you’re asked to enter the text to search for) now actually does something.

If you are feeling brave, you can download the latest version here.

I’ve been working on some improvements to Ambientweet, for the next version, which I’m currently calling 1.1.2 (although it’s getting to the point where there are enough changes to warrant calling it 1.2).

Internally, the code that manages the communication with Twitter has been modified a fair bit. This was partly a case of tidying up some old code, and partly improving the design to make it easier to support multiple Twitter accounts at once.

I’ve not quite got as far as multiple Twitter accounts in this revision, but I have added support for multiple windows. This means that you can set up more than one Ambientweet window if you wish, and have them track different timelines. One could track your normal timeline, one could focus on tweets from a single user, and one could show the results of a search for some text or a hashtag.

Because you can now have different windows, I’ve removed the window related settings from the preferences, and instead added a “Window Options” menu to the “View” menu, so that you can set them individually for each window.

The other major change that has happened relates to the automatic update mechanism. Unfortunately this got broken in version 1.1.1, and as a result automatic updating may not work, particularly if you are running Mountain Lion. These problems should now be fixed, but if you’re running one of the versions where updating was broken, then unfortunately you’ll have to manually get yourself past this version by downloading the latest beta.

Other than that, there are a couple of minor fixes:

• The preview display which says what tweets are being downloaded (before there are any to display) should now make a bit more sense, and shouldn’t show an ugly looking internal description for a user any more.
• The preference for centring the window when it shows some sort of sheet (eg when you’re asked to enter the text to search for) now actually does something.

If you are feeling brave, you can download the latest version here.