Once or twice I’ve had the situation where I ran some unit tests in Xcode, and they seem to run incredibly fast. Too fast.

Spoiler: they didn’t Instead of Xcode’s Log Navigator looking something like this:

it instead shows this:

The observant amongst you will have already spotted the flaw in this otherwise nice, friendly, green display of unit testing happiness.

There are no issues because no unit tests ran.

What’s Going On?

Unless you know where to look, just beginning to discover why this might be is a little challenging.

You might think to yourself: “silly me, I must have turned them all off”. A quick look at the Tests settings in the Scheme will verify this:

Nope, they’re on, but Xcode isn’t running them for some reason.

Console Yourself

Normally when something goes wrong with unit tests my first port of call is the console output, and the way I get to it is via the little button that appears on the right hand side of a test when you hover over it:

But with no tests apparently running, there is no little button, so we need to actually go and look in the separate console area.

If you’re the kind of person who leaves everything visible all the time in Xcode, you’ll probably be wondering why I’m even mentioning this, but if (like me), you have behaviors that hide things at various times, then it’s quite possible to not even remember that the console output is still there once the run has finished.

You can get to it like this:

When you do, you’ll discover that something very odd is going on. The normally copious unit test console output has been reduced to:

Luckily, Xcode tells us what’s happening (I’ve shortened the paths):

    The test bundle at CURLHandleTests.octest could not be loaded because its
    Objective-C runtime information does not match the runtime 
    information required by the test rig.

and why:

    This is likely because the test rig is being run with
    Objective-C garbage collection disabled, but the test
    bundle requires Objective-C garbage collection.
    To enable Objective-C garbage collection for the test rig,
    run it in an environment without the OBJC_DISABLE_GC 
    environment variable.

Thank you Mr Xcode, you are so good to me.

So, I got my build settings wrong and forgot to disable garbage collection right? I’ll just go and fix it and… oh, hang on… no I didn’t. No garbage collection here.

So, what the hell is going on?

Back to the console. It looks like otest is having trouble:

    2013-01-22 18:25:46.057 otest[5433:203] *** NSTask: Task
    create for path 'CURLHandleTests.octest/Contents/MacOS/CURLHandleTests'
    failed: 22, "Invalid argument".  Terminating temporary process.

For those who are wondering, otest is the command line tool that loads and runs your unit tests.

So it seems to be having trouble. It’s failed, with error code 22. Which is an invalid argument.

Well, that clears things up then.

Long Story Short

Ok, so let’s cut to (somewhat nearer) the chase.

What’s actually happening here, in the cases that I’ve encountered at least, is that otest is failing to load CURLHandleTests.octest - which is the bundle that contains your compiled tests.

There are probably a few reasons why it could fail. Clearly Xcode thinks that garbage collection incompatibility is right up there. However, also high up on the list is that the dynamic loader is failing to load it because it can’t find a shared library or framework that your test code depends on.

If you are using the default Xcode setup, the octest bundle is built into the Products directory, along with any other dependent targets. If your tests need to link at runtime to any custom libraries or frameworks, you need to make sure that they end up here (or somewhere else that the dyld loader can find them).

This is fairly obvious in most cases - for example if you are unit testing a framework that you’re building, it kind of makes sense that you’ll need the framework. In these cases Xcode generally does the right thing anyway - your unit test target will probably depend on the framework target, and they’ll both end up in the Products directory.

So most of the time you’ll be ok, and perhaps even blissfully unaware of how your tests are loaded and run.

However, things can get tricky when the library dependencies are a bit more complex. For example if you’re testing a framework and the framework itself relies on some other shared libraries that are custom built, and not just pre-installed in some convent location like /Library/Frameworks/.

It’s possible in this case for the framework to embed these libraries as part of it’s own build process, which makes life a lot simpler for people who use it - and will probably not cause the sort of issues I’m talking about. That doesn’t always happen though, for good reasons such as not wanting to end up with multiple copies of a library embedded in more than one framework.

In this situation, where your framework is looking for a shared library that it can’t find, what will happen is a runtime failure to load your framework. This in turn will cause your unit test bundle to fail to load, which in turn will cause otest to bomb out with the useful error 22, which in turn will cause Xcode to LIE TO YOU BY PUTTING UP A BIG GREEN THING SAYING ALL YOUR TESTS RAN.

Sorry. Deep breaths.

## The Fix

So once you’ve figured out what library you’re missing, the fix is quite simple. Just add a build phase to your unit test target which copies the relevant things into the Products Directory:

(yes, in this case we’re linking against our own builds of libssh2, libssl and libcrypto - they aren’t the normal builds, so if they are missing, everything goes pear-shaped).

Happily, if you’ve got the right libraries, all will now be well and the unit tests will run.

What To Link

Of course, working out what library you’re missing isn’t always trivial, especially if it’s a big codebase and/or you aren’t in control of it all.

How to work out which libraries depend on which is a blog post all of its own, but here are a few handy hints.

otool

You can use otool (not to be confused with otest, which I mentioned earlier), to work out what something links to.

You have to run it on the actual binary inside your octest bundle

otool -l /Caches/DerivedData/CURLHandle-flfotwdrtmazzmdpxjznosnizjlm/Build/Products/Debug/CURLHandleTests.octest/Contents/MacOS/CURLHandleTests 

This will produce a big load of output, including things like this:

          cmd LC_LOAD_DYLIB
      cmdsize 104
         name /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 744.1.0
compatibility version 150.0.0

Which tells you that you’re linking against CoreFoundation - which you could probably have guessed.

All the system ones are probably fairly expected and aren’t likely to be the problem. However, if something shows up in this dump that you weren’t expecting, it’s a good clue where to look.

Update: otool -L actually gives you a much more compact output here, just listing the libraries that are linked to:

    @rpath/Connection.framework/Versions/A/Connection (compatibility version 1.0.0, current version 1.0.0)
    @rpath/SenTestingKit.framework/Versions/A/SenTestingKit (compatibility version 1.0.0, current version 4053.0.0)
    @rpath/DAVKit.framework/Versions/A/DAVKit (compatibility version 1.0.0, current version 1.0.0)
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 945.11.0)
    /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 744.1.0)

DYLD_ variables

Another thing that may help is to enable a few DYLD_ environment variables and then run otest manually. Knock up a little script like this:

export DYLD_PRINT_LIBRARIES=YES
export DYLD_PREBIND_DEBUG=YES
export DYLD_PRINT_ENV=YES
export DYLD_PRINT_APIS=YES

export OBJC_DISABLE_GC=YES

xcrun otest /Caches/DerivedData/CURLHandle-flfotwdrtmazzmdpxjznosnizjlm/Build/Products/Debug/CURLHandleTests.octest 

What you’ll see when you run this is a whole load of debug output from the DYLD loader itself.

...
dlopen(/Caches/DerivedData/CURLHandle-flfotwdrtmazzmdpxjznosnizjlm/Build/Products/Debug/CURLHandleTests.octest/Contents/MacOS/CURLHandleTests, 0x00000115)
dlopen(/Caches/DerivedData/CURLHandle-flfotwdrtmazzmdpxjznosnizjlm/Build/Products/Debug/CURLHandleTests.octest/Contents/MacOS/CURLHandleTests, 0x00000109)
dyld: loaded: /Caches/DerivedData/CURLHandle-flfotwdrtmazzmdpxjznosnizjlm/Build/Products/Debug/CURLHandleTests.octest/Contents/MacOS/CURLHandleTests
dyld: loaded: /Volumes/titan/Users/Shared/Applications/Xcode/Xcode46-DP4.app/Contents/Developer/Tools/../Library/Frameworks//SenTest
....

This may give you a clue about which library or framework is failing.

Other References

The topic of libraries, frameworks and dynamic linking is a fairly complicated one in its own right.

If you encounter the problem that I’ve described above, the chances are that you’re working with a fairly large and/or complex project with multiple libraries, and you’ll already know a bit about dynamic linking.

If not, these links may help.

• Run-Path Dependent Libraries: This article explains how the system finds out where your libraries are at runtime, and how you can set up any frameworks or libraries that you build so that they can be hosted in the the /Library/Frameworks folder, or inside your app, or embedded in another framework, in a way which still allows them to be found.

• Mac OS X Debugging Magic: This tech note gives a lot of helpful advice on debugging.

• Friday Q&A 2009-11-06: Linking and Install Names: Mike Ash has also covered this topic.

• You can also use man dyld in the terminal to get a full list of the DYLD_ variables.

In Conclusion

This problem can be really tricky to solve.

Just knowing what the problem is would be a good start, but Xcode does it’s best not to tell you.

Of course, I have filed a radar. Feel free to dupe it.

Hopefully, if you’ve stumbled across this blog post whilst tracking it down, I’ll have pointed you in the right direction!

Well, 2012 turned out to be quite a busy year for me on the contracting front.

Some of the main highlights:

• Most of the coding for the iOS game Bag It and Bin It was actually done in 2011, but it finally came out at the end of the year, and I did a bit more work on an update during 2012. Written using Cocos2D, this was great to work on, and it was nice to get the chance to do a little game from scratch.

• On the other end of the spectrum, I spent some time in the summer wrapping up a C++ library and turning it into an Objective-C iOS framework for another client. This couldn’t be more different from writing a game, but designing code to be used by other programmers is also something that I enjoy, so this was also a nice little job.

• In the early spring, I started doing some MacOS X work for Karelia, which continued through the year (and hopefully on into 2013!). This work has covered all sorts of territory, from unit testing their libraries to some networking and user interface work for their main product Sandvox. They like to keep me on my toes…

• Finally, in the autumn I also started working on the SVG import/export component of Bohemian Coding’s excellent MacOS X vector graphics app Sketch. Once again, a very different area to work in, with a whole new set of challenges, but definitely fun. The SVG standard is a bit of a monster, and making software that can understand everything in the standard would be a monumental task, so we took a pragmatic approach, trying to identify the areas that are most often used and tackle them first. It feels a bit like painting the Forth Bridge, in that by the time we’ve done everything there will probably be a new version of the standard out :)

So a pretty varied year, and those are just the highlights.

The one down side of this is that the time I’ve had to spend on Neu, Ambientweet, and some other as-yet-unreleased projects of my own, has been somewhat limited. I really hope that I can rectify that in 2013, and give them a bit more love.

The only problem is that the work for Karelia and Bohemian is continuing, and I’m enjoying it too! If anyone invents a time machine, please let me know…

Unit testing is great, but when you’re working on networking code, it can be a bit of a pain in the, erm, transport layer.

I’ve been doing quite a bit of network-related code for Karelia recently, and I wanted to be able to unit test, so this became an issue for me.

If there are two things that you definitely want from a suite of unit tests, it’s that they run fast, and that they are consistent. You want to be able to fire off the tests regularly, you want to be able to trust the results, and you want to be able to run them anywhere.

Throw a network server or two into your testing mix, and these bets are generally off. If the server is internal, the tests may not run when you find yourself on the wrong side of a firewall. If the server is public, you probably have no control over it. In any case, you have no control over the bit of soggy string between the test machine and the server, and any number of problems could cause your tests to fail spuriously.

This problem gets even worse when you consider that one of the best uses of unit tests is to check that your code copes when something goes wrong. Networking code is very good at going wrong in exciting and unexpected ways. Making real servers go wrong on demand in exactly the same way each time so that you can check that you handle it is challenging.

Typically the solution to this problem is some sort of “mocking” - i.e. finding some API or object that you can swap out or inject into the system in such a way that your tests can work as if they are talking to a real server, whilst actually talking to something else.

One approach is to somehow intercept the network request and, instead of passing it on to a real server, fake up the relevant response and return it directly. In the Cocoa world, a custom NSURLProtocol is a good way to achieve this, and it’s the approach taken by things like Nocilla. If you can work at such a high level, this is a nice solution as no actual networking need ever take place. You just intercept the URL request and deliver back an NSData or NSError object depending on what you’re trying to simulate.

The only problem for me was that this approach relies on the networking going via an NSURLConnection, and not all of the code that I wanted to test did this - I was testing code using libcurl, for example.

So I needed another approach, which would work with any networking code without me modifying it.

The second solution is essentially to implement a real (albeit very cut down) server. Bind a local port, listen on it for a connection, and respond to it as if you were a server.

Providing that you’re happy to require that your tests all connect to a custom port on localhost, rather than to real addresses, this solution is infinitely flexible.

There’s only one problem… the bit where I said “respond to it as if you were a server”. Isn’t writing servers a bit tricky?

Well, yes, if you want to write an actual server. But if you think about it, most interactions with servers are just a series of query, response, query, response. In a testing scenario you know what network operation you’re going to do, so you should be able to figure out what query the underlying code will actually send to the server. All you need to do then is to arrange for the right responses to come back.

Even better, in a testing scenario you generally only have to deal with one request happening at once (not always quite true, but near enough a lot of the time).

Subsequently I’ve discovered that there are a few solutions out there that do this sort of thing, but often they seem to just pretend to be a particular kind of server, most commonly a web server. I needed something that could pretend to be an FTP server, a web server, do WebDAV, maybe do SFTP, etc.

More to the point, being a reckless sort, I thought: how hard can it be?

I decided to find out, and thus MockServer was born.

MockServer

In a nutshell, the way MockServer works is this:

• at the start of your unit test, you make a KMSServer object, and give it a responder (more on these later)
• you then fire off the network operation that you want to test, but instead of pointing it at a known port like 80 on a real server you point it at localhost, at the port that the KMSServer object told you to use
• you now tell the KMSServer object to run until your operation is complete - this pauses your test
• once your operation has completed (or failed), your normal completion or error routines will be called
• in your completion routine you tell the KMSServer object to pause
• pausing the KMSServer object returns control to your test, and you can check for the right responses.

Internally, MockServer just opens up a port and listens for connections.

When it gets a connection, it opens up a socket and waits for input.

When it gets input, it passes it off to a responder, which matches it against things that it knows about, and sends back some commands. Mostly of these commands are just “send xyz back to the client”, but they can also include other things like “wait for x seconds”, or “close the connection”.

There is a bit more cleverness going on (for example, MockServer can fake the second connection that an FTP transfer requires), but essentially that’s it.

Responders

As I mentioned above, for the purposes of testing, most server interactions are just a series of query, response, query, response.

MockServer models this query/response behaviour using an abstract KMSResponder class - an instance of which you give to the server object at startup. So depending on the actual responder class, you can pretend to be any kind of server you like.

In real life some servers rely on a lot of state in order to give the correct response to a given query. In theory a responder could keep any state it liked, to deal with this problem. You could even genuinely implement an actual server protocol as a responder class if you were mad enough.

However, for testing this often isn’t necessary. MockServer supplies the KMSRegExResponder class, which is based on performing regular expression pattern matching on the input that it receives, and choosing from a series of stock responses.

It turns out that you can cover an awful lot of ground with this approach alone. Especially when you add in the ability to perform some limited substitution into the stock responses - things like inserting the current time in the correct format, or including part of the query you matched in your reply.

Responses

So now the meat of testing a particular network interaction comes down to identifying the queries that are going to arrive at the server, and arranging the right responses to send back.

This is where some low level knowledge of the underlying network protocol becomes useful, but even here it turns out that a lot of work can be done by trial and error.

You can configure MockServer so that it will log out everything it receives, so when it comes to figuring out the query, you can literally watch to see what arrives, and then write a pattern to match it.

Working out what to send back is slightly harder, as you can’t just watch MockServer, you have to actually do something yourself. The trick here though is to watch a real server, either by sniffing a network connection with it, or by using a tool such as Transmit and examining the log, or even by opening up a telnet session with the real server and pretending to be the client.

The good news is that a lot of this work only needs to be done once or twice for each kind of server.

You ask yourself “how does an FTP client send the password, and what does the server do?”.

Once you know that the client sends PASS thisismypass, and the server responds with 230 User user logged in. or 530 Login incorrect., you can just set up these a couple of alternate responses to match against a pattern like PASS (\w+). Normally you’ll want the first response, but if you want to test your error handling for the wrong password, you use the second one instead.

To support this sort of thing, MockServer provides the KMSResponseCollection class, which lets you define a series of patterns and responses in a JSON file, and then group them into higher level sets of responses.

So far in the course of my own tests, I’ve developed a couple of the JSON files - one for FTP and one for WebDAV, with quite a few useful responses in them. These files are included with MockServer, and can be used as the basis for your own tests. As time goes by I expect that I’ll add more response files for other protocols.

An Example Test

This is a simple example of how you might set up an FTP test, using the FTP responses supplied with MockServer.

#import "KMSTestCase.h"
#import "KMSServer.h"

@interface ExampleTest : KMSTestCase
@end

@implementation ExampleTest

- (void)testFTP
{
    // setup a KMSServer object, using the ftp: scheme and taking the "default" set of responses from the "ftp.json" file.
    if ([self setupServerWithScheme:@"ftp" responses:@"ftp"])
    {
        // set up the data that the server will return
        NSString* testData = @"This is some test data";
        self.server.data = [testData dataUsingEncoding:NSUTF8StringEncoding];

        // setup an ftp request for an imaginary file called "test.txt", on the local port that
		// our mock server is running on
        NSURL* url = [NSURL URLWithString:[NSString stringWithFormat:@"ftp://user:pass@127.0.0.1:%ld/test.txt", (long)self.server.port]];
        NSURLRequest* request = [NSURLRequest requestWithURL:url];

        // perform the request using NSURLConnection
		__block NSString* string = nil;
		[NSURLConnection sendAsynchronousRequest:request queue:[NSOperationQueue currentQueue] completionHandler:^(NSURLResponse* response, NSData* data, NSError* error)
		 {
		     if (error)
		     {
		         STFail(@"got unexpected error %@", error);
		     }
		     else
		     {
		         string = [[NSString alloc] initWithData:data encoding:NSUTF8StringEncoding];
		     }

		     [self pause]; // this call causes runUntilPaused to return
		 }];

		[self runUntilPaused]; // we hang here until the completion block above has executed

        // check that we got back what we were expecting
        STAssertEqualObjects(string, testData, @"got the wrong response: %@", string);
    }
}

Obviously not all tests will use FTP, and not all tests will use NSURLConnection. Some networking APIs may be callback based, or require delegates. As long as you call [self pause] in your callback or delegate method, the test should behave correctly - waiting at the runUntilPaused call until the callback or delegate method has fired.

For simplicity in this test we encoded the user name and password into the request URL. More complicated examples would probably rely on some sort of NSURLCredential callback or delegate method (such as implementing connection:willSendRequestForAuthenticationChallenge:), as a way of obtaining the details to send to the server. The KMSTestCase class has handy user and password properties that you can use to store the details that your authentication method can pass back. This lets you set this information on a per-test or per-request basis, as a way of exercising the different code paths for authentication success and failure.

For More Information

This post is just a quick overview, but there is also a fair bit of documentation which you can find over on github.

Also, of course, you can look at the code.

I wrote this code for Karelia as part of some other work. Many thanks (and much kudos) to them for allowing me to open source it.

In my Mac Pro Update - Think Different post a few weeks ago, I said that I thought we don’t really need a great big new Mac Pro from Apple.

What we need instead (and I admit that I don’t have an exact technical solution for this problem, but nevertheless) is a unit or units that can seemlessly enhance the processing power of a laptop when it’s on our home network.

You could argue that there’s still a need for a big box that can host all sorts of large expansion cards, but I’m not really convinced. Thunderbolt should deal with most of that.

Something else has occurred to me recently.

People have been talking about a potential switch away from Intel to ARM for the Mac range, and assuming that it would happen first with a Mac Mini or an iMac, since they are seen as “consumer”, and there’s a perception that consumer machines will be easier to switch over.

I actually think that the opposite will happen. I think that the first ARM Mac is likely to come in the guise of a new “Pro” desktop box. I also think that it’s likely to be loaded to the gills with cores, to the extent that it blows away anything out there in terms of raw processing power.

With GCD / blocks / NSOperation, Cocoa has a really nice clean model for making use of multiple cores. Better than anything I’ve used before (although admittedly I’m now a bit out of date with regard to the competition).

With the expansion of privelege separation and things like XPC, we’re also moving closer to a world where even a single app could be run seamlessly across multiple machines.

It seems to me that switching to a relatively low power, low cost, risc architecture makes perfect sense in this world.

Before you ask - I’ve absolutely no evidence for this. Just thinking aloud…

Simon Wolf wrote a blog post recently talking about passing NSError pointers into methods like this:

NSError* error = nil;
[something someMethod:@"blah" error:&error];

He talked about the fact that when you do this, you should always check the result of the method before using the error. In other words, you do this:

NSError* error = nil;
if (![something someMethod:@"blah" error:&error])
{
	NSLog(@"got error %@", error);
}

and not this:

NSError* error = nil;
[something someMethod:@"blah" error:&error];
if (error)
{
	NSLog(@"got error %@", error);
}

This is good advice. The method you’re calling isn’t guaranteed to set error to a non-nil value if nothing goes wrong. It might leave it alone, but it might set it to a temporary value, even though it eventually returns an “ok” result.

However, Simon then went on to talk about this line at the top:

NSError* error = nil;

His assertion was that by setting error to nil, you’re indicating that you intend to test it after the call - in other words that it’s an indicator that you don’t know what you’re doing.

He suggested that you should just do this:

NSError* error;

Strictly speaking, this is correct. If the method you’re calling follows the implicit contract for such functions, this should be fine.

However, I always set the error to nil before I pass it in, even though I never test it after the call. Why do I do this?

Consider the following method:

- (BOOL)someMethod:(NSString*)string error:(NSError**)error
{
	if (error)
	{
		if (something)
			*error = [self doSomeStuff];
		else (somethingelse)
			*error = [self doSomethingElse];
		//... lots of other cases here...
		
		//... later
		
		if ([(*error) code] == someCode)
		{
			[self doSomeExtraStuff];
		}
	}
	
	return YES;
}

This is a bit of a contrived example, but the point is that this method might be the one you’re calling, and it might be in a third party library that you have no control over.

Now lets imagine that some refactoring or other code change introduces a bug where *error doesn’t get set in the first block of if/else statements. Let’s also imagine that the developer has the relevant warnings turned off, and doesn’t noticed (shock horror - some people actually do this!).

What happens when the code starts accesses *error? Well, it depends what’s in it. If you did this:

NSError* error = nil;

you’ll be fine.

If you did this:

NSError* error;

you get undefined behaviour, based on what happened to be on the stack in the position that the error variable occupies. It’s a classic uninitialised variable bug, and potentially a bastard to track down.

At this point you may well be saying “if the quality of the library you’re using is that bad, you’ve only yourself to blame”. You might have a point, but bugs do creep in, even to good libraries.

Admittedly too, when you’re accessing an uninitialised Objective-C pointer, you’re way more likely to crash straight away than you would have been if you were just dereferencing a pointer to read a member in C/C++.

However, all of this is the reason why I still do:

NSError* error = nil;

even though I’m not going to test error afterwards.

You could call it paranoia, but don’t mistake it for misunderstanding the NSError** contract!