The state of XCode Unit Testing

In the world of iOS development, unit testing is one of the most talked about things, yet to my surprise, something that is almost never done. Perhaps the state of XCode's unit testing support has something to do with this, perhaps it's developers trying to finish projects within deadlines which are more of a clients wish than a practical target. Perhaps there are just more lazy developes out there than I think.

Whatever it is, unit testing is often treated as something of dubious benefit, and XCode has not helped, often giving the impression that Apple don't consider it that important.

Pre Xcode 5

Previously XCode has one option for writing unit tests. SenTestKit (aka OCUnit) and it wasn't that great. It covered all the core functionality required for unit testing and you could write tests in it just fine, however it's integration with XCode was rather crude. Developers need to enable/disable tests easily whilst they are writing code and the ability to run just the current test is a must when trying to fix an issue. OCUnit's integration didn't have any of these features. At least, not that easily accessible.

So when I started with iOS development I soon went looking for something else and came acoss GHUnit. Functionally not much different to OCUnit, it did however have it's own UI which made it very easy to find and run individual or groups of tests. GHUnit became my "Goto" tool for testing as I found I could do a lot of things with it, up to and including testing UI widgets. It wasn't integrated with XCode and you had to create a new target, but other than that it just worked.

Then Xcode 5 came out.

Post XCode 5

Xcode 5 has a whole new revamped unit testing framework called XCTest.

In terms of writing tests there is really nothing different between OCUnit, GHUnit and XCTest. In fact they are so close that chaning from one to the other is simply a matter of switching #import ... statements, asserts and tweaking the odd setup method.

Integrating with XCode 5 is where the differences shine. UCUnit sucked, GHUnit does it's own thing (very well), and XCTest has whole new tab and other integration with XCode 5. Not to mention integration with Apples CI server software, which BTW, is a free download from the developers center for registered developers. XCTest makes it easy to run one or many tests. So Apple appears to be understanding that developers actually want to do these things.

I've been using XCTest for a few months now and I think I understand it enough to outline what I think are it's strengths and weaknesses.

XCTest strengths

First and foremost would have to be it's integration into XCode and command line. Thats really all that makes it stand out and simply put, thats enough to put it on top, making it really easy to use. Another of it's strengths is that you don't have to switch targets when running tests (Units).

XCTest weaknesses

Perhaps the most obvious weakness of XCTest is documentation. There simply isn't that much around. Yeah, I know it's a unit test framework and when you have seen one you have seen them all. But more documentation on it would be nice.

XCTest also has a couple of really nasty bugs in it. I've reported these to Apple so lets hope they get fixed. but before I look at those ....

Unit - Functional - the differences

When it comes to unit testing in XCode, most people don't seem to understand that there are two different types. Apple's documentation is particularly poor in detailing the differences and in a lot of situations there is no really obvious difference. But in fact there are some quite significant ones which are important to know about.

Unit tests

  • Setup using a test target with a Target of None. The lack of a target indicates unit testing.
  • You don't need any Target Dependencies setup.
  • Include all app code and unit test code in compilation step.
  • Starts the simulator but does not install or run an app.
  • Ideal for very fast low level deep tests that do a lot of mocking and numerous variations on parameters when calling methods.
  • Are not good for testing the UI or large chunks of code as a whole (Black box style).
  • Are your first line of defence against bugs.
  • Can test out functionality on code that would normally lock a finite system resource. For example, a Unix socket. Simply because the app is not running.
  • This test target is normally the one you want to associate with you main build target so that you can simply execute them using a keyboard shortcut quickly and easily.
  • Have a couple of nasty bugs in XCode 5 which require workarounds when dealing with resources. See below.

Functional tests

  • Setup a test target with a Target set to your app. The presence of your app as the target signifies functional tests.
  • Need your app target setup as a Target Dependency so it is compiled before the test code.
  • Exclude app code as your app will actually be running. You can also exclude a lot of dependant libraries in the link step as they will have been linked into your app.
  • Starts the simulator, installs your app and starts it.
  • Ideal for testing UIs and functionality which involves large parts of your app. Also good for testing section of you app which are dependant on external resources not readily accessible to unit tests. Media libraries is one example.
  • Are often considerably slower that unit tests. This is one main reason to seperate them from your unit tests.
  • Are the second line of defence, usually by this point your unit tests have hammered the edge cases and corners of individual classes. So these should be testing a higher level functionality.
  • Because the app is running, don't have the bugs that unit tests have.

Generally speaking but Unit tests and Functional tests are useful to an app. Units for fast detailed tests, Functionals for higher level testing.

XCTest's Unit testing bugs

Here are the two bugs I have found when writing unit tests using XCTest and the workarounds I'm employing the deal with them.

NOTE: These bugs only occur when doing Unit Tests!

XCTest unit tests bug 1: Incorrect bundle (Apple bug #15306667)

There are very few iOS projects that don't call [NSBundle mainBundle] at some time so this bug is really nasty. In essence what happens is that the [NSBundle mainBundle] call returns the XCTest program bundle instead of the bundle created when your tests are compiled. So any resources such as graphics, xibs, data files etc that you have included in your testing bundle are not visible via this call.

There are two workarounds that I know of for this. The first is that you change

[NSBundle mainBundle]

into

[NSBundle bundleForClass:[self class]]

Works, but you cannot be sure that the developers on your project will remember to do this. And frankly, I'm not even sure that I remember to do this. So it's not much of a solution. Plus if you are working with a legacy code base, or third party APIs, you may not be able to easily change the code.

So here is the second fix which you do within the unit test code. This fix relies on OCMock which is my "goto" library for mocking. Look it up, it's really good.

static id _mockNSBundle;

+(void)setUp {
    _mockNSBundle = [OCMockObject niceMockForClass:[NSBundle class]];
    NSBundle *correctMainBundle = [NSBundle bundleForClass:self];
    [[[[_mockNSBundle stub] classMethod] andReturn:correctMainBundle] mainBundle];
}

What this does is create a local nice mock of the NSBundle class which then overrides the class method for mainBundle and tells it to return the correct unit testing bundle. In effect we are swizzling in a new method that does what we want.

It's a very effective solution and by putting it in the class setup of the unit test, we ensure it's always in place for all tests in the unit test class.

XCTest unit Tests bug 2: Too many objects returned when re-constituting a xib (Apple bug #16323674)

This one is also a major problem if you are dealing with UI code. Whenever your code loads a xib file, either manually or when dequeuing table cells, it makes a call to [UINib instantiateWithOwner:options:] to create the objects in the xib and return them in a NSArray. Most of the time you don't see this because it's behind such calls as [UINib loadNibWithNibName:owner:options:] or [UItableView dequeueReusableCellWithIdentifier: forIndexPath:].

In your app the instantiateWithOwner call returns an array of top level objects from the xib. But when running in a XCTest unit test additional objects are added to this array. Most annoyingly, at the front. So for example, if you code is dependant on one object being returned and accesses the array to retrieve it via objArray[0] then you will get a different result depending on whether you are running in the app or in a unit test. The number of extra objects returned in a unit test is either 1 or 2. If the passed "owner" object is nil, then 2 objects are added, if not then 1.

This bug will cause a unit test crash when testing table views as UITableViews have asserts for the number of returned cells in a dequeue. Instead of an array with one object, it gets 3.

To fix this we need to be able to analyse the number of returned objects in the array and correct them so the app code sees exactly the same result as if it was running normally. Unfortuneately this is not that simple. We need to allow the calls to occur before we adjust the results. This is different to a normal mocking situation where we completely override the call and return a predefined results.

So the guts of what we need to do is this

  1. Allow [UINib instantiateWithOwner:options:] to excute.
  2. Intercept the returned array and trim off the extra leading objects.

I tried a number of ways to mock through this with OCMock, but could not really get it to work. It's simply too complex due to the fact that we need to allow methods to execute and then intercept the results before returning them. It is doable, but not when combining calls to class methods and instance methods as we would have to do in this case.

The answer came about when I dropped attempting to mock through this and did a straight method swizzle to override the [UINib instantiateWithOwner:options:] method. Here's the code

#define UINIB_ORIGINAL_INSTANTIATE_SELECTOR NSSelectorFromString(@"originalInstantiateWithOwner:options:")
#define UINIB_INSTANTIATE_SELECTOR @selector(instantiateWithOwner:options:)

@interface MIBTestCase (_hack)
-(NSArray *) replacementInstantiateWithOwner:(id) ownerOrNil options:(NSDictionary *)optionsOrNil;
@end

@implementation MyTestCase 

+(void)setUp {
    Method originalMethod = class_getInstanceMethod([UINib class], UINIB_INSTANTIATE_SELECTOR);
    class_replaceMethod([UINib class], UINIB_ORIGINAL_INSTANTIATE_SELECTOR,     method_getImplementation(originalMethod), method_getTypeEncoding(originalMethod));
    Method replacementMethod = class_getInstanceMethod(self, @selector(replacementInstantiateWithOwner:options:));
method_setImplementation(originalMethod, method_getImplementation(replacementMethod));
}

-(NSArray *) replacementInstantiateWithOwner:(id) ownerOrNil options:(NSDictionary *)optionsOrNil {
    NSArray * objs = objc_msgSend(self, UINIB_ORIGINAL_INSTANTIATE_SELECTOR, ownerOrNil, optionsOrNil);
    NSAssert([objs count] > 0, @"Zero objects returned from xib, are you sure it's in the bundle?");
    int objStartIdx = ownerOrNil == nil ? 2 : 1;
return [objs subarrayWithRange:NSMakeRange(objStartIdx, [objs count] - objStartIdx)];
}

@end   

Essentially what we are doing here is this

  1. Create a new method in the UINib class with a copy of the implementation of instantiateWithOwner:options:. This is so we can call the original code to read in the xib.
  2. Replace the implementation of instantiateWithOwner:options: with a custom one that calls the original implementation we have stored, gets the results and extracts from them the objects we want to form a new return array.

Now that we have this in place, all calls to read in xib files should behave the same way in unit tests as they do within your app.

Clean up

As part of ensuring tests don't interfer with each other I also have a tearDown method to clean up the mocking and swizzling I'm using.

+(void) tearDown {
    [_mockNSBundle stopMocking];
    _mockNSBundle = nil;
    Method originalMethod = class_getInstanceMethod([UINib class], UINIB_ORIGINAL_INSTANTIATE_SELECTOR);
    Method hackedMethod = class_getInstanceMethod([UINib class], UINIB_INSTANTIATE_SELECTOR);
    method_setImplementation(hackedMethod, method_getImplementation(originalMethod));
}

Summary

Where I'm at is that I'm now using XCode 5 and XCTests, along with OCMock for dealing with mocking. With the two hacks I have above I can now rely on the code I am testing behaving the same way as it does in the app. it's a shame that these two severe bugs are in XCTest because it's a nice tool to use. Hopefully Apple will fix them if they get enough pressure from developers. So if you want to help out, register a bug report. If Apple gets enough of them they will respond.


Comments powered by Disqus