Quick and Nimble - Sometimes...

iOS   testing   xcode   BDD  

Some time ago I wrote a post trashing Kiwi (At least I'm honest about it!), a UI testing framework the uses dynamically generated tests to validate your iOS apps. Kiwi follows a BDD style of testing and at the time my opinion was not very complementary.

Give that, I was very hesitant to allow a similar tool to be added to the project I'm currently on. But the team wanting to introduce it were very adamant that the tool was far better than Apple's XCTest framework and the benefits would nullify the reservations I expressed.

Where they correct?

The two frameworks they wanted to bring in were Quick and Nimble. Quick is an alternative testing framework which works in a similar manner to Kiwi. Nimble is a framework of matchers designed to provide an alternative to XCTest's assertions. Whilst often regarded as one, Quick and Nimble are in fact two seperate products. They're just usually promoted as it they were one and you'd be forgiven for thinking there's a dependency between them.

Before

I'm not going to waste time going through how to install or use Quick and Nimble as there's plenty of blogs outlining that already. Use Google, you'll find them easily. In this post I'm interested in talking about the things those blogs ignore and trying to provide a more pragmatic view of Quick and Nimble.

But first lets just do a quick example of some tests written in Quick and Nimble vs XCTest so you can see what they look like at a glance.

XCTest example

In my experience XCTest is a pretty simple framework to learn and use. I've talked with developers who claim it isn't, but usually that's turned out to be more of a lack of general testing experience than XCTest being hard to use. Once people are familiar with it, there's no real thinking required to write hundreds or even thousands of tests.

import XCTest  
@testable import MovieAPI

class MovieTests: XCTestCase {

    private var movie:Movie

    func setUp() {
        super.setUp()
        self.movie = Movie(name: "Avatar")
    }

    func testMovieListsActors() {
        XCTAssertGreaterThan(1, movie.actors.count)
        XCTAssertTrue(movie.actors.contains("Sam Worthington"))
    }

    func testMovieReturnsDuration() {
        XCTAssertEqual("2:42", movie.duration)
    }
}

XCTest is fairly easy to follow. You start by importing the XCTest framework and then your project. Next you extend XCTextCase and write each test as a function whose name must begin with test.... Xcode automatically recognises these methods as containing tests. The setUp() method is an optional method that allows you to perform common setup applicable to all the tests, saving you having to repeat it in each test function.

Quick and Nimble example

Here's the same tests written using Quick and Nimble.

import Quick  
import Nimble  
@testable import MovieAPI

class MoviesTests: QuickSpec {

    override func spec() {

        describe("Testing the Movie class") {
            private var movie:Movie

            beforeEach {
                self.movie = Movie(name: "Avatar")
            }

            context("The actors list") {
                it("Should contains the actors") {
                    expect(self.movie.actors.count) > 1
                    expect(self.movie.actors.contains("Sam Worthington")).to(beTrue())
                }
            }

            context("The movie's details") {
                it("Should show the correct duration") {
                    expect(movie.duration) == "2:42"
                }
            }
        }
    }
}

The most obvious difference to XCTest is the use of function calls to give English like descriptions of the tests and the context within which they run. With Quick and Nimble, Quick provides the structure for the tests and Nimble provides the assertions to validate the results.

You start by inheriting from QuickSpec. Then filling out the spec() function.

The describe(...) {...} and context(...) {...} functions are used to group up tests and provide a scope for any setup executed by the beforeEach() {...} calls. The actual tests are contained within the it(...) {...} function.

When you run the tests, QuickSpec runs the spec() method and creates a series of dynamically generated XCTest functions using the descriptions from the describe, context and it calls to create the test function names. Xcode then sees these dynamically create functions and runs them like normal.

Test

I could go on for a large amount of time dissecting Quick and Nimble in detail, but most people aren't interested in that sort of detail and really just want to understand the day to day pros and cons. So here's my highlights:

  • Remember Quick and Nimble are not co-dependant. You can use them together or independently. It's up to you. Practically this means that if you aren't fussed on Quick, but quite like Nimble, you can use it with XCTest instead.

  • Quick does not auto-magically mean you are doing BDD or TDD, nor does it mean you're immune from writing crappy code or tests, or that your tests will necessarily be any clearer. It's really just a different way to write the same tests.

  • Quick's descriptive text arguments can provide an easier to understand description of a test than XCTest function names, but both are only as good as the effort you put into them.

  • Quick's design philosophy of using one huge function for everything in the test class runs against a number of proven programming practices such as the Single Responsibility Principle, KISS and keeping implementations manageable. It can trigger warnings from code quality scanners and be a pain in the rear to breakpoint and debug.

  • The ability to place beforeEach() {...} calls within describe and context scopes can facilitate more flexible test setups than XCTest's single setUp method.

  • And I can't emphasis this enough - Quick DOES NOT play well with Xcode:

    • The tests only appear in the test navigator after you have run them and will disappear when you clean the project.

    • You cannot click on a test in the test navigator or log to open it in the editor.

    • The 'run' arrows in the test navigator and code editor don't work.

    • To run a subset of tests, you have to change context to fcontext to run the tests in that context, or change it to fit to run individual tests. It may seem a minor thing, but it's surprisingly aggravating having to edit and 'un-edit' all the time. It's also easy to accidentally checkin a fcontext or fit resulting in your CI's not fully testing your app until someone notices the mistake.

    • The test method names generated by Quick are so long as to be almost useless when viewed in the test navigator. You can shorten the descriptions to address this, but that defeats the purpose of having the descriptions in the first place.

  • In almost every case where I've done a comparison like the above, the amount of boiler plate required to write Quick tests is greater than XCTests because of the nested levels of function calls.

  • Nimble's style of writing assertions can be easier to read that XCTest's XCTAssert... functions and it's toEventually(...) functions are much easier to use that XCTest's Expectations.

  • Quick and XCTest are NOT compatible. Quick's implementation disables a lot of Xcode's XCTest support, meaning it's effectively impossible to use Quick and XCTest at the same time. It's an all or nothing thing.

After Quick

When I see products like Quick, I invariable wonder why people are using them. Bloggers are a significant factor in this. All the posts I read concentrated on what the author perceived as it's strong points. They'd spend a lot of time and screenshots telling you how to setup it up and use it. Then completely ignore the Xcode integration issues, or admit that perhaps Quick might not be the best tool for all your testing needs. The way they promote it you'd even be forgiven for thinking that merely using Quick would somehow make you a BDD/TDD testing guru.

I suspect this is symptomatic of a lot of technology writing. The author concentrating on the positive aspects of the subject and avoiding anything that could be seen as negative. As such, some blogs feel more like technical infomercials than the honest examination I'd rather have.

A second though is simple human nature. The software industry tends to attract people who have drunk way too much 'Kool-Aid', promoting and pushing ideas that any sanity check will raise questions over. I suspect that Quick falls into this category, being pushed beyond it's niche by people high on it's coolness. This doesn't mean it's necessarily bad and there are situations where it may be a good idea to use, but it's hard to judge when it's being pushed as being superior to XCTest when in reality it's just a different way to do the same thing and has some serious issues.

After Nimble

I actually quite like Nimble. It's like Quick in that it's not offering anything that you cannot do with XCTest's assertions. It's just offering a different approach and some more concise alternatives.

Unlike Quick though, it doesn't have any nasty side effects, or force you into constructs that can compromise your coding practices.

Nimble is simply a neat little package of alternative assertions that are easier to read that XCTest's and can simplify some of it's more complex asynchronous code.

Finally

After using Quick and Nimble on a large project with some 3,000+ Quick tests I'm still no more convinced that Quick is a good idea than I was before it was added. Perhaps I'm just old and cranky, or perhaps I just like frameworks that I feel actually make a positive difference.

To some degree it's not Quick's fault. Apple doesn't provide any method for 3rd party tooling to integrate with Xcode and if Apple opened up Xcode for 3rd party tools and Quick used a collection of smaller more concise functions instead of a single huge function, I'd probably have quite a different opinion. But as it is, I find it's limitations and iffy practices far out way the few positives it offers.

Nimble on the other hand, I'll use again.


Comments powered by Disqus