Kiwi - BDD with an identity crisis

BDD   kiwi   testing   iOS  

BDD or Behaviour Driven Development is a hot term with management. It’s sold as a methodology which directly maps the requirements of the business to working code via the use of “Stories”. The basic flow looks like this:

  1. The business requests a feature of the app to be developed.

  2. A Business Analyst translates the request into a Story. A Story details who is doing something, what they are doing, and what the expected outcome is. Stories are written using an English like language easy readable and understandable by non-developers. ie. the customer.

  3. The developer uses a special tool which reads the Story’s text, and maps it to actions on the app. Taps, clicks, text entry, etc. He also maps the outcomes to code which validates what the app does.

  4. The developer then modifies the app to make the story work. This is very much like another common practice known as TDD or Test Driven Development where the test is written before the code to make it work.

  5. When everything is ready, the developer can then show the customer the running BDD test so the customer can see the feature working.

One of the most popular tools around for doing BDD is Cucumber. It’s a tool written in Ruby and run from a shell or command line. It’s easy to learn and use, and is very flexible. Cucumber mandates that Stories and the test code are stored in separate files. This helps by keeping the stories accessible for non-developers and much easier to read. Cucumber is so flexible it can pretty much be used on anything.

Another tool is Kiwi. I’ve come across Kiwi a couple of times now. It tends to turn up in projects where the devs have decided they need to use a BDD tool, probably to impress the clients, and don’t want to learn Cucumber or Ruby. Kiwi takes a very different approach to Cucumber.

Kiwi:

  • Stories are written in with the code that maps them to the app.
  • Everything is written in Objective C.
  • Attempts to make use of XCode's builtin Unit Testing tools.

Having used it a couple of times now, I’ve come to the conclusion that it was a good idea gone wrong. The good idea - making Objective C available for mapping stories to app functions. The bad idea - how it was implemented. Here’s why:

  • Kiwi expresses Stories as strings within Objective C code. Customers don’t understand code so there is an immediate barrier to them being able to both read or supply stories. With Cucumber files a customer can easily type the stories themselves.

  • Source code to map a line of story text is followed by the Objective C code to action it. Bad idea for several reasons:

    • It trashes the ability to have the same story text in a number of stories all mapped by a single mapping. IN other words, everything is hardwired together.

    • It visually makes it difficult to read the story text because you have to read around all the Objective C code.

  • Stories are wrapped up within features, all of which are written in a single method. The result of this is extremely long methods. Often just one huge one.

  • Kiwi makes use of XCode’s test runners by making use of a very old and ill considered idea of dynamically creating “Test suites” so that Kiwi tests pose as unit tests in XCode. There are numerous problems with this:

  • You cannot see what tests you have until after you run them because XCode doesn’t recognise Test Suites until you execute a test run. In contrast, XCode immediately recognises individual Unit Tests the moment you write them.

  • You cannot turn off tests. Developers often just want to run a single test to debug it. Trying to do this with Kiwi is near impossible because Test Suites bypass XCode’s test enabling mechanisms. Even XCode 5’s new tools are fooled by Kiwi.

  • Tracing problems with a test is difficult because Kiwi’s own classes pollute the call stacks. It’s fails to tell you where an exception was generated, merely pointing at Kiwi code which of course, is not where the error came from.

  • When looking at test run results it’s very hard to tell what tests have failed. In order to pose as unit tests, kiwi creates massively long mangled names for the tests. In the XCode interface, it can be quite difficult to see the name and therefore work out which test has the problem.

  • Making use of Kiwi UI interaction functions to simulate taps and other interface actions introduces long delays in the tests. This is fine for small test runs, but if you want to test a lot of things, the runs can get very long. Plus if you write Kiwi tests in the same target as your unit tests, you unit tests will now take a long time to run, waiting for the Kiwi tests to finish. BDD tests and Unit tests should never co-exist.

So the summary is this, Kiwi is not a good idea. It messes up the parts of BDD's key features, it’s a poor fit within XCode, difficult to control, it's implementations are long and messy, and it doesn’t play well with unit tests.

So why the identity crisis? Simple, Kiwi is a BDD style framework, attempting to work like a unit test framework. It doesn’t know what it is.


Disclaimer: Some time ago I looked around at BDD frameworks for testing iOS apps and concluded there was nothing that suited my needs. So as a learning exercise and to potentially fulfil a need I wrote my own BDD tool. Similar to Cucumber it keeps stories in separate files. It uses Objective C methods to map each line of the story to actions on the app. And it can be run from the command line. So very Cucumber without the Ruby side. It took much longer than I thought and after writing it I looked around again and decided to stop working on it. The main reason was by that time, Frank (based on Cucumber) had made great strides and I was not able to keep up with it in terms of development. So despite not being Objective C, which I would prefer, I currently recommend Frank as the tool to use. Kiwi was also one of the reasons I decided to write my own.


Comments powered by Disqus