Cucumber tests on iPhone/iPad

I am sure everybody has heard about Cucumber ( https://github.com/aslakhellesoy/cucumber) – a tool for Behaviour Driver Development where you describe software behavior in natural language that your customer can understand. Through step definitions these behavior descriptions are executed as automated tests. Cucumber serves as documentation, automated tests and development aid.

My friend and colleague Christian Hedin gave me tips on iCuke. Cucumber has been widely used for testing web applications, but now it’s also possible to test iOS (iPhone and iPad) apps with help of the iCuke library (https://github.com/unboxed/icuke). iCuke uses AppleScript to drive XCode in order to launch your application into the iOS Simulator. A preloaded library is used to inject a small HTTP server into your application. The HTTP server allows you to see an XML representation of the iOS device screen and to emulate input, such as taps, swipes and pinch gestures.

You can read more about it here:
http://www.unboxedconsulting.com/blog/cucumber-iphone-icuke
http://pragprog.com/magazines/2010-07/bdd-on-iphone-icuke

Really excited about Cucumber on iPhone we decided to give it a try. After installing iCuke and doing some test runs it was clear that some challenges had to be overcome to make this a truly useful tool for Behaviour Driven Development and automated testing of iOS apps.

Challenge 1: Screen returns the previous screen’s xml

In our test, we wanted to tap a button, come to another screen and expected to see a text. When the test runs the iPhone is driven to the correct screen, e can see the expected text but the test fails anyway. After little debugging we realized that the method screen.xml returns old xml directly after changing the screen.

Solution:

We needed to refresh screen before checking if the expected text is on the new screen. iCuke has a method to refresh screen but it is private and we could not use it. So we just added a new method to the existing ICukeWorld class.

Calling this method before checking for the presence of the text solved this problem.

Later on, I forked iCuke and added this and some other methods.

git://github.com/DavorC/icuke.git

Challenge 2: Timing

Everybody knows that using sleep and delays in code is not so flexible.

We want to check something on the screen and give it 3 seconds to finish its loading.

Is it enough? Maybe, maybe not. Screen content loading could take 0.1 second or 5 seconds or… – You know what I mean.

In the first case loading is finished quickly and we unnecessarily spend 3 seconds for doing nothing . If we have a lot of delays in our code then our tests would waste a lot of precious time.

In the second case the delay is not long enough and the test fails.

Solution:

We need to write some help functions to wait for different items which are expected: some text, a button, downloading spinner etc.

Example with wait for text:

As you can see the method is waiting for the text to appear and does checking every 0.1 second. As soon as the text is found the test continues. If, after given timeout, the text is still not found, the test fails.

I prefer to use unit test assertions in my tests (flunk is an assert which always fails). To use assertions with cucumber you need to add assertions to the Cucumber World:

Challenge 3: Different tappable object on screen can have the same text label.

Identifying objects on the screen only by text is not enough. By default, the first tappable object is tapped. What if we want to tap the second one?

Solution:

I created a set of help functions for:

returning all objects that satisfy some criteria
returning a specific object
waiting for a specific object
checking if a specific object exists

Objects are described in xml by: type, label, traits and index.

Here is an example of getting an array of all elements satisfying given attribute values.

Observe how it is easy to parse xml using ruby’s REXML library.

Of course I could write more generic methods and decrease number of code lines – something like:

but I like readability so I wrote a set of help functions with more specific naming:

get_element_by_type(type, index = 0)
get_element_by_type_and_label(type, label, index = 0)
get_element_by_type_and_traits(type, traits, index = 0)
get_element_by_type_label_and_traits(type, label, traits, index = 0)
get_all_elements_by_type(type)
get_all_elements_by_type_and_label(type, label)
get_all_elements_by_type_and_traits(type, traits)
get_all_elements_by_type_label_and_traits(type, label, traits)
get_all_static_texts()
get_all_labels_by_type(type)
get_all_labels_by_traits(traits)
element_by_type_exists?(type, index = 0)
element_by_type_and_label_exists?(type, label, index = 0)
element_by_type_and_traits_exists?(type, traits, index = 0)
element_by_type_label_and_traits_exists?(type, label, traits, index = 0)
text_exists?(text)
wait_for_element_by_type(type, index = 0, timeout = @@timeout)
wait_for_element_by_type_and_label(type, label, index = 0, timeout = @@timeout)
wait_for_element_by_type_and_traits(type, traits, index = 0, timeout = @@timeout)
wait_for_element_by_type_label_and_traits(type, label, traits, index = 0, timeout = @@timeout)
wait_for_text(text, timeout = @@timeout)
get_center_of_the_element(element)
tap_coordinates(x, y)
double_tap_coordinates(x, y)
tap_element(element)
double_tap_element(element)
tap_text(text)
double_tap_text(text)

In order to use these functions you need to use iCuke from:

git://github.com/DavorC/icuke.git

Don’t forget to use –recursive flag when you clone it:

After building and installing the iCuke gem you need to

instead of:

Example of usage

In your feature-file:

step definitions:

Testing both iPhone and iPad
If you’re testing a universal app that runs on both iPhone and iPad I recommend writing different scenarios for the platforms. iPad in landscape mode is most likely to reuse most of code you have written for iPhone. Place them in different feature files and tag them with e.g. @iphone respective @ipad tags.

In your env.rb file:

Use in your feature files:

Implementation:

If you wants to run iPad simulator in landscape mode:

where get_ipad_orientation is some application specific method to decide if the simulator is in portrait or landscape mode.

Running iCuke tests on Hudson server

It’s really nice to be able to run your test suite at given intervals, or when you commit to the source code repository. To run your iCuke tests on Hudson (which is a popular continuous integration build server) you must start your iPhone simulator from a terminal window. This is easiest to do by launching an AppleScript from Hudson.

Below are two scripts, one AppleScript and one shell script, that I used to run my iCuke tests from Hudson.

- run_cuke.scpt

- cuke.sh

Add these two scripts to your project.
Then, add this Cucumber hook to your env.rb file:

On Hudson, create a new job and copy the configuration from your project’s existing job.
Configure job:
add to description:

add build step (execute shell):

Of course you can change all scripts according your needs.
At the end you should have your iCuke tests running on Hudson and you will get both a nice test report in HTML and the debug output as a plain text.
There is a lot of potential for doing automated feature tests for iOS using Cucumber and with iCuke and the additions above you’ll hopefully be well on your way for doing BDD in your next iOS project!

5 Responses to “Cucumber tests on iPhone/iPad”

  1. Eva says:

    Hey, sounds interesting, i’ll try that test. here is another test for your iPhone/iPad.
    http://www.keepautomation.com/products/iphone_barcode/
    iPhone/iPad barcode generator. you can generate different kinds of barcodes for your iPhone/iPad.

  2. husnain says:

    Hi,
    Great work. I am a newbie and i am stuck on installing icuke on my Mac OSx can you please refer some step by step guide.
    Thanks.

  3. Jamie Kirkpatrick says:

    Good work there: your extra functions are really useful so thanks for adding those.

    Question: I’m playing with iCuke having worked with Frank which feels a lot less mature than this setup. One thing I’ve noticed is that the entire view hierarchy that is part of the window’s content view is returned when i query the screen – how are you determining whether something is really visible on the screen or not given that something could be hidden by another element?

  4. Excellent work!! I recently was looking into iCuke and noticed that the main repo (along with other forks) are no longer maintained. I am glad to see the project is at least being worked on by someone and I’ll be following your fork from now on!

  5. [...] This post was mentioned on Twitter by jayway, Fabien Devos . Fabien Devos said: RT @jayway: Cucumber tests on iPhone/iPad http://ff.im/-xMBnu [...]

Leave a Reply