Mobile testing with Calabash

A long long time ago, when mobile devices were small and j2me was the only one adequate way to write mobile applications, bugs were happy. But then iOS and Android came to a place and everything changed. Applications became more complicated, smarter and got their own business logic. Of course the smarter application became the more attention we need to focus on the support of the software quality and the main instrument that can help us with that is testing.

But what should we test? Usually in mobile development we use 2 kinds of testing: unit-testing and acceptance testing. After your unit tests will pass you'll know that your code is written according to a specification (or not). After acceptance tests will pass you'll know that your product is written according to a specification.
So that is the main difference: unit testing - for code and acceptance testing - for product.


It is wrong to compare these kinds of testing. In a perfect world it is a good idea to write both tests for your application. In real life this may be quite difficult. Refactoring and changing requirements is a normal process of agile development process. Unfortunately the more tests you have the more time you need for their updating (surprise!). And time is money.

I am fan of behaviour-driven development (BDD) methodology. According to it you should write tests first and only after that you start coding. When I start working on some issue I'm writing some user scenarios. As they are user scenarios they contain no information about classes and internal realization. To find a bug (like a user) you need to be like a user. So my tests are as simple as:

Feature: Login
Scenario: User can login to the system
Given I am on the login screen
When I enter test user credentials
And I press "Login" button
Then I am in the system

As you probably have already guessed this is the acceptance test. It is written in a language that is called Gherkin and that is the language which Calabash speaks.

Calabash is a tool for mobile application testing. Is supports iOS and Android platforms. Unfortunately this does not mean that your tests will be cross-platform out of box. Tests sharing is still possible, but it is a topic for a whole another article.

To install Calabash you need to enter in Terminal:

gem install calabash

After that you'll need to add Calabash library to your mobile app project. How to do this? Well, actually this depends on your language and a platform for which you are developing. E.g. if you are native iOS developer then you can find manual for including Calabash here. If you are native Android developer then lucky you - no additional steps needed. If you are .Net developer and work with Xamarin, then just add a Calabash component to your iOS project and write this lines:

#if DEBUG
Xamarin.Calabash.Start();
#endif

somewhere in the app initialization. Please note the first and the third lines. You need this condition in order to prevent Calabash linking with your release version of application.


What next?
Calabash has 2 modes: discovering and testing.
Discovering mode may be helpful during tests creation. For example you have an iOS project with Calabash included and properly initialized. To start discovering just open this project root folder in Terminal app and type:

calabash-ios console

 If everything is ok you'll get an irb session. To make some magic just type:

start_test_server_in_background

After that simulator will start and you will see your application. Console is waiting for your commands, so let's start with:

query("*")

This command will show every component and property that Calabash "sees". You can find list of all Calabash commands here.
Just play around with your app and you will love Calabash!

After a while you'll want to save your tests. In order to create all needed directories and files just write:

calabash-ios gen

in a project root folder.

After that open "my_first.feature" file from the new "feature" directory (I highly recommend to use Sublime Text with cucumber syntax highlight plugin). In that file you'll find a sample test. Calabash ships with a set of predefined steps for testing. Now try to modify this sample test with predefined steps. After you finish you will be able to test your tests (got it? :-)) with the following command:

cucumber

I think now you know enough to start playing with Calabash. For the next steps I would highly recommend to go through Wiki of the calabash-ios and calabash-android projects and to read a great book "The Cucumber Book: Behaviour-Driven Development for Testers and Developers" by Matt Wynne and Aslak Hellesøy.

Next I would like to tell you about Xamarin Test Cloud platform, or about approaches for tests reuse among several mobile platforms, or about Calabash analogue for Windows Phone apps testing. 
But maybe next time. :-)
Good luck!