Evaluating a tool for creating automated tests for mobile telephone applications

By
Marie Kyletoft
– 2013-09-17Posted in: Artiklar

handsets

Introduction

This article is not going to be an argument for or against test automation for mobile applications. I will assume that you have already made that decision and are now wondering which of the many solutions available to select. This article will not cover any of the points on tool selection in general (such as cost, available support or IDE integration to existing tools). The focus in this article will be on the issues specific to mobile applications testing.

However if you would like some pointers on how to select a tool in general then try the following links:

http://www.softwaretestinghelp.com/choosing-automation-tool-for-your-organization/

http://www.softwaretestingtimes.com/2010/04/selecting-software-testing-tool-testing.html

 

The first question…

The first question that arises is whether you need to test on real handsets or whether it is ok to go with an emulator. There are hundreds of emulators out there of all shapes and sizes, but it all boils down to can you rely on the emulator to behave as a real handset. Many of the functions that you may need to test such as application behavior when connected to a network, or running out of battery, may or may not be available (this list is changing all the time). Consider how the screen looks and feels in real life rather than on your workstation?  Will this be important for your application, in all likelihood it will.

  • Ø For some interesting links google ‘emulator vs real handsets’

Also bear in mind that emulators suffer from the same problem as real handsets: in the world of Android each handset supplier has modified the basic OS and added/modified features to make their product more attractive and then there is the fact of how many versions and suppliers there are out there.  How many emulators are you going to have to install to keep up and can you rely on them to behave in the same way as the handsets themselves?

The consensus seems to be that emulators are fine for developers to get the basic bugs and issues dealt with, but before you go to market with your application it is going to have to be tested on real handsets.

The second question…

Assuming that you agree that you need to test with real handsets then the next question is whether you want to buy in handsets or whether you can consider renting them? This question is partly one about how much control do you want/need and partly a question of logistics and costs.

AndroidFragmentation

There are literally hundreds of handsets out there, different brands, different OS, different versions. How many handsets are you going to test on? Then remember that every quarter a new batch of handsets become available. (this situation seems to be getting worse not better: http://blogs.keynote.com/mobility/2013/06/fragmentation-continues-with-yet-another-os-firefox.html).  If it is important to cover as many of these as possible then you should seriously consider the rental option in order to delegate the logistics problem.

There are a number of suppliers of this type of solution; they buy in all the handsets, connect them to their system (more about that further on) and then give the tester access via the cloud, usually there is a reservation system in place so the tester can book and access the required handset at the required time.

The selling points of this kind of solution are that you do not have to think about:

  • Buying the latest models and disposing of those that are out of date.
  • You do not have to keep track of the handsets, where they are, the purchase requests etc.
  • You do not have to take care of storage, extensive maintenance and connecting.

This needs to be balanced against your company’s requirement for security. If your test data is very sensitive, for instance banking details, or your application is very ground breaking you may not want to take the chance on the supplier’s promises of security and customer data separation. In which case you have three options:

  • You use the private cloud option available from the supplier i.e. they still administer the handset procurement and connection for you but in a guaranteed separate network.
  • You buy the tool in order to manage your own cloud solution.
  • You manage your handsets yourself and you connect directly via USB to your workstation rather than remotely via a network.

The core technical questions …..

So if you are still considering an automation tool and have decided on the access method you can finally start looking at the technical questions. The main question in this area is how is the handset connected to the test system?

Here we have to deal with the phrase “Jail-breaking”. This is basically the legally grey area method for gaining access to iOS; so you can get around Apple’s restrictions on what you can install and modify. For a more detailed description see http://ipod.about.com/od/iphonesoftwareterms/g/jailbreak-definition.htm. The reason I describe this as a legal grey area is that it is illegal in the US, but a three year dispensation is currently in place (October 2012, the Library of Congress released its latest exemptions to the Digital Millennium Copyright Act (DMCA)).

The matching activity for Android is ‘rooting’ which means giving system administrator rights on the root level to get around manufacturer restrictions (see http://google.about.com/od/socialtoolsfromgoogle/a/root-android-decision.htm for more information). The first reaction to learning about jail-breaking is probably that you do not want to get involved with that kind of solution; and you don’t have to as nearly all the tool suppliers now have a non jail-broken solution.

So exactly how do these tools access the handset? There are three main methods:

  • Attach a video camera to the handset or ‘read’ the screen digitally and then interpret what is happening via optical character recognition (OCR) and picture recognition.
  • Modify the application by including extra code to enable the test tool to read and modify the inputs and output of the application. This is known as source instrumentation (for an overview of this method see: http://www.stridewiki.com/index.php?title=Source_Instrumentation_Overview).
    • For Android handsets this basically consists of modifying the manifest.xml file in order to modify the permissions i.e. put into debug mode.
    • For iOS this means adding a proprietary code library from the supplier to your application code before compilation.
    • Add an agent to the handset to enable the reading and modifying of handset system inputs and outputs.

The first method, OCR, is approximately where we were with web application testing 15 years ago, but do not write it off straight away as unnecessary. There may be times when OCR is the only way forward. You will have to analyse the application under test and actually test drive some scripts to see if OCR and picture recognition is required. A number of suppliers have this functionality and they will be trying to impress you by describing their algorithm as stronger than the competitors. This is very difficult to judge and I will leave that to the experts, but don’t believe any supplier who promises 100% guaranteed character recognition, and look for key phrases like “best match” rather than “exact”.

native vs web

 

Before talking about the second method a quick review of the terms for the mobile novices. There are two kinds of handset application: native (the kind of application that you buy on Play Store or Apple iStore) and web (the kind of application you access via the web browser on your handset). Each application is made up of things like text boxes and buttons, these are called objects. These objects can be identified by an ID and then manipulated by the code calling the object id and running a certain action. A test automation script has more chances of success (i.e. it runs every time on multiple handsets) if it can also call these objects and apply an action, as opposed to trying to identify, via OCR, each object every time it is referred to.

The second method, instrumentation, is a must if you want to test with Native objects. This is the only way to gain access to the object IDs in order to be able to manipulate them in a script. Of course if you are testing a web application then it is enough to just access the API as for a pc web application to find the object IDs. The issue is that currently a large number of applications are actually hybrids (combination of native and web applications) and the current trends seem to be leading towards an increase in this type of development, so make sure you select a tool that can handle both types of object recognition. To read more on what instrumentation is, how it works and why it is so necessary in the mobile environment then read: http://mobilelabsinc.com/mobile-application-testing-instrumentation-effects/.

The third method is also a type of instrumentation but the code is on the handset rather than in the application. For a description of an open source agent that can be installed on handset try: https://blogs.akamai.com/2012/03/open-sourcing-mobitest.html.

In conclusion when considering an automation tool ask the supplier in detail exactly how the tool is connected to the handset and how it identifies objects; can the tool recognize web, both old style and HTML5, can it recognize Native objects and hybrids? This may have a big effect on how your company develops mobile applications. (To read more about how developers view this issue try: http://wiki.developerforce.com/page/Native,_HTML5,_or_Hybrid:_Understanding_Your_Mobile_Application_Development_Options).

 

The functional must haves…..

Finally we can consider the tool functionality. There are a couple of things that the test automation must be able to cope with when testing handsets and those are:

  • Changing orientation of the screen
  • Incoming and outgoing calls and SMS in the middle of application usage, and other interruptions such as low battery warnings.
  • Swipe, zoom, drag and drop and other now standard gestures by the user.
  • Handset vitals, ability to see battery levels, memory usage, CPU usage etc.

If the tool under consideration doesn’t cope with these standard issues then stop looking at it. You will probably want to add to this short list commands or actions that are vital for your application.

The functional should haves…..

  • swipe

Although this article is about tools for test automation there really should be the possibility to carry out and record manual / exploratory testing with the same tool. If you are using a cloud solution this is a must otherwise you still need to have handsets on your desk. Another item to consider if you select the cloud solution is how will you execute the tests? A major saving in test automation is the ability to execute the tests when you are not at your desk so check with the supplier how the tests will be started and where the results are recorded, and how.

Finally many of these tools also include the possibility to expand into performance testing; which is a whole other article.

 

 

Conclusion

As stated at the beginning this is an article describing the key issues in selecting a an automation test tool for mobile applications, one which focuses on the handset specific issues rather than the more generic questions applicable to all tool selection initiatives. I have tried to steer a neutral course and not favour one supplier over another but this is hard to do as so much of the material available comes from the suppliers themselves. However, in working my way through this material I found it was easy to get lost in the varying terminology and level of detail which is why I have concentrated on the more basic difficulties; which type of supplier and which type of connection. I think that availability of commands (such as swipe) are a more simple issue that you can more easily deal with yourself.

Appendix

In order to help you on your tool evaluation I leave you with basic list of suppliers (not including open source) with their links as of August 2013.

The main suppliers of a cloud based solution are:

In addition the following suppliers can do both the desktop and the cloud solution:

The main suppliers of desktop solutions are:

 

 

About Marie Kyletoft

Jag är en konsult på Softronic med uppdrag som testledare. Jag ramlade in i testbranschen över 15år sedan, via TQM, ISO9000, processförbättring osv. Sedan dess har jag arbetat med funktionstestning och acceptanstestning, med embedded och web applikationer, i projekt med PRINCE och med SCRUM. Ganska varierad men det finns mer att lära sig därute.