Mobile application testing

Mobile application testing is a process by which application software developed for hand held mobile devices is tested for its functionality, usability and consistency.^[1] Mobile application testing can be automated or manual type of testing.^[2] Mobile applications either come pre-installed or can be installed from mobile software distribution platforms. Mobile devices have witnessed a phenomenal growth in the past few years. A study conducted by the Yankee Group ^[3] predicts the generation of $4.2 billion in revenue by 2013 through 7 billion U.S. smartphone app downloads.

Key Challenges in Mobile Application Testing

1. Variety of Mobile Devices- Mobile devices differ in screen sizes, input methods (QWERTY, touch, normal) with different hardware capabilities.

2. Diversity in Mobile Platforms/OS- There are different Mobile Operating Systems in the market. The major ones are Android, IOS, BREW, BREWMP, Symbian, Windows Phone, and BlackBerry (RIM). Each operating system has its own limitations. Testing a single application across multiple devices running on the same platform and every platform poses a unique challenge for testers.

3. Mobile network operators- There are over 400 mobile network operators in the world;^[4] out of which some are CDMA, some GSM, whereas others use less common network standards like FOMA, and TD-SCDMA. Each network operator uses a different kind network infrastructure and this limits the flow of information.

4. Scripting- The variety of devices makes executing the test script (Scripting) a key challenge. As devices differ in keystrokes, input methods, menu structure and display properties single script does not function on every device.

Types of Mobile Application Testing

1. Functional Testing- Functional testing ensures that the application is working as per the requirements. Most of the test conducted for this is driven by the user interface and call flows.

2. Laboratory Testing- Laboratory testing, usually carried out by network carriers, is done by simulating the complete wireless network. This test is performed to find out any glitches when a mobile application uses voice and/or data connection to perform some functions.

3. Performance Testing--- This testing process is undertaken to check the performance and behavior of the application under certain conditions such as low battery, bad network coverage, low available memory, simultaneous access to application’s server by several users and other conditions. Performance of an application can be affected from two sides:application’s server side and client’s side. Performance testing is carried out to check both.

4. Memory Leakage Testing- Memory leakage happens when a computer program or application is unable to manage the memory it is allocated resulting in poor performance of the application and the overall slowdown of the system. As mobile devices have significant constraints of available memory, memory leakage testing is crucial for the proper functioning of an application

5. Interrupt Testing- An application while functioning may face several interruptions like incoming calls or network coverage outage and recovery. The different types of interruptions are:

• Incoming and Outgoing SMS and MMS
• Incoming and Outgoing calls
• Incoming Notifications
• Battery Removal
• Cable Insertion and Removal for data transfer
• Network outage and recovery
• Media Player on/off
• Device Power cycle

An application should be able to handle these interruptions by going into a suspended state and resuming afterwards.

6. Usability testing- Usability testing is carried out to verify if the application is achieving its goals and getting a favorable response from users. This is important as the usability of an application is its key to commercial success(it is nothing but user friendliness).

7. Installation testing- Certain mobile applications come pre-installed on the device whereas others have to be installed from the store. Installation testing verifies that the installation process goes smoothly without the user having to face any difficulty. This testing process covers installation, updating and uninstalling of an application.

8. Certification Testing- To get a certificate of compliance, each mobile device needs to be tested against the guidelines set by different mobile platforms.

The Certified Mobile Application Tester popularly known as CMAT certification exam is offered by the Global Association for Quality Management (GAQM) via Pearson Vue Testing Center worldwide to benefit the Mobile Application Testing Community.

Some Mobile Application Testing Tools

Some tools that are being used to test code quality in general for mobile applications are as follows:

Cross-Platform (Android and iOS)

1. Appium - Mobile device automation for functional testing (http://appium.io)

2. Calabash - Mobile device automation for functional testing (http://calaba.sh)

3. Testdroid - Mobile App and Game test automation on real Android and iOS devices (http://www.testdroid.com/)

4. Perfecto Mobile - Mobile device automation for functional testing (http://www.perfectomobile.com)

5. SOASTA TouchTest - Mobile test automation for functional testing of native & hybrid apps (http://www.soasta.com/products/touchtest/) ^[5]

6. Testin - This tool let you test your apps across 300+ devices. This cloud based solution comes with automated testing features such as automated compatibility, functionality, UI & performance testing.(http://www.itestin.com/)

7. Ubertesters - This is a freemium tool which helps you conduct more structured and well organized Mobile QA process. Some of the features of Ubertesters are In-app bug editing, marking, reporting and user feedback, Multi-platform support, Over-the-air (OTA) app distribution, Build management etc. Ubertesters also offers in-the-wild app testing services with its global community of professional testers. (http://ubertesters.com/)

8. Crashlytics - This is a free tool available for both- iOS and Android devices.(http://try.crashlytics.com/)

9. Ranorex - This is a cross device app testing tool through which you can record one test and run it on multiple devices and languages. You can test your iOS, Android and Windows 8 Apps with this tool. (http://www.ranorex.com/mobile-automation-testing.html)

10. Experitest - Mobile device automation for functional testing (http://www.experitest.com)

11. Remote TestKit - A device cloud for mobile application testing (https://appkitbox.com/en/testkit/)

12. Test Fairy - Mobile application testing with video recording (https://www.testfairy.com/)

For Android

1. Android Lint - This is integrated with Eclipse IDE for Android. This will point out potential bugs, performance problems (http://developer.android.com/tools/help/lint.html)

2. Find Bugs - This is an open source library for static analysis in Java code (https://code.google.com/p/findbugs-for-android/)

3. Maveryx - Maveryx for Android is an automated testing tool for functional, regression, GUI, and data-driven testing of Android mobile application (http://www.maveryx.com)

For iPhone

1. Clang Static Analyzer - An open source tool for running static analysis for iPhone code (http://clang-analyzer.llvm.org/)

2. Analyze code from XCode - done during compile time (http://matthewmorey.com/automatic-static-analysis/)

References