Quality Assurance and Testing11 February 2016
How to build an expert nearshore mobile app development team, part 3
Nowadays, quality assurance is essential for publishing high quality apps, and with a plethora of devices on the market, it isn’t an easy task. The possibilities and challenges of testing in mobile worlds are slightly different than those in desktop/server development. So, what are the best practices for this?
Ongoing mobile testing process
With the complexity of current software, the old model of testing the software at the end of the development cycle is long gone. Testing should be done throughout the development process – for every functionality added, there should be testing (both manual and automatic) to ensure that the functionality actually works. Automatic testing is also essential to ensure that further development of the project, and changes in the project environment, don’t damage any previous functionality.
Unit testing is a classical approach to automated testing. In this approach, each code functionality is tested individually in separate unit tests. This can ensure that each code unit works when it should, and behaves as planned in the case of errors. Unit testing works particularly well when testing business logic.
However, many mobile applications are written in the ‘thin client’ way, which means that most logic is on the server side, and the application’s only role is to download data from the server and present it to the user. This is often the preferred model, because one type of business logic can be implemented on the server and used on every platform (each mobile platform, web and desktop).
However it makes unit testing slightly less useful – unit testing is not the best tool for testing UI, and there’s not much business logic to be tested. Unit testing is still useful in testing the technical aspects of UI – proper initialization, handling of screen rotation, saving user data, maintaining the correctness of data downloaded from web services, etc. – but unit testing is not an effective tool for testing the entire application from the user’s perspective.
Of course, this changes when application is built in the ‘fat client’ way – in this model, the application has its own internal data model and business logic that is partially or fully independent of that which is on a server. In this case, unit testing is essential for making sure that each use case and each code flow behaves correctly in numerous possible scenarios.
UI automation testing
UI automation testing constitutes an answer to the problems inherent in unit testing. Instead of focusing on individually testing each part of the code, it tests the application as a whole (black-box testing). Contrary to its name, it doesn’t only test UI – it tests the entire application, but from a user perspective. Instead of testing which code is executed, it tests the results of user interaction. For example it tests that results of pressing the button are as expected. This allows the entire application to be tested – both UI events, internal code for web service communication, and server-side code that responds to data. At all times, it also tests to make sure that the application doesn’t crash or execute properly.
An additional advantage of UI automation is that it is much better suited to testing device-dependent problems, such as problems with layouts, specific custom behavior on some devices that needs to be handled differently, etc. Running the same tests on different devices ensures that the application will work correctly on a wide array of devices, and may find unexpected errors that can’t be detected with unit tests.
Testing mobile application on a Device Farm
Every respectable mobile application vendor has multiple devices by which to test applications – both manually and using automated tests. This is a good foundation during the development phase for finding the most common problems and correcting them as soon as they are found. However there are so many devices on the market that this is impossible to test on-site for even a noticeable fraction of them. Android has more than 11 000 device models at the time of writing, and that number is constantly growing. Samsung alone has more than 1000 device models that are sold worldwide (including local variations for selected markets). iOS and Windows Phone have a slightly smaller number of devices, but they still come in different variations and system versions. That variety makes it unrealistic to have real market coverage of testing, when testing is only undertaken on-site. Luckily there are providers that offer device farms – remote services that allow companies to run remote tests on a large number of device models in a short time. Device farms are inexpensive, because you only pay for the time used on devices, and tests typically proceed rapidly. It saves the vendor the cost of buying hundreds of devices for testing, and also makes it possible to test the same device with different OS versions.
Device farms allow for the undertaking of most tests – unit, UI automation, and others, on multiple devices. Those tests make it possible to see detailed results, which in turn enable developers to find problems even if they only arise on a small subset of devices.
Additionally, aside from automated testing, device farms also make it possible to take screenshots of every screen in the app, which then allows for manual (human) checking to spot potential UI problems.
Beta testing and automatic crash reporting
Even a perfectly tested app may still have potential problems – new devices and versions of mobile systems arise every day and some of them change important behaviors or hardware features. So before final release, it is wise to first release the application in beta test mode. The app may be available to everyone, but the reviews and scores from the beta test are removed when the full version is released. Using the framework for automatic crash reporting, every time an application crashes on the user’s device, developers get a detailed report including which devices it was on, what version of OS, and when the application crashed. Due to this, it is possible to fix all the potential problems that weren’t spotted before going public, and allows for the release of a “final” version that is bug-free.
Automated crash reporting is essential even after full release – this mechanism makes it possible to fully monitor the market and spot potential problems even before users report them. For example, if the new OS version makes the app crash, developers receive instant reports of such events and are able to quickly pinpoint the reason and fix it.
Quality Assurance and Testing summary
Testing is essential for the ongoing quality assurance process, and should be done throughout the development process, not just before releasing the application. Combining unit testing and UI automation testing allows for a sufficiently acceptable initial release, while beta testing each version and monitoring crash reports throughout allows developers to make sure the application works well throughout its lifecycle.