Navigating mobile software testing

Many companies have the misperception that an app is part of any attempt to enter the mobile space.

Read time 5min 30sec

The mobile software testing market is experiencing somewhat of a Jekyll and Hyde complex at the moment.

The app boom seems to be well and truly over.

On the one hand, mobile device growth is continuing apace, with predictions that mobile users will overtake PC and laptop users for most business and Web-related tasks having long since been realised. On the other hand, the app boom seems to be well and truly over, with recent figures in the US showing users are downloading an average of zero apps per month.

This isn't difficult to explain; it simply means the mobile market, which began its upward trajectory more than eight years ago, has reached saturation, and most users are now familiar with the apps they need and use every day. There are some exceptions - like Snapchat and Uber - that defy the trend and are still growing at a phenomenal rate, but unless an app developer is very good or very lucky, it's going to be difficult to get the new app noticed and downloaded among the crowd.

This affects mobile testing in two fundamental ways. First, the drop-off in new app development means companies have a decision to make when it comes to reaching out to their customers through mobile platforms. Apps are no longer the first step in creating a mobile presence; for many companies, a responsive mobi (mobile-oriented) site makes more sense.

Secondly, device selection is vital, and increasingly so. The rapid growth and maturity of mobile devices in general and smartphones in particular have seen the market ultimately settle on two major platforms - iOS and Android. Some smaller platforms such as Windows Mobile and Blackberry are shrinking, and even (in the case of Blackberry) migrating their users to various flavours of Android.

To test or not to test?

Because of this polarity, and the loyalty of most users to one platform or another, developing and testing native apps is not always the smart choice, especially for newcomers to the mobile space. But, if testing on a limited number of devices is counter-intuitive, testing on a large number of devices is often prohibitively expensive.

So, the starting point for any conversation on mobility and mobile testing should always be a company's digital strategy. Unfortunately, most companies don't have a fully formed digital strategy, and those that do have half-cooked strategies, which are based on the perception that an app is part and parcel of any attempt to enter the mobile space. The truth is that a well-built, responsive and intuitive mobile Web site is almost as important (if not more so) as an app, and can also perform most if not all the functions of an app, depending on the type of business it's used for.

Deciding between apps, Web sites, or a combination of the two is just one of the challenges. A comprehensive digital strategy also needs to cover factors like device management, device types, usability testing and automation.

Going AWOL

From a testing perspective, device management is critical because mobile devices are susceptible to damage, loss and theft more frequently than almost any other device type. It may seem inconsequential, but given the high cost of devices, the risk of valuable IP taking a walk at a critical development stage is very real.

The issue of device types is important regardless of the software being tested, be it an app, a mobile site or a desktop site on mobile devices. Even a closed platform like iOS comes with the challenge of users with previous generations of iPhones and iPads, and multiple iterations of previous generations as well.

A modern, responsive mobi site or app might light up the screen of the latest iPhone, but could bring previous legacy iPhones with older versions of iOS to a standstill. And iOS is fairly straightforward compared to the permutations of the hundreds or thousands of Android devices from dozens of different manufacturers on the market today.

Once device management and types have been narrowed down to a manageable grouping, usability testing on these devices is a third major consideration. This is where mobility testing also differs the most from other forms of software testing, because it's necessarily hands-on. Yes, developers can test their mobile apps or Web sites on simulators and the odd device, but neither of these options are anywhere close to sufficient for a proper functional test of a new (or new version) of the software.

It's almost impossible to remotely test mobile software, not because the technology is lacking, but because usability is such a big factor in the success or otherwise of a mobile app or Web site. And when it comes to usability, this means testing by experienced human operators, not machines.

Which brings me to the last point, automation. Even if it were practical to automate some parts of the mobile testing process, the rapid rate of change in both devices and apps (and Web sites) means by the time a company has invested in solid testing scripts for its software, a new version rolls out, users upgrade to new devices, and a new OS has been released. That's not to say automation won't play an important role in a company's mobility strategy, but manual testing will probably play a much bigger one.

It's becoming clear that jumping into the mobile space - or growing a company's current mobile presence - is a much bigger ask than previously thought. Navigating the mobile testing minefield can be a nightmare if companies don't have a solid, thought-out digital strategy that informs every decision they make based on the value of the investment to the business.

A good place to start would be finding a likeminded partner with the experience to guide the company through the creation or refinement of its digital strategy, before providing access to the resources needed to make it the success it needs to be.

Mario Matthee

COO at DVT Global Testing Centre

After holding numerous positions within DVT, Mario Matthee was given the opportunity to realise his dream of building the biggest test automation centre in South Africa. Today, he is the COO at DVT Global Testing Centre and is motivated by his belief that the youth are the future. It is a philosophy that Matthee incorporates into his client-centric approach to test automation solutions, by blending experience with young, bright talent. Before joining DVT in 2007, Matthee applied his passion for software testing and test automation at Allan Gray, JP Morgan and Vodacom, among others.

See also