Amadeus Consulting Discusses Android App Development QA Testing and Fragmentation

One of the big challenges to developing Android apps is the number of Android devices on the market. In fact, with almost 100 unique Android devices available there is a severe challenge with quality assurance and ensuring compatibility on all devices. The different sizes and capabilities of each device create a fragmented platform that makes quality app development more challenging. Although fragmentation has been a longtime issue for Android, Google is taking steps to help alleviate the problem with future devices.

The Problem: Android Fragmentation

In the PC market, every piece of software comes with “system requirements” printed clearly on the box. These state the basic requirements for properly running software, including operating system, disk space, and memory requirements. Compatibility isnt usually a big issue with most commercial PC software, but the system requirements help the user judge whether or not their computer can run it.

For smartphone app development, there is no “system requirements” listing, and consumers mostly assume that all Android phones and devices should be able to run all Android apps. The truth is that there is not universal compatibility among all Android devices.

Compatibility problems can be caused by:

Hardware differences, including different processors, various screen sizes and available memory (This is reportedly why the Netflix Android app is initially only launching on five devices).

Different OS versions, which may be due to different hardware abilities. Newer versions of the OS provide different capabilities and capacities which an app may need to run.

Different Carrier Software. Many carriers create or sell devices that use software built on top of the standard Android operating system. This may be different interfaces, or other software that changes how the OS acts. This can make it challenging to update the underlying OS which creates compatibility problems, or it may actually directly interfere with the app.

In the past Google has only loosely controlled hardware requirements for a handset to use the Android operating system, which has led to a very broad range of devices and device capabilities. Compatibility for apps was controlled through the licensing of the Android Market, which let manufacturers provide access to the 200,000+ Android apps.

Unfortunately with so many devices and so many unique differences (not to mention the sheer number of devices), it has become challenging as an Android app developer to properly test an app before launch.

The Current Solution

The common strategy for launching an app on Android is pretty straightforward: you test on a few devices and then work with the community to identify problems on other devices and resolve them as best you can.

Many major brands do this and state that their app is only compatible with specific devices. Of course it may work  or may partially work  with other devices as well, but that helps defer some of the bad press if the app has problems. For example, when the popular app Angry Birds was released in Android it came with a disclaimer which listed 21 specific devices that were “not officially supported by Angry Birds” at the time, as well as any devices running certain older versions of Android.

Choosing those initial devices to test with also takes a bit of background knowledge. You need to balance having a good representation of different types of devices with a representation of the most popular devices. In other words, your list of testing devices will change every few months and then you may need to update your testing process and purchase new units.

For example, when TweetDeck was creating an Android version of its immensely popular Twitter client, it ran into extreme fragmentation issues which the company summed up nicely in a chart and explanation on their blog. In this case TweetDeck used it as a point of pride to show how well the app worked across hundreds of different devices, customizations, roms, and what can best be described as hacks.

At Amadeus Consulting, we generally have a select list of representative devices that we use when developing Android apps for clients, but it is very hard to guarantee compatibility beyond that. For our clients, we find that it is most cost efficient to test on the “representative” devices, and then work out a support agreement to address issues that arise on less popular devices.

The other challenge is what to do when you have launched an app and it doesnt work on a particular handset. With the exception of a tiny handful of app developers (most of whom receive sponsored handsets anyways since manufacturers want to ensure that the apps work on their devices), most developers do not have $3+ million to spend buying every available Android device and then spend the time fully testing it on every one.

In some cases developers can find (or borrow) that particular device for testing, or in other cases it may be most practical to create and foster a community that can help each other address these kinds of problems.

Googles New Solution

At the Google I/O developer conference last week, Google announced important changes to how it will be managing new Android devices. First, Google will begin requiring that all new handsets be able to handle OS updates for at least the next 18 months following their launch.

This will increase the minimum capability of hardware devices as well as make it easier to provide upgrades to all devices at once. Currently some new devices are incompatible with newer versions, or must wait several months to receive the OS.

Along with this will be the launch of a new version of Android (version 3.1 presumably) called Ice Cream Sandwich. This will upgrade the current versions 2.2 (Froyo) and 2.3 (Gingerbread) as well as version 3.0 (Honeycomb). And in case you havent caught on, Googles naming scheme for Android versions is alphabetical with a dessert themed acronym.

The biggest benefit and feature of Ice Cream Sandwich is that it is specifically designed to be a cross platform OS and will combine tablets, smartphones, and other devices into a single cross-compatible OS.
This helps resolve incompatibility problems in two ways. First, it ensures that all new devices will be able to run the same OS, which helps reduce hardware compatibility problems. And since all devices are all running the same base OS, it will help reduce compatibility problems there too.

Starting in June, Google will also give developers more control on the Android Market and let them filter out incompatible devices. This is a positive move that gives developers the ability to offer quality, tested apps to the market, but it also has the potential to exclude devices that would otherwise work fine.

These changes will help solve some of the larger issues with Android fragmentation, but it will not reduce problems with custom ROMs or other carrier-specific customizations. Even so, these changes will make life much easier for developers during QA testing and will be extremely important as Android expands outside of just smartphones and powers tablets, TVs, and possibly everything in your house via Android@Home.