Written by Ellinor Kwok – Product Marketing & Pre-Sales at Genymobile

Developing high quality applications requires lots of testing. Sometimes hundreds or thousands of automated tests need to be run continuously before a pull request gets validated. If the tests take too much time this can slow down a project velocity.

How can the whole test cycle time be reduced?

 
This is how test sharding comes into play. It is a good way to parallelize your tests. It allows you to evenly divide up your tests into groups, called shards. For example, you need to run 100 tests that would take 30 mins to complete on one single device. With test sharding, you can speed up your tests by splitting your tests into two shards and run one half on one device and another half on another device, resulting in reducing the time spent for the tests by two, reaching theoretically 15 mins instead of 30 mins.

In this blog post, we will show you two different ways to do Android test sharding on Genymotion Cloud virtual devices to efficiently speed up your testing time. This will enable you to test at scale and increase your project velocity by being able to start many cloud virtual devices in a flash, you are not limited to 2 or 3 virtual devices if you were to run them locally.

Integration of Genymotion Cloud into your existing testing environment is seamless: you are able to do direct ADB calls to cloud virtual devices which makes Genymotion Cloud compatible with any tools relying on ADB, testing frameworks and Continuous Integration servers (e.g. Espresso, Appium, Jenkins, Bamboo…).

Quick jump

 
1. Prerequisites
2. Test sharding manually with ADB
3. Test sharding using an existing framework: Spoon
 

Prerequisites

 

Get Genymotion Cloud (if not already)

 
To use Genymotion Cloud, if you are already a Cloud customer see below. Or if you are interested in evaluating Genymotion Cloud, please fill up this form and we will get back to you.
To control your cloud devices from Gradle or from a command line you need Genymotion Desktop Enterprise 2.8.1 or later.

Configure your Android project

 
To shard your project you will need to use the AndroidJUnitRunner from the support library (or equivalent) that can be setup like this on your project:


 

Test sharding manually with ADB

 

Setup your test run

 
1. First, you need to build the application and instrumentation APKs.

./gradlew assembleDebug
./gradlew assembleDebugAndroidTest

Get both APKs at ../app/build/outputs/

2. Start Genymotion Cloud virtual devices using gmtool

In this example we are going to start two virtual devices :

gmtool --cloud admin startdisposable "Google Nexus 5 - 5.1.0 - API 22 - 1080x1920" nexus5
gmtool --cloud admin startdisposable "Google Nexus 4 - 4.3 - API 18 - 768x1280" nexus4

Where “Google Nexus 5 […]” and “Google Nexus 4 […]” are the names of the device templates we want to start. You can get the list of all available templates with gmtool --cloud admin templates

3. Once the devices are started in Genymotion Cloud, ADB connection is automatically established

You can see those cloud devices are connected to the local ADB:

adb devices
List of devices attached
localhost:46428 device
localhost:39483 device

4. You can then install both APKs on them:

adb install <app.apk>
adb install <instrumentation-app.apk>

Run your test shards

 
You can then run the instrumentation APK using test sharding:

adb -s <deviceSerial> shell am instrument -w -e numShards <shardsNumber> -e shardIndex <shardIndex> <app_package>/android.support.test.runner.AndroidJUnitRunner"

-s <deviceSerial>: on which device those tests will be run
-e numShards <shardsNumber>: in how many shards the test suite should be divided. In our case, it would be 2 since we want to divide the tests in 2 groups.
• -e shardIndex <shardIndex>: which shard should be run on the device. Here it would be 0 for one device and 1 for the other device to have all the tests run on both devices.
• <app_package> should be com.mycompany.myapp.test if your application has a package like : com.mycompany.myapp

For example, using Genymotion binocle project (get the source code on Github). This project showcases the use of Genymotion JAVA APIs that enable the use of Genymotion sensors values in your tests such as the battery, gps, telephony…

• Run first half of tests on a virtual device:

adb -s localhost:46428 shell am instrument -w -e numShards 2 -e shardIndex 0 "com.genymotion.binocle.test/android.support.test.runner.AndroidJUnitRunner"

• Run second half of the tests on the other virtual device:

adb -s localhost:39483 shell am instrument -w -e numShards 2 -e shardIndex 1 "com.genymotion.binocle.test/android.support.test.runner.AndroidJUnitRunner"

When tests are done, stop Genymotion Cloud virtual devices

gmtool --cloud admin stopdisposable nexus5
gmtool --cloud admin stopdisposable nexus4

Analyzing the results

 
This approach does not require any external library or dependency but it is more cumbersome since everything needs to be done (building and installing app, manually launching the instrumentation command for each device, …). But it could be easily scripted.

Another problem here is how results are handled. By running the tests “by hand” you won’t get a report like the one that Android Gradle plugin provides by default. The results are displayed on the console output, like this for example :

adb -s localhost:39483 shell am instrument -w -e numShards 2 -e shardIndex 1 "com.genymotion.binocle.test/android.support.test.runner.AndroidJUnitRunner"
com.genymotion.binocle.test.TestBattery:.
com.genymotion.binocle.test.TestGps:.
com.genymotion.binocle.test.TestPhone:.
com.genymotion.binocle.test.TestRadio:.
 
Time: 53.095
OK (4 tests)

When you run the gradle connectedAndroidTest task of the Android Gradle plugin, output results are more usable, but you can’t shard them. Here is an example of the output at ../build/reports/androidTests/connected/com.genymotion.binocle.test.html

package com.genymotion.binocle.test

You will see in the next paragraph how you can shard and get clear and detailed results.

Test sharding using an existing framework: Spoon

 
Square Android team developed an open-source framework called Spoon, that distributes instrumentation tests execution and displays the results with an HTML page. In this article, we will only focus on test sharding.

With Spoon, you can do auto-sharding, this will automatically shard all your tests across devices. Also if you organized your tests with package name, you can specify which set of tests you want to run by using the package name.

There are two ways of running the tests: using Spoon jar and using Spoon & Genymotion Gradle plugins. We will present you with each one of them.

Using Spoon jar

 
1. You need to download the latest runner Spoon jar

2. Then, build the application and instrumentation APKs:

./gradlew assembleDebug
./gradlew assembleDebugAndroidTest

3. Start Genymotion Cloud virtual devices using gmtool:

gmtool --cloud admin startdisposable "Google Nexus 5 - 5.1.0 - API 22 - 1080x1920" nexus5
gmtool --cloud admin startdisposable "Google Nexus 4 - 4.3 - API 18 - 768x1280" nexus4

4. Run the command

java -jar spoon-runner-1.7.0-jar-with-dependencies.jar --apk <pathToApk> --test-apk <pathToInstrumentationApk> \
-serial <deviceSerial> \
--shard

You can specify as many devices as needed via the -serial argument. This is optional, if you don’t specify it, Spoon will automatically distribute on available devices, until there are no more tests. If you also want to run a specific set of tests from a package name you can add the –e package argument, which gives:

java -jar spoon-runner-1.7.0-jar-with-dependencies.jar --apk <pathToApk> --test-apk <pathToInstrumentationApk> \
-serial <deviceSerial> \
--e package=<packageName> \
--shard

If you want to use a different sharding strategy you can get more information on Spoon documentation.

5. After finishing the tests, make sure to stop the virtual devices:

gmtool --cloud admin stopdisposable nexus5
gmtool --cloud admin stopdisposable nexus4

Test Sharding using Spoon and Genymotion Gradle plugins

 
This part of the tutorial has also been recorded in video.

In your build.gradle file:

1. Add the dependency towards Spoon and Genymotion in the dependency block:

2. Apply Spoon and Genymotion plugins:

3. Add Spoon configuration of the application module:

4. Add genymotion configuration:

In taskLaunch, we specify spoonDebugAndroidTest gradle task that will be launched after all Genymotion Cloud virtual devices are started

5. Define Genymotion Cloud virtual devices that will be used to run the tests in genymotion block:

Finally, run ./gradlew spoonDebugAndroidTest task so that cloud virtual devices will first be launched before instrumentation tests are run. Cloud virtual devices are automatically being stopped after all the tests are finished.

6. Once tests are done, spoon creates results at ../build/spoon/debug/index.html

All cloud devices are listed like this: Genymotion + <device name>

binocle debugAndroidtest

You can click on one device to have more details on the tests that have been run on it. You can also take screenshots using Spoon Gradle plugin.

Genymotion nexus7

You can also get logs of the device per test so if you have a regression you can easily find the root cause.

Gps Warning

That’s it! To make things easier, you can directly visit Genymotion Binocle GitHub page and get the sample code with Genymotion Cloud & Spoon Gradle plugin implementations. Further details related to Genymotion Gradle plugin can be found in our documentation.

Oh and make sure to check out Spoon Gradle Plugin documentation on GitHub 😊

GET THE SAMPLE CODE