Written by Ellinor Kwok – Product Marketing & Pre-Sales at Genymobile
Developing high quality applications requires lots of testing. Sometimes hundreds or thousands of automated tests need to be run continuously before a pull request gets validated. If the tests take too much time this can slow down a project velocity.
How can the whole test cycle time be reduced?
This is how test sharding comes into play. It is a good way to parallelize your tests. It allows you to evenly divide up your tests into groups, called shards. For example, you need to run 100 tests that would take 30 mins to complete on one single device. With test sharding, you can speed up your tests by splitting your tests into two shards and run one half on one device and another half on another device, resulting in reducing the time spent for the tests by two, reaching theoretically 15 mins instead of 30 mins.
In this blog post, we will show you two different ways to do Android test sharding on Genymotion Cloud virtual devices to efficiently speed up your testing time. This will enable you to test at scale and increase your project velocity by being able to start many cloud virtual devices in a flash, you are not limited to 2 or 3 virtual devices if you were to run them locally.
Integration of Genymotion Cloud into your existing testing environment is seamless: you are able to do direct ADB calls to cloud virtual devices which makes Genymotion Cloud compatible with any tools relying on ADB, testing frameworks and Continuous Integration servers (e.g. Espresso, Appium, Jenkins, Bamboo…).
Quick jump
1. Prerequisites
2. Test sharding manually with ADB
3. Test sharding using an existing framework: Spoon
Prerequisites
Get Genymotion Cloud
Install gmsaas
It is our tool to manage Genymotion Cloud SaaS instances. It is a Python package, so you can download it using pip3 install gmsaas.
Configure your Android project
To shard your project you will need to use the AndroidJUnitRunner from the support library (or equivalent) that can be setup like this on your project:
Test sharding manually with ADB
Setup your test run
1. First, you need to build the application and instrumentation APKs.
./gradlew assembleDebug
./gradlew assembleDebugAndroidTest
Get both APKs at ../app/build/outputs/
2. Start Genymotion Cloud virtual devices using gmsaas
In this example we are going to start two virtual devices :
#start a Google Nexus 5X - 8.0 device
instance1=$(gmsaas instances start bf1a9765-c743-4cdc-95ec-c40a74493055 device_8.0)
#start a Google Nexus 6P - 7.1 device
instance2=$(gmsaas instances start 00cd578a-14de-4f47-b4f1-d454d613d9da device_7.1)
Where bf1a9765-c743-4cdc-95ec-c40a74493055 and 00cd578a-14de-4f47-b4f1-d454d613d9da are the UUID of the device templates we want to start. You can get the list of all available templates with gmsaas recipes list
3. Once the devices are started in Genymotion Cloud, the instance UUID is printed on standard output. Then you need to connect those devices with adb.
port1=10000 && port2=20000
gmsaas instances adbconnect $instance1 --adb-serial-port=$port1
gmsaas instances adbconnect $instance2 --adb-serial-port=$port2
An adb tunnel will be set up and the devices in the cloud will be seen as local devices. You can see those cloud devices are connected to the local ADB:
adb devices List of devices attached localhost:10000 device localhost:20000 device
4. You can then install both APKs on them:
adb install <app.apk>
adb install <instrumentation-app.apk>
Run your test shards
You can then run the instrumentation APK using test sharding:
adb -s <deviceSerial> shell am instrument -w -e numShards <shardsNumber> -e shardIndex <shardIndex> <app_package>/android.support.test.runner.AndroidJUnitRunner"
• -s <deviceSerial>: on which device those tests will be run
• -e numShards <shardsNumber>: in how many shards the test suite should be divided. In our case, it would be 2 since we want to divide the tests in 2 groups.
• -e shardIndex <shardIndex>: which shard should be run on the device. Here it would be 0 for one device and 1 for the other device to have all the tests run on both devices.
• <app_package> should be com.mycompany.myapp.test if your application has a package like : com.mycompany.myapp
For example, using Genymotion binocle project (get the source code on Github). This project showcases the use of Genymotion JAVA APIs that enable the use of Genymotion sensors values in your tests such as the battery, gps, telephony…
• Run first half of tests on a virtual device:
adb -s localhost:10000 shell am instrument -w -e numShards 2 -e shardIndex 0 "com.genymotion.binocle.test/android.support.test.runner.AndroidJUnitRunner"
• Run second half of the tests on the other virtual device:
adb -s localhost:20000 shell am instrument -w -e numShards 2 -e shardIndex 1 "com.genymotion.binocle.test/android.support.test.runner.AndroidJUnitRunner"
When tests are done, stop Genymotion Cloud virtual devices
gmsaas instances stop $instance1
gmsaas instances stop $instance2
Analyzing the results
This approach does not require any external library or dependency but it is more cumbersome since everything needs to be done (building and installing app, manually launching the instrumentation command for each device, …). But it could be easily scripted.
Another problem here is how results are handled. By running the tests “by hand” you won’t get a report like the one that Android Gradle plugin provides by default. The results are displayed on the console output, like this for example :
adb -s localhost:39483 shell am instrument -w -e numShards 2 -e shardIndex 1 "com.genymotion.binocle.test/android.support.test.runner.AndroidJUnitRunner" com.genymotion.binocle.test.TestBattery:. com.genymotion.binocle.test.TestGps:. com.genymotion.binocle.test.TestPhone:. com.genymotion.binocle.test.TestRadio:. Time: 53.095 OK (4 tests)
When you run the gradle connectedAndroidTest task of the Android Gradle plugin, output results are more usable, but you can’t shard them. Here is an example of the output at ../build/reports/androidTests/connected/com.genymotion.binocle.test.html
You will see in the next paragraph how you can shard and get clear and detailed results.
Test sharding using an existing framework: Spoon
Square Android team developed an open-source framework called Spoon, that distributes instrumentation tests execution and displays the results with an HTML page. In this article, we will only focus on test sharding.
With Spoon, you can do auto-sharding, this will automatically shard all your tests across devices. Also if you organized your tests with package name, you can specify which set of tests you want to run by using the package name.
There are two ways of running the tests: using Spoon jar and using Spoon & Genymotion Gradle plugins. We will present you with each one of them.
Using Spoon jar
1. You need to download the latest runner Spoon jar
2. Then, build the application and instrumentation APKs:
./gradlew assembleDebug
./gradlew assembleDebugAndroidTest
3. Start Genymotion Cloud virtual devices using gmsaas:
#start a Google Nexus 5X - 8.0 device
instance1=$(gmsaas instances start bf1a9765-c743-4cdc-95ec-c40a74493055 device_8.0)
#start a Google Nexus 6P - 7.1 device
instance2=$(gmsaas instances start 00cd578a-14de-4f47-b4f1-d454d613d9da device_7.1)
4. Connect the devices to adb
port1=10000 && port2=20000
gmsaas instances adbconnect $instance1 --adb-serial-port=$port1
gmsaas instances adbconnect $instance2 --adb-serial-port=$port2
5. Run the command
java -jar spoon-runner-1.7.0-jar-with-dependencies.jar --apk <pathToApk> --test-apk <pathToInstrumentationApk> \
-serial <deviceSerial> \
--shard
You can specify as many devices as needed via the -serial argument. This is optional, if you don’t specify it, Spoon will automatically distribute on available devices, until there are no more tests. If you also want to run a specific set of tests from a package name you can add the –e package argument, which gives:
java -jar spoon-runner-1.7.0-jar-with-dependencies.jar --apk <pathToApk> --test-apk <pathToInstrumentationApk> \
-serial <deviceSerial> \
--e package=<packageName> \
--shard
If you want to use a different sharding strategy you can get more information on Spoon documentation.
6. After finishing the tests, make sure to stop the virtual devices:
gmsaas instances stop $instance1
gmsaas instances stop $instance2
7. Once tests are done, spoon creates results at ../build/spoon/debug/index.html
All cloud devices are listed like this: Genymotion + <device name>
You can click on one device to have more details on the tests that have been run on it. You can also take screenshots using Spoon Gradle plugin.
You can also get logs of the device per test so if you have a regression you can easily find the root cause.
That’s it!
