When you get into Android testing for the first time, wrapping your head around the different solutions out there can quickly become a pain.
Should you test on your own device? Maybe in an emulator? Maybe in a bunch of emulators running at the same time? Maybe in a bunch of emulators running at the same time in the cloud?
Luckily enough, we’re here to guide you through! Here’s a complete showdown of four of the most popular types of testing solutions, with their advantages and drawbacks. We hope this will help you choose the right one for your specific use case. Have a good read!
Quick jump
Hardware devices
Virtual devices
Cloud physical devices
Cloud virtual devices
Bonus: cloud virtual devices on demand
Wrap up
Hardware Devices (a.k.a physical or real devices)
Real device testing comes as an easy and obvious choice. After all, there’s a high chance you already have an Android device at hand – and most probably your own.
And you’re definitely right! What’s better than testing your app in a real environment with real conditions (e.g. outside, at night, under a strong rain with cold hands)? Not to mention that what you’ll see will be the closest to what your users will experience. Especially in terms of performance and look & feel. Last but not least, real world testing allows you to take the most of each device specifics in terms of hardware (screen quality, fingerprint sensor, etc.).
However, you’ll quickly realize hardware testing doesn’t come as a sustainable choice, especially if you’re an indie developer. Why? Because there are dozens – heck – hundreds of different Android devices and versions around… See where we’re going? You’ll have to buy each of these devices, or at least the most popular ones, to efficiently perform your testing sequences.
That’s why Android real device testing is only suitable for big companies willing to put in place mobile labs and maintain them (see picture above). But one question remains: how can remote employees access them?
Advantages of hardware testing:
- • Real environment and real conditions. So more realistic scenarios
- • Perfect to get the most of hardware specifics
- • Better and faster performance
Drawbacks:
- • Too. Many. Devices!
- • Plus some of them are hard to get your hands on.
- • Requires space and dedicated maintenance
- • What about remote teams? They cannot access these devices
- • Overall expensive and time consuming
Virtual Devices (a.k.a. mobile emulators)
If you look around online, you’ll find out two popular solutions for Android developers :
The main advantage of Android emulators? They give you instant access to any device or version available currently. Android OS is completely there, just as on a real device and, even better, you can run thousands of different configurations without needing extra-space! Just as long as your computer RAM is holding up.
“Alright what’s the catch?”
You’re not holding a real device – obviously – so there’s no way to know how your app really looks, on, say, an AMOLED screen. Plus, there’s no way to access fingerprint sensors or other device-specific hardware features.
Another thing to note: Android virtual devices such as Genymotion emulate Android OS with an x86 architecture. As a consequence, apps compiled for ARM architectures will not work. So you might want to be careful with that.
But overall you get the whole Android experience, allowing you to test at least 95% of your app use cases before releasing it to the big world.
Advantages of Android emulators
- • Access to all Android versions and most devices
- • Android OS is there as a whole, just as on any physical Android device
- • Easy to set-up
- • Breaking things is not an issue. At worse, you’ll need to reconfigure VDs from scratch
- • Thousands of devices can be run at once without needing extra physical space
- • Free or relatively inexpensive
Drawbacks:
- • By definition, some hardware functions cannot be tested
- • Incompatibility issues with some apps (due to their architecture)
Cloud physical devices (a.k.a. device farms)
Still there? OK so pay attention because this is where things get a little bit more complicated.
Cloud physical devices give you the advantage of real Android devices without owning them. This is what we also call “device farms.” Let us explain to you: a company buys hundreds of devices (as described in our first section), but instead of keeping them for itself, it shares them with you via cloud computing technologies. Amazon does this for instance, and even provides detailed reports after each test sequences.
Of course this comes at a little cost, but in the long run you save a lot as you don’t need to buy any device, nor putting in place maintenance policies.
Unfortunately, performance tends to be inconsistent – if not bad. And some devices are sometimes even unusable*! Not to mention that not everything can be tested as devices are usually stored in data centers. Add up to this no ADB capabilities and a bunch of other limitations…
*There’s no way to run tests on devices being already used. Sometimes there can be a huge queue before being able to access desired phones!
Advantages of cloud physical devices:
- • Access to hundreds of physical devices without having to own them
- • No extra space or maintenance needed
- • Detailed reports after each tests
Drawbacks:
- • By definition, physical access to devices is impossible
- • Real world testing is limited
- • Performance and stability are erratic
- • Requested devices can take a long time before before being available
- • No ADB capabilities
- • File size uploads cannot exceed 4GB
- • Only 5 simultaneous tests can be run at once
- • Remote access sessions cannot last more than 60 minutes
Cloud virtual devices (a.k.a cloud emulators)
On to the next level of testing! Cloud emulation is a concept we’ve been the first to introduce with Genymotion Cloud.
It comes as a reliable solution for one main reason: team collaboration.
Imagine a bunch of Android virtual devices available from absolutely anywhere. Well that’s actually what it is. Cloud emulation allows your company as a whole to run and access Android devices and their configurations from anywhere, with everyone at the same time, even from a mobile phone!
Another interesting point: it works with desktop emulators, existing Android development solutions (e.g. Android Studio) and even instrumentation frameworks.
The catch up? Well… It’s expensive. Only big companies and enterprises can afford this kind of solution, unless you’re willing to pay 20,000 per year. But would it really be interesting for you?
The main advantage of cloud virtual devices lies in team collaboration and full compatibility with development and testing tools. They bring the strengths of physical device farms and virtual devices in one go-to solution. And in the long run, they dramatically improve productivity as they provide good performance and wide test coverage.
Advantages of cloud virtual devices:
- • Virtual devices… from anywhere!
- • No infrastructure or admin constraints
- • Better project visibility thanks to synchronous and asynchronous collaboration features
- • Better test coverage, so better productivity
- • Backwards compatibility with Genymotion Desktop
- • Suitable with most Android development solutions (Android Studio, ADB, Gradle…)
- • Also works with most instrumentation frameworks (Espresso, Appium, Calabash…)
Drawbacks:
- • Relatively expensive
- • Mostly suitable for big enterprises
HAVE A LOOK AT GENYMOTION CLOUD
Bonus: cloud virtual devices on demand
Cloud virtual devices on demand provide the same benefits than the solution above but with a pay as you go access! Which means short amounts and, most importantly, no long term contractual commitment. It comes as a perfect answer to Genymotion Cloud for small companies and independent developers.
And that’s not all! These kind of cloud virtual devices bring you all the advantages of the cloud platforms they’re hosted on. So you get access to all the APIs and features out there. With Amazon Web Services, you can for example embedded your virtual devices within any website. Doesn’t that sound awesome? ?
The main disadvantage? Well there has to be one: these are only available on Amazon Web Services… For now! We’re actively working on a solution to port them to other cloud providers. Brace yourself, it’ll be ready soon ?
GIVE GENYMOTION ON DEMAND A TRY
So which one should you go with?
Well you must start asking yourself: what is my structure size and how important is testing to me?
If you’re in a big corporation, with a huge test volume, cloud physical devices and cloud emulators are your go-to solution. The former for real world testing and the latter for collaboration purposes (e.g. developers, testers and designers on a same project).
If you’re an indie developer or working in a small company, with small volumes, then regular emulators and on demand virtual devices will answer most of your needs. The former for solid testing capabilities and the latter for more flexibility (each cloud provider brings different API and features on the table).
As for pure hardware devices, they come as an interesting solution, but are only suitable at the end of testing processes. However, money and space must not be a problem. So that’s one more for big companies.
That’s a wrap! Thank you for reading. We hope this made things more clear and will help you choose the right solution for your needs. Whatever may be, best thing to do is to try for yourself. And for that we can help ? Give us a shout, see you around and have a great one ✌️
