Parallelising tests on real devices with calabash-android
When testing one of our products, we have to run our automation suite on multiple types of device, sometimes with added permutations when trialling components. We looked into using a third party cloud service with real devices, like Xamarin Test Cloud, but some of the devices weren’t available. Simulated devices weren’t an option as we are often testing push notifications. With the added advantages of being able to watch tests running and interact with them manually if necessary, we decided to look into strategies on site.
Running tests on one device at a time would be very time consuming, slowing feedback loops and release cycles so we looked into a way to run our tests in parallel. This would speed things up, be more scalable and as a bonus, help to highlight any concurrency issues.
The approach we chose was to set to run all of our tests on each device simultaneously using a rake multitask.
multitask :run_parallel => @task_list
This multitask would invoke other tasks that would run against each device. The information about the devices to run on would be passed in via an environment variable, which would be used to generate the subtasks and put into a task list that was passed into the multitask.
ENV['USERS'] ||= 'User_1, User_2, User_3'
users = ENV[‘USERS’].split(',')
@task_list = []
users.each do |user_name|
user_name.strip!
task_name = user_name.gsub(/\s+/, "").to_sym
report_path = "Results/" + get_device_info(user_name)['device_id'] + '/features_report.html'
Cucumber::Rake::Task.new(task_name, 'All completed scenarios') do |t|
t.cucumber_opts = "--tags @finished --format html --out=#{report_path}"
end
@task_list << task_name.to_s
end
The next step was to make all of the variables in our code thread-safe to prevent conflicts. This meant replacing environment variables with thread variables.
Thread.current['NAME_OF_THE_VARIABLE']
One place this was particularly inconvenient was when passing the id of the target device into the cucumber task, as using the standard method of passing in environment variables was no longer an option. Using a thread variable wouldn’t work either, as the cucumber task runs on a separate thread to the one where it was created. Our solution was to use the id from the report title for that thread and pull it out as part of the setup for each test run.
The final step was to make our tests run against multiple user accounts, so that we could avoid clashes between all of the threads. We started by making our step definitions agnostic to the account it was using, and then created an account per device that was selected from a mapping yml file when initialising.
If these kind of challenges interest you, we are hiring.
Thanks for reading!
Alan & Adrian