Automating Release Tests

Removing pain from a process

In the Consumer Web Apps team, before every production release goes out we run a set of “Release Tests” manually. We introduced this process because we found that when our responsive website was being released, while it looked great on desktop browsers, we had a lot of visual bugs on mobile devices. Release testing involves putting a delivery, collection, cash and card order through on an iPhone, iPad, Android Phone and Android Tablet. The process takes about 30 minutes, and is run as soon as our release candidate reaches our QA environment. We share the responsibility for this testing between every single team member, including product, developers, test automation engineers and UX.
I have recently tried to utilise our existing functional test suite to automate this process to create a set of screenshots that can be viewed by the person on release testing duty, to try and remove some of the pain for this process.
For context, the functional test suite is written using Ruby, Cucumber and Capybara. In order to extend this to be able to run on devices, I have been using BrowserStack.

Set up

The first step was to create a new feature file to run through all the pages in our app. I tagged the scenario with @ipad, which is used later in the environment setup code. I already have methods to do these actions, so I was able to reuse those with some modifications as some of them didn’t translate onto mobile devices too well.

Feature: Visit pages on iOS devices
  Scenario: Load pages on iPad
    * I am on the homepage
    * I am on the search_page
    * I am on the menu
    * I add items to the basket
    * I navigate to the basket page
    * I navigate to the login page
    * I navigate to the order details page
    * I navigate to the time and note page
    * I navigate to the payment options page
    * I navigate to the order confirm page

Connecting to BrowserStack

I added the following code to our env.rb file:

 Capybara.register_driver :ipad do |app|
  caps =
  caps['browserstack.debug'] = 'true'
  caps[:name] = 'JUST EAT Tests'
  caps[:browserName] = 'iPad'
  caps[:platform] = 'MAC'
  caps[:device] = 'iPad 4th'
  caps[:emulator] = 'true'
  caps[:acceptSslCerts] = 'true'
  start_browserstack(app, caps)
Before('@ipad') do
  @browser = 'ipad'
  Capybara.current_driver = :ipad

Making the test useful

I hooked the test into TeamCity, and put a trigger on it to run on every deploy to our QA environment. This means that with every release candidate that gets built, this test will run and take screenshots across the four devices. It also published a test results html report, with the screenshots embedded. So now, the job for the person on Release Testing duty is just to go to TeamCity (where they’d go to look at the other test results) and access the html report, which looks something like this:
Release test results