Shock – Better Automation Testing for iOS

March 5, 2019Written by Jack and Sneha

UI testing is a relatively young addition to Xcode, and while Apple provides a good bedrock of APIs from which to build in XCUITest, it does not provide a complete picture in terms of writing end-to-end or comprehensive feature tests.
In particular, using only the tools provided in XCUITest, Xcode provides no tools for dealing with staging the environment in which your app is running – this includes, but is not limited to: external APIs, 3rd party SDKs and mock configuration.

In particular, external API dependencies can be difficult to work with. Usually developers are faced with one of two choices: point to an unstable and/or rapidly changing QA or Staging environment, or point at production APIs.
Neither of these options are suitable if you aren’t in control of the data in the environments in question, and there are several other issues with going down this road:

  • It is bad for the health of a production environment to pollute it with test data
  • Even production APIs rarely have 100% uptime across all endpoints
  • You have to coordinate testing with other teams consuming these environments.

These are just a few of the many issues with using live APIs in UI tests. We built Shock to avoid these issues.

Alternative Solutions

Before diving further into what Shock provides to solve the aforementioned problems, it’s useful to examine the benefits and drawbacks of existing solutions in this space.

NSURLSession Stubbing (OHHTTPStubs)

A common solution is to use OHHTTPStubs to stub the networking layer.
OHHTTPStubs uses the Objective-C runtime to swizzle methods on NSURLSession and related classes to allow a locally defined response to be returned.
In practice, this is accomplished by setting up rules for testing an NSURLRequest (e.g. host, resource, headers, query string) and providing the name of a JSON file in the local bundle.
There are a few advantages to this approach:

  • It’s very easy to integrate and use.
  • There are lots of existing blog posts and tutorials.

However when it comes to using this for UI Tests, there are some drawbacks:

  • It’s reliant on the Objective-C runtime which comes with it’s own set of drawbacks and considerations – this is particularly undesirable if you are working purely in Swift.
  • JSON responses and code must be bundled with and executed in your app.
  • No actual network connection is made / attempted.
  • Lack of an actual system-level network request means some classes of network errors and issues cannot be tested (e.g. latency issues).
  • Relatedly, the Github repo has been largely inactive for over a year.

Using a QA Environment

Another solution is to use a real – but staged – environment that’s available. This is a great method for testing as it’s as close to calling APIs in a real environment as is possible without the lack of predictability that can result from production data.
Other benefits include:

  • Latest changes are (usually) available meaning new tests.
  • Tests can be written in parallel to API changes being made.
  • Minimal mocking or stubbing is required other than changing the target host name.

Drawbacks:

  • QA environments are subject to instability. This means a lot of time can be wasted trying to track down test failures caused by data inconsistencies, API outages, and so on.
  • QA environments are sometimes only available during business hours / days – this is inconvenient if you have nightly test runs.
  • QA environments are usually in use by all clients and, as such, automated testing usually needs to be scheduled during a period where the data in the environment isn’t being used or altered.

Why Shock?

Shock aims to provide a middle-ground between the above solutions, while also eliminating some of the drawbacks.

    • Real network connections
      Shock runs a real HTTP server so that real requests can be made. This opens up many options in terms of how you want to form or constrain requests, and how you want to generate and return mock data.
    • Templatable Mock Data
      Shock includes tools with which to generate or modify your mock data based on the incoming request. The Mustache templating engine allows you to structure your dynamic responses clearly and declaratively.
    • Atomicity
      Shock runs entirely within the test process. This means that you don’t have to clutter your app with excessive test code or test data, and that you have full control over the server and data.
    • Simple readable syntax
      Setting up Shock is very simple. You can have a test running using a mocked endpoint in a matter of minutes.

Getting Started

 

Preparing your App

To connect to the Shock server, your app will need to use the host address of the machine running the tests.
In your code change the base URL you are using to the following:

let baseUrl = "localhost:9000"

It’s advisable to make sure this is easy to switch between when running UI tests versus running a debug build. This can be accomplished by writing code similar to the following:

let isRunningUITests = ProcessInfo.processInfo.arguments.contains("UITests")
if isRunningUITests {
    apiConfiguration.setHostname("http://localhost:6789/")
}

If you have multiple base URLs (i.e. distinct APIs), don’t worry, you can modify them all to point at the Shock server – alternatively you can run multiple versions of the server on different ports and point at them individually.

let baseUrlA = "localhost:9000"
let baseUrlB = "localhost:9001"

(Note that these server instances will be separate and not share any mock data.)

Creating a Test

Create a new XCUITestCase subclass in your test target and add the following test method:

func testHappyPath() {
   mockServer = MockServer(port: 6789, bundle: Bundle(for: HappyPathTests.self))
   let route: MockHTTPRoute = .simple(
    method: .GET,
    urlPath: "/my/api/endpoint",
    code: 200,
    filename: "my-test-data.json"
)
   mockServer.setup(route: route)
/* ... Your UI test code ... */
}

Let’s take a look at this line by line.
First we create an instance of our mock server:

mockServer = MockServer(port: 6789, bundle: Bundle(for: HappyPathTests.self))

Here we specify the port we want the server to listen on; this should be the same port that your app is going to specify for it’s base API URL.

Next we create a route:

let route: MockHTTPRoute = .simple(
    method: .GET,
    urlPath: "/my/api/endpoint",
    code: 200,
    filename: "my-test-data.json"
)

This specifies:

  • the type of request (GET/POST/PUT/etc.)
  • the path we expect the app to call (/my/api/endpoint, which will become the absolute path http://localhost:6789/my/api/endpoint)
  • the status code we want to return to the app (200)
  • (and finally) a filename containing the response that should be included in the body of the request.

Lastly we pass the route to the mock server.

   mockServer.setup(route: route)

 
After this, run your test code as normal and you should see your UI being populated with the data found in my-test-data.json

Route Collections

Sometimes we want to create collections of routes that work nicely together for different parts of our application flow. To accomplish this we can use the “collection” route type, which takes an array of other routes as it’s only associated value

let route1: MockHTTPRoute = .simple(
    method: .GET,
    urlPath: "/my/api/endpoint",
    code: 200,
    filename: "my-test-data.json"
)
let route2: MockHTTPRoute = .simple(
    method: .GET,
    urlPath: "/my/api/endpoint",
    code: 200,
    filename: "my-test-data.json"
)
let routeCollection: MockHTTPRoute = .collection([ route1, route2 ])
mockServer.setup(route: routeCollection)

You can even make collections of collections and these will be handled and applied recursively by the mock server.

Templated Routes

Templates allow you to add a level of dynamism to your mocks. The problem with simple mocks is that they are static. For example, if you call the same endpoint with different query items – say /search?q=term vs /search?q=term2  – you’ll get the same response.
Templates provide a simple solution to this:

.template(
    method: .POST,
    urlPath: "/my/api/endpoint",
    code: 200,
    filename: "template-route.json",
    data: [ "templateKey": "A templated value" ]
)

The above will replace all instances of the templateKey token in your JSON – the templated JSON “template-route.json” looks like this:

{
    "Text": "{{templateKey}}"
}

When your app calls this endpoint it will get the following response:

{
    "Text": "A templated value"
}

Future Development

We use Shock extensively in our UI testing suites here at Just Eat, and so we are committed to keeping this repo up-to-date and bug free.
Suggestions for features are always appreciated and you’re free to submit a PR for any improvements you’d like to see – and it would be great to here what they are in advance: raise an issue in the repo and label it “Feature Request” if you have something cool in mind.
And of course, it goes without saying that bug reports – and fixes – are welcome too!