Acceptance Testing with Specflow

What is Acceptance Testing?

If you are an avid unit tester, you will know the value of breaking down your production code, component by component, and running modular tests. Acceptance testing takes a high level approach, testing the streams of functionality that run through your system and attempts to recreate the actions of a user or consumer of your application. The idea behind acceptance testing is that your Product Owner, or the representative of your users or customers, should be able to write the tests in plain English and the developer should be able to implement the test script in code. Frameworks such as Specflow and Cucumber provide the infrastructure to join up the plain text specifications with the supporting code.

Why Acceptance Test?

Acceptance tests add value to a code base in several ways:

  • They help you to verify the code that you’ve written fulfils any functional requirements you had for writing that code and they help you to pick up on defects.
  • A suite of acceptance tests can help you to regression test your system. When a developer makes a change he or she is able to run the acceptance tests locally before checking-in to make sure his or her changes haven’t broken existing code. Many teams also set their test suite to run on the build server and fail the build if any tests fail.
  • Well written tests provide a form of living documentation for your system. They set out functionality supported by the system and they provide detail of how your software should handle both the happy path and other common routes through your application.


Acceptance testing at JUST EAT

At JUST EAT we use acceptance tests to quality assure our code. We run our tests on the Team City build server every time someone checks in some new code to protect against regression. We leave the specifics of acceptance testing up to each team but the general approach is to write acceptance tests every time a new piece of functionality is introduced. The responsibility for writing tests sits with the developer or developers who wrote the functionality as we believe in taking ownership of testing our own code and making sure it’s of a high quality standard.

Writing Tests

If you work with user stories, the acceptance criteria for these can be a good place to start. If not, it still makes sense to break down your tests so that each is only responsible for a discrete area or feature of your system. This way, if you get a failing test you will have a better idea which area of your system is at fault and you’ll be able to keep your testing code more light weight. Acceptance tests are commonly written in the “Given…When…Then” format. This annotation sets out your test’s starting point (Given), some form of interaction with your system (When) and the outcome that you expect (Then). For further detail on the “Given…When…Then” syntax a good place to look would be Martin Fowler’s blog

Getting Started

Specflow is an acceptance testing tool that can be added as an extension into Visual Studio. Specflow uses Gherkin which is a language designed to be easily readable so that business users can write test definitions which developers can then implement as an acceptance testing suit. A specflow scenario looks something like this:
A Valid User Can Login
Under the hood, Specflow binds to steps in “Step Definition” files using attributes. Here, the “When I login” step from the above example is bound to the method WhenILogin(). The method also has an attribute tying it to a “Given I login” step. You can add as many “[Given()]”, “[When()]” or “[Then()]” attributes to your steps as you like.When I Login

Passing Parameters and Variables

Often, it’s useful to pass different variables through to a step as in the “Given” step from the example above.A Valid User Can Login - Given
To provide an implementation for the above, in your step file you would simply add a method which accepts two strings and uses the “‘(.*)’” expression to identify to Specflow where to expect the variables to be passed through in the string. The same implementation would work with integers and enums.Given I Am A User
If you want to pass through lists of variables you can use a table. Specflow provides a Table type which allows you to do this using a table format in your scenario which would look something like this:A Valid User Can Login - Table
The “|” annotation is used to define individual cells and the header names are used as keys for each column when accessing the values in your code. In your step implementation you can then loop through each row in your table to access the values passed in.When I change my address details
Specflow also allows you to pass blank values; simply leave the cell blank in your scenario.When I change my address details table

Passing or Failing?

In order to pass or fail a test, you need to add an assertion within your “Then” step. A unit testing tool such as NUnit works well for this and allows you to make assertions based on the outcome of your scenario. Specflow works really well alongside a tool such as Selenium which provides a framework for interacting with a browser in the same way that a user might. Selenium would allow you to provide an implementation for your scenario which spun up a browser, logged in and then checked the title of the page to make such that the user had landed on the welcome page, as expected. Selenium is a topic to be covered in a later post, so for now this implementation has been abstracted away into the “IAmOnTheWelcomePage()” method call.I should be on the welcome page

Scoping Scenarios

Specflow will bind scenario steps from any feature file to any matching step definitions it finds. If you have more than one matching binding your test will fail. If you want to reuse the same binding text but you want to provide different implementations you can use the scope attribute. You can scope steps or whole step classes to to features, scenarios or to a given tag.
Here’s an example of two different implementations for the same “When I click save” scenario step. In the left hand example, the implementation relates to creating orders, whereas on the right the example relates to editing details. In order to distinguish between the two I can use the scope attribute in conjunction a tag.Scoped
For my scenario, in order to distinguish that I want to use the “CreateOrderSteps” version of my “When I click save” step, I can simply add the “@Order” tag and Specflow will bind to the scoped step.Users Are able to edit their orders

Organising your Code

In Specflow you will create your scenarios in “feature” files. A sensible approach is to create many feature files with relating step definitions for sets of user-centric functionality within your system. Building your features on tasks which users would undertake will help you to maintain focus on how a user would use your system and make sure that’s what you’re testing.Solution

Structuring your tests in this manner will also allow you to run suites of tests more easily, either locally through visual studio or within your Continuous Integration environment on your build server. This can be very useful if you build up a larger suite of tests and one to focus on subsections in order to reduce the feedback loop.