Getting started with automated acceptance tests in an Agile project

Automated testing can bring great benefits to an Agile project but it is not always clear how to best integrate the processes and practices effectively. In this post I will outline a few ideas that might help, along with some pitfalls, based on experience gained from my own projects.

What is automated acceptance testing?

I am using the term automated acceptance tests to mean tests that exercise a system from it’s most external interface. For the purposes of this post I will be mainly talking about tests that exercise the user interface but it could also mean tests that exercise an externally visible API such as a web service. The important thing about the tests is that they test the expected behaviour of the system, ensuring that the desired functionality is met. This builds an executable specification for the system.

What does it bring to an Agile project?

In this post I am going to use terms that are from Scrum in particular, but the points covered are applicable to any Agile process (and more than likely non Agile processes as well).

Agile projects are focused on meeting the needs of their customers (Product Owner in Scrum terms) and one of the ways this is done it to use acceptance criteria to define the expected behaviour of a particular user story. It is very valuable to be able to prove that that the acceptance criteria have been satisfied, whilst building a very comprehensive suite of automated regression tests. A suite of automated acceptance tests gives incredible confidence when building the system, especially in Agile developments where there may be refactoring and changes of direction as the changing needs of the Product Owner are met.

Where does automated testing fit into the process?

One of the most important things I have learnt from using automated acceptance testing on an Agile project is that it has to be a first class part of the development process. Depending on the maturity of your team this may mean having an explicit step or task to ensure it gets done, or having it as part of your definition of done. Either way it needs to be given the same level of importance as the other practices. A story cannot be done without automated tests that at least cover the acceptance criteria of the story. It is almost certain that when time is tight and people are under pressure that the automated tests will be the first things to be skipped. This is almost always a mistake, and one which I have learnt the hard way. Missing acceptance tests will come back to haunt you. Include them in your process and account for them in your planning activities.

Who writes the automated tests?

The creation of automated acceptance tests requires input from all the members of an Agile team. The roles and make up  in your Agile team may differ, but typically acceptance criteria will be generated with the input of the product owner and analyst (and maybe also the tester in some teams) who will also create specific scenarios that will be automated. As the tests are automated there is obviously some development work in order to perform the automation. There are ways to gain a lot of reuse form a test suite and to reduce the amount of additional development required to extent the suite. I will talk about some of these when looking at some specific tools and technologies in a later section.

What does it mean to ‘traditional’ testing for the project?

In recent years I have become more convinced that it is no longer possible to speak about the traditional silo’d roles in software development. The roles of analyst, developer and tester continue to become blurred as roles change and responsibilities overlap. The creation of automated acceptance tests require a little bit of all these traditional disciplines. Who will be involved in your team depends on the skills and abilities of the team members. To make automated testing work in your team it will be necessary to let go of the idea that developers don’t do testing, or that testers don’t write code.

That said, automated acceptance tests are not meant as a replacement to traditional exploratory testing or to undermine the role of traditional testing in any way. Instead they should be seen as complimentary.

This is a great post that discusses the aims of automated and exploratory testing:

Automated vs. exploratory – we got it wrong!

Automated acceptance testing as part of the build process

To get the most out of automated acceptance tests they need to be part of your Continuous Integration build. Acceptance tests are never going to be fast running, and this can pose some challenges to getting the best feedback from your tests. As an indication, my current project has about 300 Selenium based tests and takes about an hour to run. We are 7 sprints in and the number of tests is going to get much higher. At present we have a fast CI build running only the unit tests to give instant feedback, followed by an integration build containing all the acceptance tests that runs after a successful CI build. We are finding that the feedback is a bit slow and are planning to introduce the concept of a ‘current’ integration build that will run a subset of the acceptance tests that have been tagged as being in the area which fast feedback is required (i.e. area in which work is being undertaken). The full set of tests will continue to run at their slower cadence.

Using the right tools

My tools of choice are SpecFlow and Selenium Web Driver, but there are others that perform a similar function. It is important to understand the tools and the best way to use them. When talking about automated acceptance tests, and specifically using Selenium to drive the browser, I often encounter people who have tried to do automation and then have stopped as the tests become brittle and the cost of maintenance becomes too high.  Here are a few tips from my experience of using automated acceptance tests on a large project:

  • Aim to get a lot of reuse from your SpecFlow scenarios. Use the features built into SpecFlow, such as the Scenario.Context.Current and it’s DI capabilities, to make steps that can be reused across many features. If done correctly you will find that it is very easy to add new tests without writing any new steps at all.
  • Us the page object pattern to encapsulate the logic for the web pages. This is the one of the single biggest factors to ensuring your test suite is maintainable. Without page object you will find yourself fixing lots of tests due to a small change to a single page.
  • Thing about the wider ecosystem. We are using Selenium Server to give us the ability to run tests across multiple browsers.
  • Learn about you tools to get the best from them. Writing the automation will be a lot more pleasurable of you understand how things are working.

The posts below discuss how to make a start writing maintainable automated acceptance tests:

Behavioural testing in .Net with SpecFlow and Selenium (Part 1)

Behavioural testing in .Net with SpecFlow and Selenium (Part 2)

Managing SpecFlow acceptance test data with Entity Framework


  • Automated acceptance tests are about building an executable specification for your system.
  • The executable specification is very useful to to prove the acceptance criteria have been met and to give confidence that changed have not broken functionality.
  • Make automated tests part of your process. Do not skimp on them, even if you are under pressure to get things done. Add them to your definition of done.
  • Creating the tests is an activity for all the team, although traditional roles and responsibilities may be blurred.
  • Automated tests are in addition to traditional exploratory style testing and are not meant as a replacement.
  • Get the most from your tests by choosing a sensible CI strategy to give the best feedback
  • Choose the right tools for the job, and spend time learning to get the best from them
  • Become an advocate of automated testing in your organisation

2 Responses to Getting started with automated acceptance tests in an Agile project

  1. Pingback: Behavioural testing in .Net with SpecFlow and Selenium (Part 1) | James Heppinstall: On Development

  2. Pingback: Behavioural testing in .Net with SpecFlow and Selenium (Part 2) | James Heppinstall: On Development

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: