Generating SpecFlow reports for your TeamCity builds

You may or may not be aware that SpecFlow has the ability to produce some reports giving information about the features and steps after a test run has finished. The test execution report contains details about the SpecFlow test features, scenarios and steps. The step definition report shows the steps and their statuses, including unused steps and steps with no implementation.

The SpecFlow documentation details how the reports can be generated using SpecFlow.exe at the command line. Whilst it is useful to be able to manually generate these reports, it would be better if we could generate them as part of the TeamCity build process and make the reports available from within the build details of TeamCity. This makes the reports more accessible to all team members and especially useful when showing your product owner the scenarios covered by your acceptance test suite.

Lets run through getting things up and running.

1. Running the tests using nunit-console.exe

In order to generate the SpecFlow reports you must first run the tests using nunit-console.exe in order to generate the correct output files to feed into the report generation. The SpecFlow documentation tells us this is the command we need to run.

nunit-console.exe /labels /out=TestResult.txt /xml=TestResult.xml bin\Debug\BookShop.AcceptanceTests.dll

As we need to run nunit-console.exe it is not possible to use TeamCity’s NUnit runner. There are a few different runner types we could run this command such as Command line or Powershell, but the one that best fits our needs is probably MsBuild, but as always the same results can be achieved in lots of different ways.

In Visual Studio create a build file. I have given mine the imaginative name of nunit.build

<?xml version="1.0" encoding="utf-8" ?>

<Project ToolsVersion="3.5" DefaultTarget="Compile"
xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

<PropertyGroup>
    <NUnitHome>C:/Program Files (x86)/NUnit 2.6.2</NUnitHome>
    <NUnitConsole>&quot;$(NUnitHome)\bin\nunit-console.exe&quot;</NUnitConsole>
    <testResultsTxt>&quot;$(teamcity_build_checkoutDir)/TestResult.txt&quot;</testResultsTxt>
    <testResultsXml>&quot;$(teamcity_build_checkoutDir)/TestResult.xml&quot;</testResultsXml>
    <projectFile>&quot;$(teamcity_build_checkoutDir)\CrazyEvents.AcceptanceTests\CrazyEvents.AcceptanceTests.csproj&quot;</projectFile>
</PropertyGroup>
 
<Target Name="RunTests">
  <Exec Command="$(NUnitConsole) /labels /out=$(testResultsTxt) /xml=$(testResultsXml) $(projectFile)"/>
</Target>  
    
</Project>

I am not an MSBuild expert and it took me a while to get the paths correctly escaped.  If you are getting build errors check the paths first. TeamCity has some properties that you can reference in your build files to give you access the build related things. I am using teamcity_build_checkoutDir to specify that the output from the build is placed in the root of the checkout directory.

In TeamCity your build step should look something like this:

SpecflowTeamCityNUit1

With the build file in place and the step configured you may want to give things a run to make sure everything is working as expected. When you do you will notice that there is something missing from TeamCity.

SpecflowTeamCityNUnit2

If you had used the built in NUnit test runner you would expect to see a Test tab in TeamCity with the details of the tests run as part of the build. The tab is missing because TeamCity doesn’t know about the tests been run by the MsBuild task. The next step will show how to get the test tab and results back.

2. Getting the test tab to show in TeamCity

Luckily for us the good people at TeamCity have created an NUnit addin that does just that. To use the addin in needs to be present in the NUnit addin folder. The best way to ensure the addin is present in to add a Copy task to the MSBuild target. The full build file now looks like this:

<?xml version="1.0" encoding="utf-8" ?>

<Project ToolsVersion="3.5" DefaultTarget="Compile"
xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

    <ItemGroup>
        <NUnitAddinFiles Include="$(teamcity_dotnet_nunitaddin)-2.6.2.*" />
    </ItemGroup>

    <PropertyGroup>
        <NUnitHome>C:/Program Files (x86)/NUnit 2.6.2</NUnitHome>
        <NUnitConsole>&quot;$(NUnitHome)\bin\nunit-console.exe&quot;</NUnitConsole>
        <testResultsTxt>&quot;$(teamcity_build_checkoutDir)/TestResult.txt&quot;</testResultsTxt>
        <testResultsXml>&quot;$(teamcity_build_checkoutDir)/TestResult.xml&quot;</testResultsXml>
        <projectFile>&quot;$(teamcity_build_checkoutDir)\CrazyEvents.AcceptanceTests\CrazyEvents.AcceptanceTests.csproj&quot;</projectFile>
    </PropertyGroup>

    <Target Name="RunTests">
        <MakeDir Directories="$(NUnitHome)/bin/addins" />
        <Copy SourceFiles="@(NUnitAddinFiles)" DestinationFolder="$(NUnitHome)/bin/addins" />
        <Exec Command="$(NUnitConsole) /labels /out=$(testResultsTxt) /xml=$(testResultsXml) $(projectFile)"/>        
    </Target>

</Project>

An addins directory is created and the two addin files are copied from TeamCity to the directory. It is important to specify the correct version of the addin files in the NUnitAddinFiles section for the version on NUnit you are using. With the build file checked in the next build will have the missing Test tab, and the test summary will be displayed on the run overview.

SpecflowTeamCityNUnit3

3. Running the SpecFlow reports

To run the SpecFlow report I have used another build task to make two calls to SpecFlow.exe with the files generated as output by NUnit. The final build file looks like this:

<?xml version="1.0" encoding="utf-8" ?>

<Project ToolsVersion="3.5" DefaultTarget="Compile"
xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

    <ItemGroup>
        <NUnitAddinFiles Include="$(teamcity_dotnet_nunitaddin)-2.6.2.*" />
    </ItemGroup>

    <PropertyGroup>
        <NUnitHome>C:/Program Files (x86)/NUnit 2.6.2</NUnitHome>
        <NUnitConsole>&quot;$(NUnitHome)\bin\nunit-console.exe&quot;</NUnitConsole>
        <testResultsTxt>&quot;$(teamcity_build_checkoutDir)/TestResult.txt&quot;</testResultsTxt>
        <testResultsXml>&quot;$(teamcity_build_checkoutDir)/TestResult.xml&quot;</testResultsXml>
        <projectFile>&quot;$(teamcity_build_checkoutDir)\CrazyEvents.AcceptanceTests\CrazyEvents.AcceptanceTests.csproj&quot;</projectFile>
        <SpecflowExe>&quot;C:\Users\Admin\Documents\Visual Studio 2010\Projects\CrazyEventsNew\packages\SpecFlow.1.9.0\tools\specflow.exe&quot;</SpecflowExe>
    </PropertyGroup>

    <Target Name="RunTests">
        <MakeDir Directories="$(NUnitHome)/bin/addins" />
        <Copy SourceFiles="@(NUnitAddinFiles)" DestinationFolder="$(NUnitHome)/bin/addins" />
        <Exec Command="$(NUnitConsole) /labels /out=$(testResultsTxt) /xml=$(testResultsXml) $(projectFile)"/>        
    </Target>

    <Target Name="SpecflowReports">
        <Exec Command="$(SpecflowExe) nunitexecutionreport $(projectFile) /xmlTestResult:$(testResultsXml) /testOutput:$(testResultsTxt) /out:&quot;$(teamcity_build_checkoutDir)/SpecFlowExecutionReport.html&quot;"/>
        <Exec Command="$(SpecflowExe) stepdefinitionreport $(projectFile) /out:&quot;$(teamcity_build_checkoutDir)/SpecFlowStepDefinitionReport.html&quot;"/>
    </Target>

</Project>

Now we just need to get the html output of the reports to show in TeamCity.

4. Displaying the SpecFlow reports as tabs in the build output

Again, the people at TeamCity have made this really easy. First it is necessary to set the required files as build artifacts in the build configuration.

SpecflowTeamCityNUnit4

The final part of the jigsaw is to specify the report html file artifacts to be used as report tabs. Head over to Report tabs section of the Administration page and specify the new report tabs.

SpecflowTeamCityNUnit5

Now your tabs will be visible in TeamCity.

SpecflowTeamCityNUnit6

The easy way to run Selenium and SpecFlow tests in a TeamCity build with IIS Express

The series in full:

In Behavioural testing in .Net Part 1 and Part 2 I looked at creating behavioural acceptance tests with SpecFlow and Selenium. At some point after you have created your tests you will probably want to run them as part of a Continuous Integration build, or a nightly build or similar on a CI server such as TeamCity.

As the tests rely on browser interaction they obviously need the website to be running on the build server in order for the tests to function. There are several ways to get a web application up and running on the build server as part of the build, Web Deploy is probably one of the best.

However there are times you might want to get things up and running quickly as part of a spike or to just try something out. I am not suggesting that this is necessary enterprise strength but it may well be enough depending on your requirements.

This is not a TeamCity tutorial, so I am going to focus on the build steps required.

Step 1 – Build the project

This is fairly obvious. You cannot do much without a successful build.

Step 2 – Start the web application with IIS Express

IIS Express is a lightweight web server that is generally used for development purposes. It can be used in various ways and with different types of web application. For more information on how it can be used see this page. The interesting bit is that IIS Express can launch a web application directly from a folder from the command promp, and can be used for various types of application including ASP.Net, WCF and even PHP.

A command line runner is in order.

The text for the custom script is

start C:\"Program Files (x86)"\"IIS Express"\iisexpress.exe /path:"%teamcity.build.checkoutDir%\CrazyEvents" /port:2418

This tells TeamCity to start a new command window and run the IIS Express executable. The /path: parameter tells IIS Express which folder contains the web application. I have used a TeamCity parameter to specify the checkout directory. You need to make sure this points to the directory of your web application. If it is not correct the build log will let you know exactly where it is pointing and you can adjust it as required.

The /port: parameter specifies the port that the website will run on. This needs to be the same as the port specified in your tests.

Just a quick not about the script. TeamCity has some peculiar behaviour regarding double quotes in the command strings. It took a little while for me to get it right. If you have problems but it looks correct just double check the quote placement.

As another side note, you will not be able to see the command window when the step runs, but it will be running and you should be able to navigate to the website on the build sever.

Step 3 – Running the tests

Run the tests in the usual way with the correct build runner.

Step 4 – Stop IIS Express

The final step is to stop IIS Express. We will use another command line runner.

The custom script is:

taskkill /IM IISExpress.exe /F

which forces the IISExpress instance to close. Once again, you cannot see this in action but it is working in the background.

And that is all there is to it, quick and easy. Behavioural testing doesn’t have to mean a painful setup and teardown experience.

Creating a NuGet Package from a TeamCity build

Posts in this series:

In a previous post I talked about how it was possible to set up .Net dependency management in TeamCity using NuGet. In this post I will cover the steps required to create a NuGet package as part of a TeamCity build, and publish it to TeamCity’s own NuGet package source server.

It is not uncommon to want to reuse a .Net project in another solution, maybe some common data access code. This would mean a bit of a maintenance headache when trying to keep things up to date and versioned correctly. NuGet and TeamCity have now made things very easy. Create a Nuget package for the project when it is built in TeamCity and publish it to TeamCity’s own Nuget package server. This package can then be picked up and used in any other project as a dependency like you would use an external package from the NuGet official package source.

Turn on TeamCity’s NuGet server

Since version 7, TeamCity has been able to host its own Nuget package feed. To enable it, head over to TeamCity’s Administration page and select NuGet Settings.

You will be presented with the URL’s for the NuGet feed. The feed URL will be added in Visual Studio and TeamCity as a package source to allow you to use the packages you create. If you want to turn on the public feed you will need to enable Guest login for TeamCity.

Create a package from a .csproj file

It is possible to create a NuGet package directly from a .csproj file in TeamCity. However if the project from which you wish to create the package has NuGet packages itself you will encounter a couple of errors after setting up a build step and attempting to run the build.

The first will probably be this one:

The imported project "C:\Program Files (x86)\TeamCity\buildAgent\work\c4ce4cf3674f70b0\ExampleProject\ExampleProject\.nuget\nuget.targets" was not found. Confirm that the path in the declaration is correct, and that the file exists on disk. C:\Program Files (x86)\TeamCity\buildAgent\work\c4ce4cf3674f70b0\ExampleProject\trunk\DataAccess\DataAccess.csproj

This problem is caused by some changes made by NuGet to the .csproj file resulting in TeamCity’s build agent not being able to resolve the path correctly.  A solution to this problem is outlined here. However once the fix has been applied you may well find that you encounter another error:

nuget pack: Specifying SolutionDir in properties gives Exception: An item with the same key has already been added.

This looks like it is a Nuget error. An issue has been raised here but at the  time of writing it has not been actioned. If anyone has a solution I would be very interested to hear it.

Don’t despair at this setback. Even if this approach worked correctly it is a bit of a blunt tool. Much better is to use a .nuspec manifest to describe the package we want to build.

Create a package from a .nuspec manifest

A .nuspec manifest is an XML document that defines the content of a NuGet package. A lot of effort has gone into the NuGet documentation on .nuspec files so I am not going to recreate all of it here. Instead I will just show you .nuspec file I used and talk about a few interesting parts. First of all, create a file next to the .csproj of the project you want to package. I want to package my DataAccess project so I created a file called DataAccess.nuspec.  Here is the content:

<?xml version="1.0"?>
<package xmlns="http://schemas.microsoft.com/packaging/2011/08/nuspec.xsd">
  <metadata>
    <version>0.00</version>
    <authors>James Heppinstall</authors>
    <owners>James Heppinstall</owners>
    <id>DataAccess</id>
    <title>DataAccess</title>
    <requireLicenseAcceptance>false</requireLicenseAcceptance>
    <description>Description</description>
    <copyright>Copyright James Heppinstall 2012</copyright>
    <dependencies>
        <dependency id="PetaPoco.Core" version="4.0.3" />
      </dependencies>
  </metadata>
  <files>
    <file src="ExampleProject\trunk\DataAccess\bin\Release\DataAccess.dll" target="lib\DataAccess.dll" />
  </files>
</package>

Most of it is straight forward and is covered in the NuGet documentation. It doesn’t matter what the version number you put in. It will be replaced by the build number from the TeamCity build. The dependencies section should contain a list of the NuGet packages for the project. If you are unsure which packages your project uses the best thing to do is to look in the project’s packages.config for the correct name and version.

The files section is where you list the files to be included. I have just included the single .dll as an example. You can also target specific framework versions in the same package using a convention. If I wanted to target only .Net4.0 I would have used

<file src="ExampleProject\trunk\DataAccess\bin\Release\DataAccess.dll" target="lib\net40\DataAccess.dll" />

Remember that a  NuGet package is not just limited to dependencies. It is possible to create a package that contains an entire project structure with code and configuration, including scripts to perform data transformations along the way. So you could create a template package for your organisation’s web site development.

Now we have the .nuspec file the next step is to create a new build step for the TeamCity Project.

Simply select a Runner Type of NuGet Pack, add the path to your .nuspec file and you are good to go. I have selected to publish the package to the build artifacts. This can be useful for audit-ability.

Run the build and check that the Package is created in the build artifacts. If all is good the package is ready to be used in Visual Studio. From Tools | Library Package Manager | Package Manager Settings and select package sources. Add the TeamCity feed URL you wish to use, either authenticated or guest (I have added both for illustration purposes). Of course you should use the correct server name as you will probably not be running TeamCity on your development machine.

Now when you go to Manage Nuget Packages for a project you will be able to select the TeamCity package feed and see the Package you have just created, ready to be added to a new project.

And that’s all there is to it. NuGet and TeamCity make dependency management for your .Net code base very simple.

Dependency Management in .Net with NuGet and TeamCity

Posts in this series:

Dependency management for .NET has traditionally been via dependencies being stored in a ‘Lib’ folder, or something similar, which is then checked into source control as part of the Solution. This means that the complete Solution can be pulled from source control and built with all the dependencies in place. Whilst this is seen as best practice by many .NET developers it is not without its issues, not least a rapid growth in repository size when a project has many dependencies. In most .Net shops it is done without question, as just the way it is done. But there is another way…

Developers familiar with Java will no doubt be aware of the dependency management options available with tools as Maven (which provides much more besides just dependency management), but as yet .Net has resisted this kind of dependency-less source control approach.

New additions to the NuGet package management tool now move .NET towards true dependency management, and when coupled with JetBrains’ popular Continuous Integration (CI) server TeamCity dependency-less check in can become a reality for .Net development teams.

NuGet

What is it?

NuGet is an open source package management tool for the .Net platform. The project is currently lead by Phil Haack (formally of Microsoft and now GitHub) and has several prominent Microsoft contributors. Originally a Visual Studio Extension to easily incorporate dependencies into projects; it has evolved several features to support dependency management. A lot of the pressure to move in this direction has come from the community, and it is likely to evolve further still.

What is a package?

A package can contain many different items wrapped in a .nupkg zip. A simple library may just contain a .dll to add to the project references, but it is also possible to have a package that contains an entire project, for example a website template. This could include libraries, source code, pre-processing (such as updating namespaces and project names), merging items into web or application configuration, and even PowerShell scripts to perform actions.

Installing and using NuGet

In Visual Studio 2010 navigate to Tools | Extension Manager. Search the Online Gallery for NuGet and install. You must run Visual Studio with full privileges to do this.T he Tools menu should now contain section ‘Library Package Manager’ with some options for working with NuGet.

NuGet Options

To install and use a package, there are a couple of different ways:

  • ‘Manage NuGet Packages’ Dialog – Simply right click a project in the Solution Explorer, select ‘Manage NuGet Packages…’and search for the required package and select to install.

NuGet Packages Dialog

  • Package Manager Console – For those more inclined to the command line you can use the Package Manger Console to install a package using the Install-Package command with a package Id. The console can be accessed from the ‘Library Package Manager’  menu mentioned above. Remember to select the Package Source and the project to install the package in. It is also possible to install a specific version of the package by adding the version number after the Id.

Package Manager Console

A package contains everything needed to install and use the selected library or tool. If the library selected has further dependencies they too will be downloaded and installed if they are not already present. Nuget will take care to get the correct versions of any dependencies that are required. So in the example above we can see that Unity also has a dependency on CommonServiceLocator, which is downloaded by NuGet.

As part of the installation required files are copied to the solutions ‘Packages’ folder. If it does not exist it will be created for you. Also, any references will be added along with any required changes to configuration files.

A package can be removed from a project using either of the two methods above (except using the Uninstall-Package command in the console). All changes made by Nuget are reversed when a package is uninstalled.

Package Source

Packages live in a repository called a Package Source. The default package source is the NuGet Official Package Source. At the time of writing it contains nearly 6000 packages available for download.

A public package source may not be suitable for all NuGet users and it is possible to use a private repository. This could be something simple like a folder on a file share within an organisation, or a feed hosted in a web environment.

Setting the repositories to be used by NuGet can be done from Tools | Library Package Manager | Package Manager Settings. Simply add any package sources required and arrange the order of preference to indicate where NuGet should look first.

Package Source Selection

Publish a package

A package can contain many different items wrapped in a .nupkg zip. A simple library may just contain a .dll to add to the project references, but it is also possible to build a package that contains an entire project, for example a website template. This could include libraries, source code, pre-processing (such as updating namespaces and project names), merging items into web or application configuration, and even PowerShell scripts to perform actions.

A package can be added to a source by Publishing. To publish to a public source (such as NuGet.org) a user must register and receive their API key to use during publishing. To publish to a file location, just copy the .nupkg file to that location.

Of course, if you like to go retro, there are comprehensive command line tools for the creation and publishing of packages. However the easiest way to create and publish is via the NuGet Package Explorer. This little utility allows you to:

  • Download packages from a source (handy if you want to copy a package to a local repository)
  • View the metadata and contents of a package
  • Create a new package
  • Publish a package to a repository

 Enabling Package Restore

In the early days of NuGet the common workflow for source control of the NuGet Packages was simply to add the Packages folder to source control. Due to popular demand additional functionality has been added to NuGet to support dependency less check in.

Package restore will automatically pull and install any missing packages when the project is built. To turn on Package restore simply right click on the Solution in Visual Studio 2010 Solution Explorer and select ‘Enable NuGet Package Restore’.

A .nuget folder will be added to Solution root which contains the items needed to enable package restore to function.

Next we will look at how Package Restore is used with TeamCity to allow dependency less source control for your .NET Solution in a CI environment.

TeamCity With NuGet

Once we have enabled  Package Restore for a Solution in source control we can add build configuration in TeamCity for the project. This guide is not intended to be an introduction to TeamCity, and as such will mostly gloss over the other build configuration settings not directly related to using NuGet packages.

Build steps to restore packages

After configuring the general settings and version control settings the first build step should be to restore the dependencies from a Nuget Package source.

TeamCity has done a  great job of making the configuration really easy. Specify the build runner as NuGet package installer, select the version of NuGet and specify the path to the solution file and you are ready to go. It is possible to supply package sources if you want to use any other than the default source on-line, for example if you want to use a separate repository for your organisation.

That’s about all there is to it. The dependencies specified in your solution will be populated at build time. Its time to say goodbye to your lib folder.

Why not take a look at how you can use TeamCity to create a NuGet Package from a project and publish it to it’s own NuGet Package feed?