Fluent Bytes

"The only source of knowledge is experience" - Albert Einstein-

Running XUnit test with ASP.NET DNX and TFS 2015 build

Recently I joined a project where they are working on a new application that is based on the new ASP.NET cross platform tooling. While this is still in beta at the moment I write this, we strongly believe in the future of ASP.NET and the ability to run it on multiple platforms. Of course we want this project to start with everything done right from the start, so we are not only writing clean code, we also embrace the continuous delivery concepts and with that we have a strong need of continuous integration and continuous tests. This means we are writing Unit tests from day one and want those integrated in our build system. For the build we use Team Foundation Server 2015 and the new cross platform build system. The challenge I want to discuss today is that the tooling at the moment is still a lot in flux so the unit test tooling for testing cross platform solutions is not fully up to par as you are used to in the “classic” ASP.NET world on just windows.

So when we want to write unit tests, we need to ensure we use a test framework that also supports running on multiple platforms. At the moment XUnit is the test tooling that is already available for DNX. So let me describe the steps involved for making a set of unit tests and run those tests on the new build platform in TFS 2015.

Start with creating a DNX Unit test project

First we start with a class library project that is part of the new ASP.NET toolset. In the screenshot here you can see the library project you need to select.

clip_image002

Next we need to get the XUnit Tools so we can start writing tests. For this you go to the Nuget Package manager for the project and there you select the following packages:

clip_image004

Ensure your Project.json file contains a test command

The Project.json file contains besides a set of dependencies also the option to define commands you want to run in the DNX environment. What we are going to do is specify a command called “test” and we specify we want the XUnit DNX runner to be used for that command. Since we need the test results to be captured in an XML file so we can upload them to TFS as part of the build we specify additional options at the command to output results to an xml file with the –xml option and provide it a file name of the results file. The Project.json file looks then as follows (the blue underline shows the test command:

clip_image006

Write a XUnit test

I won’t go into details about writing XUnit tests, you can read a lot more about this here (http://xunit.github.io/docs/getting-started.html) . For now I will show you the code for the class that I called “SomeTests” that contains two tests.

clip_image008

Running the test local

Since we installed the DNX runner using NuGet, after a build the visual studio environment should now show your tests in the test runner automatically after you compile the project.

clip_image010

So everything now runs in visual studio, question remains can we also do things from the command line? To verify this you can start a command prompt at the current solution and then try to run the command

dnx .\src\classlibrary1 test

This will probably result in the error that the command dnx is not recognized. This has to do with the fact that we first need to select the right dnx run environment and you can do so with the command line:

dnvm setup

This will setup the dot net environment and make it part of your current path. Now we need to select a dot net framework we want to run on. Since we want the ability to run on non windows environments you can then select the “core clr” runtime. You can first list all the runtimes available with the command line:

dnvm list

This shows a list like here below:

Active Version Runtime Architecture Location Alias

—— ——- ——- ———— ——– —–

1.0.0-beta5 clr x64 C:\Users\vries\.dnx\runtimes

1.0.0-beta5 clr x86 C:\Users\vries\.dnx\runtimes

1.0.0-beta5 coreclr x64 C:\Users\vries\.dnx\runtimes

1.0.0-beta5 coreclr x86 C:\Users\vries\.dnx\runtimes

Next I select the runtime I want to use by using the command:

dnvm use 1.0.0-beta5 –architecture x86 –runtime coreclr

Now we have set up the runtime environment and now you can type the command:

dnx .\src\classlibrary1 test

and now you will see the test results shown at the console:

xUnit.net DNX Runner (32-bit DNXCore 5.0)

Discovering: ClassLibrary4

Discovered: ClassLibrary4

Starting: ClassLibrary4

Finished: ClassLibrary4

=== TEST EXECUTION SUMMARY ===

ClassLibrary4 Total: 2, Errors: 0, Failed: 0, Skipped: 0, Time: 0.106s

You will now also find an xml file that contains the test results if you look in the location where you executed the command. The result file now contains the test results in the XUnit test format. This format is at the moment not recognized by TFS, they do support MS Test trx format and NUnit out of the box. So we need to do some additional work to make the results usable by TFS build.

Building the project using TFS build 2015

Before we can build anything on the server we need to commit our changes locally in our GIT repository and push our changes to the TFS server in a team project that has GIT as version control selected. (you can also work with Team Foundation Server Version control, but since we work cross platform and GIT is available on any platform without MS tools I selected GIT. You could also use Team explorer Everywhere to run from Unix, mac or Linux systems)

In my example here I use Visual studio Online, but this will also work with a on premise server that has TFS 2015 installed. (2015 contains the new Build features I am using here)

When at the TFS web page for build we create a new build definition and call it “ASP.NET VNext with XUnit”. Next we start with a build template that has the basic compile things in there.

clip_image012

Next we add a new build step of type PowerShell script so we can set up the DNX environment like we just did, but now on the build server.

clip_image014

Move this step to the top of the build, so it becomes the first step

Next we need to add a script to our project to install the dnvm tools and then select the right dnx environment for the compilation of the project to work. If we don’t do this you will get the error message in the compile step:
Error : The Dnx Runtime package needs to be installed. See output window for more details.

You can copy the PowerShell script from the following MSDN page that describes this first step:

https://msdn.microsoft.com/en-us/Library/vs/alm/Build/azure/deploy-aspnet5

or just copy the contents here:

I called the script Install_DNX.ps1 and I added it to the solution items folder in my solution

clip_image016

Now you can select this file in the PowerShell step as follows:

clip_image018

When we have this setup, we can now setup the unit testing part. Since the Visual Studio Test step requires an visual studio runner plugin that runs on the full .net framework, I wanted to ensure my tests run clean and also can be run on a build host that I run on a Linux system, so I don’t want to rely on MSTest with an MSTest adapter specified. So I deleted the Visual Studio Test step and replaced it by a PowerShell script that does exactly what we have done in the local run from the command line. So the script for that looks as follows:

Set-ExecutionPolicy unrestricted -Scope CurrentUser -Force

dnvm setup

dnvm use 1.0.0-beta5

dnx $PSScriptRoot\src\ClassLibrary4 test.\nxslt2.exe $PSScriptRoot\testresults.xml .\NunitXslt.xslt -o testresultstransformed.xml

Now note the last step of the script. What I did here is leverage a Xslt transformation to change the Xunit format that is currently not recognized by the TFS build an change it to a format it understands Nunit. For this I added an executable called nxslt2.exe that Scott Hanselman described on his blog (http://www.hanselman.com/blog/XSLTWithPowershell.aspx) to do simple transformations using PowerShell. I just added the executable also to my solution items folder. The Xslt file I pulled from the github repository of Xunit. In their runners they normally support Nunit as an output format, but this is not supported yet in the dnx runner at the moment. You can find the xslt file here: https://github.com/xunit/xunit/blob/master/src/xunit.console/NUnitXml.xslt

I also added this file to my solution items folder, so I have everything in one place. The output of the transformation is now the file called testresultstransformed.xml and that is the file we are going to upload to the TFS server.

I gave the script file the name RunDnxTest.ps1 and placed it in the Solution Items folder. Here you can see the screenshot of my solutions Items folder in my project that includes the Xslt file, the nxslt2.exe file and the PowerShell script:

clip_image019

One final thing to note is that the script assumes a directory from which it runs. You can specify this in the PowerShell task and I specified it will be the solution root of my project when it is pulled from the GIT repo:

clip_image021

And after adding the script to the solution items folder in my solution and pushing it to the remote repo, you can select it as the PowerShell build step to run the tests.

clip_image022

Ensure that for the PowerShell task you check the option Continue on Error, since when one of the tests fails otherwise your build will be flagged as failed and no results will be uploaded to the server which is the next task.

clip_image024

The final thing we want to add is the task to upload the results file after a test is completed.

clip_image026

When we configure this task we need to select which output result we want to use. Here we select NUnit, since that is the format we transformed the output to in the final step of our PowerShell script that runs the tests. We select the transformed results file and then we are done.

clip_image028

Now we can queue a build on one of our hosted build machines or on your on premise deployed agent.

In my case I selected my own build server that runs in an Azure virtual machine:

clip_image030

And after running the build you can now see the test results uploaded to the TFS server:

clip_image032

And clicking the results there brings you to a view with some charts about your run:

clip_image034

Summary

So after all these steps we have now an ASP.NET DNX Class library project that we can use to define XUnit test and we can run them on the new build infrastructure TFS 2015 provides. The final step here would be to change the scripts from PowerShell to SSH and then run them on a Linux machine, but for that I do need to change the way I do the XSLT transformation ,since the tool is windows only at the moment. I also should change the install DNX script and the Execute DNX tests script to a SSH and then I should be able to run it on a Linux based agent. I leave this as an exercise for later

Hope this helps

Marcel

CTO at Xpirit, Microsoft Regional Director, Visual studio ALM MVP, Speaker, Pluralsight Author and IT Architect Consultant

12 Comments

  1. Hi,

    I’m trying to do the same and I get a well formed NUnit xml output, and the discovery also goes well, but it doesn’t show the test results. (All tests passed: http://i.imgur.com/3mDPzRL.png and 0 tests http://i.imgur.com/uAHdUyL.png)

    Also I have one test explicitly made to fail, but that also doesn’t show up. It seems that TFS doesn’t interpret the file correctly. When I commit this file it does show up nicely, so maybe there is a problem with the test results. I’ve posted a gist with more info: https://gist.github.com/Alber70g/1d315eda01509cb10aa2

    Maybe you can help me?

    • I looked at it and uploaded your transformed file using my build in my post (just by changing the publish task to take your file in stead of my file)
      the results in the build show me this:
      buildresultsnunit
      and the test run details:
      testrunnunit

      so it looks like it does work. are you sure you are uploading the right file? and did you hit the refresh button on the test, since it sometimes shows stale data at the moment?

    • It may be that you’re missing the “View Test Runs” permission. The default Build will show is that no tests were found, not that you don’t have access to them.

  2. Hi, Marcel and Albert.

    The instructions are correct, except for one detail: the XSLT file needs to be changed for test result reporting to work. It only takes a couple of minor changes, though, and the result is here:

    https://gist.github.com/rytmis/f2319693dd33049eeab0

    I went through the same steps a while back, got the tests to run and got the dashboard results like Marcel — but when a test failed, I couldn’t find the results, like Albert. Took me a while of communicating with the VSO team to figure out that the transformation didn’t produce a valid NUnit file because the new XUnit output format had changed slightly.

    Hope this helps. 🙂

  3. Hi
    Follow the step during creating RunXunit.ps1

    Set-ExecutionPolicy unrestricted -Scope CurrentUser -Force

    dnvm setup

    dnvm use 1.0.0-beta6

    dnx $PSScriptRoot\test\Wallet.Core.Tests test.\nxslt2.exe $PSScriptRoot\testresults.xml .\NunitXslt.xslt -o testresultstransformed.xml

    I got an error like this

    System.InvalidOperationException: Failed to resolve the following dependencies for target framework ‘DNX,Version=v4.5.1’:

    EntityFramework.Commands 7.0.0-beta6

    EntityFramework.Core 7.0.0-beta6

    EntityFramework.SqlServer 7.0.0-beta6

    Microsoft.Azure.SqlDatabase.ElasticScale.Client 1.0.0

    Microsoft.Framework.Caching.Memory 1.0.0-beta6

    Microsoft.Framework.Configuration 1.0.0-beta6

    Microsoft.Framework.Configuration.Json 1.0.0-beta6

    Microsoft.Framework.Runtime.Abstractions 1.0.0-beta6

    xunit 2.1.0-beta3-build3029

    xunit.runner.dnx 2.1.0-beta3-build99

    xunit.runner.visualstudio 2.1.0-beta3-build1069

    Searched Locations:

    C:\a\798feaf9\Wallet\test\{name}\project.json

    C:\a\798feaf9\Wallet\src\{name}\project.json

    C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5.1\{name}.dll

    C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5.1\Facades\{name}.dll

    C:\Windows\Microsoft.NET\assembly\GAC_32\{name}\{version}\{name}.dll

    C:\Windows\Microsoft.NET\assembly\GAC_64\{name}\{version}\{name}.dll

    C:\Windows\Microsoft.NET\assembly\GAC_MSIL\{name}\{version}\{name}.dll

    Try running ‘dnu restore’.

    at Microsoft.Framework.Runtime.DefaultHost.GetEntryPoint(String applicationName)

    at Microsoft.Framework.ApplicationHost.Program.ExecuteMain(DefaultHost host, String applicationName, String[] args)

    at Microsoft.Framework.ApplicationHost.Program.Main(String[] args)

    — End of stack trace from previous location where exception was thrown —

    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

    at Microsoft.Framework.Runtime.Common.EntryPointExecutor.Execute(Assembly assembly, String[] args, IServiceProvider serviceProvider)

    at dnx.host.Bootstrapper.RunAsync(List`1 args, IRuntimeEnvironment env, FrameworkName targetFramework)

    at dnx.host.RuntimeBootstrapper.ExecuteAsync(String[] args, FrameworkName targetFramework)

    at dnx.host.RuntimeBootstrapper.Execute(String[] args, FrameworkName targetFramework)

  4. I am trying to run our xUnit tests using the xUnit test adapter in Visual Studio Online s Build vNext platform. As stipulated in this article, we need to provide a custom test adapter path pointin.

    • At the moment you can not provide a path outside the standard version control or build scope. since the adapter is in your dnx profile under packages you can not point to it. hence the work around using the command line using a script.

      • This must have been fixed some time ago, as you can just enter the path manually (don’t use the file picker) and it helps to pre-fix it with $(Build.SourcesDirectory)\Path\To\Packages\ this will be transformed to the right path on disk automatically.

        Or by keeping the path empty, as packages restored by NuGet should automatically get searched, though this may not work for DNX solutions (yet).

  5. Hi,

    Thanks for posting these instructions, very helpful. When I run my tests I’m getting an error message “Could not load file or assembly ‘Microsoft.DNX.PackageManager'” – any ideas?

  6. Great post, very descriptive but I am getting confused by the names you used for your main and test projects. Is ClassLibrary1 the main project and is ClassLibrary4 the test project? In my solution I have ‘Solution Items’ folder that contains the global.json, prebuild and RunDnxTest.ps1, nxslt2.exe and NunitXml.xslt. I have two more folders in my solution, ‘src’ which holds the main project and ‘test’ folder that contains the test class library.
    When running the xunit tests through command line I opened the command prompt for the solution and after setting the dnvm should I execute
    ‘ dnx .\src\SampleProject test’ or ‘dnx .\test\SampleProject.Test test’ ?
    I get the Unable to resolve project error.

    Also to the the test results in TFS build do I need to update the following line in RunDnxText.ps1 to use the name of my test project?
    ‘dnx $PSScriptRoot\test\SampleProject.Test test.\nxslt2.exe ‘

    nxslt2.exe is actually located in the solution items folder of the solution.

    Could you please provide a sample project. Appreciate any help.
    Thanks

  7. Is there a way to run individual tests instead of all?

Leave a Reply

Your email address will not be published.

*

© 2017 Fluent Bytes

Theme by Anders NorenUp ↑