For a while I have been working on a fairly large implementation of Team Foundation Server. During this implementation we phase out several silo based systems, like rational tools and HPQC. Moving from HPQC to MTM and TFS has been rather easy if you understand that the two systems just have a complete other way of treating test cases and test runs. To Satisfy some end user data needs we needed to create a custom report that is similar to the reports that can be created with HPQC out of the box. This is the test detail report, which I added here below to see what it is about. As you can see it shows not only the test suite outcome, but also the outcome of the test case and the detailed steps and where things failed. Creating a report as above, apeared to be more challanging then I thought at first, but that has to do with the fact that the standard solution we have, don’t have all the data we need. Work item queries, don’t have test case results. The data warehouse contains data to the test case run level, but not the step details. So how can we fix this? Apparently my fellow MVP Tarun Arora (http://geekswithblogs.net/TarunArora) had an excellent post (http://geekswithblogs.net/TarunArora/archive/2011/10/02/mtm-testing-scorecard-using-tfs-api.aspx) that got me out of the gate towards a solution. What he did is just create a simple windows forms application that uses the TFS API to get to all the data. So while less ideal, we can have a solution, but unfortunately not one that integrates out of the box. So I picked up the sample code from Tarun and altered it in such a way that it would give full details on test runs and render them using an embedded report viewer control. The only alternative to get this data is using the TFS object model. In the object model I can get to all the details I need, by using the ITestManagementService. So when you want to get to the test details, you first need to understand how the relationships are set up in the API. So first let start with understanding the basics, and then dive into the solution.
1.1 TFS and MTM ITestManagementService data
To get access to the service we use the standard pattern of connecting to TFS and then requesting the service we need.
var tfsUri = textBox1.Text;
var tfs = TfsTeamProjectCollectionFactory
var service = tfs.GetService<VersionControlServer>();
// try to save last url as a convienience, don't crash if not possible
if (Properties.Settings.Default.lastTfsUri != tfsUri)
Properties.Settings.Default.lastTfsUri = tfsUri;
/* i know this seems bad, but I don't care if it fails,
it was just an added bonus*/
return service.GetAllTeamProjects(true).Select(i => i.Name).ToList();
Now that I can get to the service, we need to walk the hierarchy. We have a Team Project and a Team project can have multiple test plans. A Plan has the notion of Test Suites and a suite can be either a static suite, a Query based suite or a Requirements Based Suite. Now the tricky part is that a test case is a work item that can be associated with multiple test suites and a test suite can contain multiple test cases. The following diagram shows the relationships. Now this model is the logical setup when you look at how you create a test plan. The moment you want to execute a test, we need to add some additional things to the mix. Test runs are a mixture of Test suites that contain test cases and based on the selected configurations we want to run for. So if you start to look at test results, the data structure is a bit more complex. So For our report to succeed, we need to retrieve the following data:
- For a selected team project ( a dropdown selection in the tool) and a selected Test Plan (Also a drop down selection) we are going to retrieve the test suites.
- Then for each suite we need to get the test cases and for each test case we need the test results.
- These test results can be found by using a concept call “Test Points”. A Test Point represents a Test Case to be run against a Test Configuration in a Test Suite.
The results become even more complex, since we can execute a test that contains test cases with parameters (using the @ symbol to denote the parameters in the test steps) and this will result in multiple iterations of the same test steps. The below picture shows the result structure we need to use to get the right data: So as you can see it involves quite some iterations to get to the final data we want. What I have done is used the solution Tarun build and extended it to support my needs. I had to alter some of the functions that get the test suite hierarchy and the actual function that retrieves the data and creates dataset that can be fed into the report viewer control. To give you an example of the results, I have a couple of screenshots to show you how it looks. You first create a test plan in MTM, like this: You can see I have a test plan with several test cases in there and I ran a set of manual tests so all my tests have some test results associated. Now when you start the report tool, you need to connect to your TFS server After you connect, you select the team project and the plan you want to view the results from. When you selected them, click the generate report button, and it will get the data from TFS Next you can see the detailed report with all test suites, test cases per suite, per test case the iterations and per iteration the test steps and the result of each step. As a bonus, I added also the capability that when you are using parameters in a test case and have multiple iterations on a test run, you get the parameter values that were used in a run as well. I thought it might be useful for others to use as well, so I placed this work on codeplex. Now everyone can contribute. The codebase is not the greatest code in the world, but it gets the job done for me at the moment. Feel free to make changes and tidy things up as you use the tool yourself. I love contributions You can find the project here: https://tfstestreport.codeplex.com/ If you want to join, please let me know and provide me your login name at codeplex when requesting. Hope this helps, Marcel
CTO at Xpirit, Microsoft Regional Director, Visual studio ALM MVP, Speaker, Pluralsight Author and IT Architect Consultant