Fluent Bytes

"The only source of knowledge is experience" - Albert Einstein-

Creating a detailed test report
showing results of test case steps

For a while I have been working on a fairly large implementation of Team Foundation Server. During this implementation we phase out several silo based systems, like rational tools and HPQC. Moving from HPQC to MTM and TFS has been rather easy if you understand that the two systems just have a complete other way of treating test cases and test runs. To Satisfy some end user data needs we needed to create a custom report that is similar to the reports that can be created with HPQC out of the box. This is the test detail report, which I added here below to see what it is about. clip_image002 As you can see it shows not only the test suite outcome, but also the outcome of the test case and the detailed steps and where things failed. Creating a report as above, apeared to be more challanging then I thought at first, but that has to do with the fact that the standard solution we have, don’t have all the data we need. Work item queries, don’t have test case results. The data warehouse contains data to the test case run level, but not the step details. So how can we fix this? Apparently my fellow MVP Tarun Arora (http://geekswithblogs.net/TarunArora) had an excellent post (http://geekswithblogs.net/TarunArora/archive/2011/10/02/mtm-testing-scorecard-using-tfs-api.aspx) that got me out of the gate towards a solution. What he did is just create a simple windows forms application that uses the TFS API to get to all the data. So while less ideal, we can have a solution, but unfortunately not one that integrates out of the box. So I picked up the sample code from Tarun and altered it in such a way that it would give full details on test runs and render them using an embedded report viewer control. The only alternative to get this data is using the TFS object model. In the object model I can get to all the details I need, by using the ITestManagementService. So when you want to get to the test details, you first need to understand how the relationships are set up in the API. So first let start with understanding the basics, and then dive into the solution.

1.1 TFS and MTM ITestManagementService data

To get access to the service we use the standard pattern of connecting to TFS and then requesting the service we need.

Now that I can get to the service, we need to walk the hierarchy. We have a Team Project and a Team project can have multiple test plans. A Plan has the notion of Test Suites and a suite can be either a static suite, a Query based suite or a Requirements Based Suite. Now the tricky part is that a test case is a work item that can be associated with multiple test suites and a test suite can contain multiple test cases. The following diagram shows the relationships.   image Now this model is the logical setup when you look at how you create a test plan. The moment you want to execute a test, we need to add some additional things to the mix. Test runs are a mixture of Test suites that contain test cases and based on the selected configurations we want to run for. So if you start to look at test results, the data structure is a bit more complex. So For our report to succeed, we need to retrieve the following data:

  • For a selected team project ( a dropdown selection in the tool) and a selected Test Plan (Also a drop down selection) we are going to retrieve the test suites.
  • Then for each suite we need to get the test cases and for each test case we need the test results.
  • These test results can be found by using a concept call “Test Points”. A Test Point represents a Test Case to be run against a Test Configuration in a Test Suite.

The results become even more complex, since we can execute a test that contains test cases with parameters (using the @ symbol to denote the parameters in the test steps) and this will result in multiple iterations of the same test steps. The below picture shows the result structure we need to use to get the right data: image So as you can see it involves quite some iterations to get to the final data we want. What I have done is used the solution Tarun build and extended it to support my needs. I had to alter some of the functions that get the test suite hierarchy and the actual function that retrieves the data and creates dataset that can be fed into the report viewer control. To give you an example of the results, I have a couple of screenshots to show you how it looks. You first create a test plan in MTM, like this: clip_image008 You can see I have a test plan with several test cases in there and I ran a set of manual tests so all my tests have some test results associated. Now when you start the report tool, you need to connect to your TFS server image After you connect, you select the team project and the plan you want to view the results from. image When you selected them, click the generate report button, and it will get the data from TFS image Next you can see the detailed report with all test suites, test cases per suite, per test case the iterations and per iteration the test steps and the result of each step. As a bonus, I added also the capability that when you are using parameters in a test case and have multiple iterations on a test run, you get the parameter values that were used in a run as well. Smile clip_image016 I thought it might be useful for others to use as well, so I placed this work on codeplex. Now everyone can contribute. The codebase is not the greatest code in the world, but it gets the job done for me at the moment. Feel free to make changes and tidy things up as you use the tool yourself. I love contributions Smile You can find the project here: https://tfstestreport.codeplex.com/ If you want to join, please let me know and provide me your login name at codeplex when requesting. Hope this helps, Marcel

CTO at Xpirit, Microsoft Regional Director, Visual studio ALM MVP, Speaker, Pluralsight Author and IT Architect Consultant

9 Comments

  1. Hello,
    Great Post! Thank you.
    I am trying to create a report with the result of each step executed in a Test Case and I am able to retrieve the tests steps, expected result, step outcome, error message, attachments. The tests steps and expected result are listed in the correct order as listed in MTM, but the step outcome and error message seem to be re-arranged for all test case that I retrieve. This is my code, please assist.

    Also I would like to list the bug ID that is created for the failed test steps. How can I retrieve that?

    foreach (ITestSuiteEntry testcase in ts.TestCases)
    {
    var testResults = testProject.TestResults.ByTestId(testcase.TestCase.Id);
    foreach (ITestCaseResult result in testResults)
    {
    for (int actionIndex = 0; actionIndex < testcase.TestCase.Actions.Count; actionIndex++)
    {
    resultData = new TestResultData();
    var actionStep = testcase.TestCase.Actions[actionIndex] as ITestStep;
    if (actionStep != null)
    {
    resultData.TestCaseName = result.TestCaseTitle;
    resultData.Step = Regex.Replace(actionStep.Title, @"]+>| “, “”).Trim();
    resultData.ExpectedResult = Regex.Replace(actionStep.ExpectedResult, @”]+>| “, “”).Trim();
    resultData.StepID = count;
    }
    var topIteration = result.Iterations.FirstOrDefault();
    if (topIteration != null && actionIndex < topIteration.Actions.Count)
    {
    var actionResult = topIteration.Actions[actionIndex];
    resultData.StepOutcome = actionResult.Outcome.ToString();
    resultData.Comment = actionResult.ErrorMessage;
    foreach (var attachment in actionResult.Attachments)
    {
    resultData.AttachmentName = attachment.Name;
    resultData.AttachmentUri = attachment.Uri.ToString();
    }
    }
    resultDataList.Add(resultData);
    }
    }
    }

  2. Great article, also I downloaded the code on codeplex. Worked out of the box against TFS 2012. Note: You need to “Connect”, after that your projects and testplans are selectable.

    I used the code, and added another level in the test suites, enabling the user to select a specific testsuite defined in the selected testplan.. In our case some of the testplans are automated coded tests, so it does’t make sense to print these for an exmple.

    The report is valuable because the outcome of each step visible, further more with comments, which in our case, contains bug numbers (bugtracking not in TFS…) and testers comments.

    I wonder why this doesn’t have so much attention. I wish that Microsoft Test Management had a feature like this, but the TFS API is great, and thanks to the author of this article!

  3. Hi, So glad I found this page as the report is excellent! The only issue I am having is that it seems that test cases are only reported on if the steps have been passed or failed. Any tests that are at an Active (non-run) state are not returned. Is this project still being actively worked on, or am I too late to the party?

    Thanks in advance.

    Neil.

    • I am not actively developing this anymore, but there are others that showed interest on extending it. If you would like to get commit rights to the repo, just contact me and I provide you access

  4. Hi
    I went to the download page and I cannot see the program.
    I know its abandoned, BUT someone have the latest version to try it out?

    • If you log into codeplex, you should be able to get the sources. it is in the repo there

      • Hi,

        While using this report application, I am unable to generate any report because we have almost 70000 test cases (However majority of test cases do not have any test steps ) in our test plan and it is running for ever and not getting completed. Can you please help me that how can i implement the code where it can run faster and generate the reports?

        Any Suggestion or Advice will be helpful…

        Thanks,
        DJ

Leave a Reply

Your email address will not be published.

*

© 2017 Fluent Bytes

Theme by Anders NorenUp ↑