Coding best practices are becoming the norm. More and more development teams are acquiring habits during their developments such as TDD and even BDD. Despite this meaning having to shift left completely the testing, it’s something that for some teams still take years to digest, so we have to go step by step, and the first one is to enable visibility.
Just having TDD and BDD properly applied, doesn’t mean that we are enabling full test automation in our project. Automation is not just about having my code covered by tests and defining features and scenarios on Gherkin in conjunction with some frameworks such as Cucumber, Specflow or Cinammon triggered by some build jobs. It’s also about automating the results, and enabling traceability and transparency when the release happens.
One thing that is not quite often used in Azure DevOps (formerly VSTS), is the Test Automation traceability.
First, let’s talk about test results and where you can find them…
- Track the report of my unit tests/component tests running on my build.
That can be done from a few places one is the Build status report:
The other one is the Tests Tab:
- Using the new Analytics tab in the Build main page. For that, first, we have to install the Analytics plugin which appears the first time we access to analytics.
Once installed we can have a more granular detail of our test failures
More information here - Every test run, of any kind, is registered on the TEST section of Azure DevOps. For accessing this section you need to have or either a Visual Studio Enterprise subscription associated with your account or a Test Manager Extension, which also offers you much more than just test reports.
I really recommend using Microsoft Test Manager / Test Extensions in conjunction with Azure DevOps to get the full potential of test reports and test automation.
Second, on traceability we are not just linking our test runs to the builds, we want also to go a step further and link our test cases to user stories. This part is easy, right?
We just have to create a test case and link it to our user story. This can be done manually from our workspace or it can be done from Microsoft Test Manager when we create test cases as part of Requirements.
At the end this would look like this inside the user story (displayed as Tested By):
You can find more information here about how to trace test requirements
Third. This leads me to the last part which is my test automation. When you open a test case it usually looks like this:
But this is a test case that can either be created as a manual test or automatically through an exploratory test session (one of the cool features of MTM). The good thing of them is that if they are properly recorded, you can replay them again and again automatically using one feature called fast forwarding.
Our need is to link our coded test automation (MSTest, NUnit, etc) to these test cases, that why Microsoft gave us that “Automation status” tab inside our test cases.
This tab is just telling us if it has an automation test associated with the test case or not.
An easy and quick way to enable this is:
- Go to Visual Studio Test Explorer
- Right-click over your test
- Associate to a test case
Sometimes we don’t need to go through test cases, we just want to set up a test plan and run all the automation against that test plan. With this, we don’t have to create our Test Cases in Azure DevOps, we just need to create a test plan, configure its settings and modify our test tasks in build/release pipelines to run against that plan.
There is a good article from Microsoft that explains the whole process here.
Last but not least, I want to write briefly about Microsoft Test Manager. This tool has been around since the early versions of Visual Studio with Test Edition and with the Premium/Enterprise editions.
Initially, it was meant to be used as a Test Management tool for manual testing and exploratory testing, but it has acquired more capabilities over the years, up to the point that today it’s mostly integrated with Azure DevOps inside the TEST tab.
If you have the MTM client, you can connect to your Azure DevOps project and manage from there your test cases, test environments, manual tests, exploratory test sessions and you can also record not only your sessions but your test steps too through exploratory sessions. With this, you can run most of your manual tests automatically using fast forwarding, which replays all the actions the tester takes.
It is REALLY good for managing test suites and test packs and it has integration with your Builds and Releases, and you can even tag the environments you are using and the software installed on them.
This adds to your Test Capabilities what you need in order to complete your plan.
As a last note, if you are using Coded UI as your main UI Test Automation Framework, it has direct integration with MTM too, so you can associate your Coded UI tests to your Test Cases.
There is also one forgotten feature called Test Impact Analysis, which integrates not only with your builds and releases but also with MTM, which allows you to re-run only the tests that have been impacted by code changes since the last time the code was pushed into the repository, so then we save testing time.
I hope this article shows you the capabilities of Azure DevOps in terms of Automation and Traceability.
References:
Associate automated test with test cases
Run automated tests from test plans
Workarounds: Associate test methods to test cases
Test Impact Analysis in Visual Studio
A Microsoftee in IBM 