Managing New Testing Efforts
You can use the Testing Center in Microsoft Test Manager from Visual Studio 2010 Ultimate or Visual Studio Test Professional to help you plan your testing effort, based on your approach. Microsoft Test Manager is a new application that you use to create a test plan that is associated with your team project. In your test plan, you can define which tests you plan to run for a specific iteration. Also, you can create test configurations that specify the test set up that you want to use to run your tests. By defining what tests you want to run on which test configurations, you can now use the test plan to measure your progress as soon as you begin to run your tests.
You can create test cases for your test plan that can be used for both manual and automated tests. You can add all the information that you need in order to run a test manually by adding test steps with both actions and expected results. In addition, you can share common test steps with other tests to reduce the overhead of maintaining your manual test steps.
When you run these manual tests using Microsoft Test Manager, you can collect details of the actions that you perform both in a log and as a recording that you can use the next time to fast forward your manual testing. You can capture video of your desktop, create a snapshot, and add comments to your test results. You can also gather other diagnostic information when you run a test and save it as part of the test result. You can create bugs as necessary when you run these tests, and automatically add any data that you collect to a bug.
Using Microsoft Visual Studio 2010 you can create different types of automated tests. You can create unit tests to test individual methods in your code, coded UI tests to test your UI interface, generic tests that call API methods, and load tests to check the performance of your application under different levels of stress. You can associate automated tests with your test cases to run these tests as part of your test plan.
After you have run your tests, you can report on your progress. When you use test plans to structure your testing approach, you can answer the following questions:
How many tests have passed or failed?
How many tests still have to be run for this iteration?
When will the testing be complete?
Which areas of the product have high test failure rates?
Which test configurations have high test failure rates?
Who has the most tests left to be run?
Can resources be reallocated to better balance the remaining testing?
Which build should the testers be using based on code changes and bug fixes?
For more information about strategies for testing, see Test Early and Often.
Getting Started With Test Case Management
To test software, you plan your testing strategy first and then you run your tests and submit any bugs that you find. Then you can review your progress and decide whether you may want to rerun tests, verify bug fixes, add more test cases, add test configurations, or decide whether your testing is complete for your current iteration. The following steps will help you start to use Microsoft Test Manager.
Connect to Your Team Project
Plan Your Testing Effort
Run Your Tests
Track Your Progress
Connect
You must first connect to Team Foundation Server and your team project by using Microsoft Test Manager. The team project is the same one that is used to add requirements for your application, maintain source code, and build the application that you want to test. Your testing artifacts are created and managed in this same team project. Your test results are also associated with this team project. When you have connected to this project, you can create a test plan in that project to use for your test planning.
The following illustration shows you how to connect to your team project.
.png "Connect to Your Team Project")
Note
Only team projects that you have permission to access will be displayed. For more information about permissions, see Team Foundation Server Permissions.
For more information, see How to: Connect to a Team Project For the First Time from Microsoft Test Manager.
Plan
You have to define your testing effort using a test plan in Microsoft Test Manager. This plan can be as simple or complex as you require for your project. This test plan lets you select which tests to run and measure the testing progress.
You can specify test configurations to define the software or hardware that you want to use to run your tests as part of your plan. Then you create a test suite hierarchy in the plan. This can just be one test suite that contains all your test cases, or you can use the test suite hierarchy to provide structure to group test cases together. This structure can be grouping based on requirements or user stories in your team project. Finally, you can add manual test cases, with both action and validation steps, or automated tests into a test suite.
This following illustration shows the testing artifacts in your test plan.
.png "Components of a Test Plan")
Use the following topics to help plan your testing effort:
Tasks |
Associated Topics |
|---|---|
Planning the Configuration Matrix for How You Plan to Run Your Tests: You can create test configurations to define the software or hardware that you want to use to run your tests. You can specify default configurations for your plan and which tests you plan to run on which configurations. |
|
Creating Your Plan: You must create a plan for your tests and add the test configurations that you want to use as your default configurations. |
|
Adding Test Suites and Test Cases to Your Plan: You can create test suites to group your test cases together. You can create test suites based on requirements or user stories. You can also create suites by selecting existing test cases or adding new test cases. You can then add manual test steps to these test cases. You can also associate automated tests with your test cases so that you can run them from a plan. |
|
Import Test Suites from Another Test Plan: You can import test suites from an existing test plan if you need the same test suites in another test plan. |
|
Assigning Who Will Run the Tests: You can assign the tests in your test plan to specific testers on your team. By default, the tests are assigned to the owners of the test cases. But you can change this assignment. |
|
Planning the Setup You Need to Use for Your Testing: You can plan what environments and test settings that you might need to run your tests. The environments can be physical or virtual environments. |
Test
When you have created your plan and there is a build of your application to test, you are ready to run your tests. You can select that build as the build you are using for your testing. The combinations of test configurations and test suites that you have created in your test plan are available to be run as shown in the following illustration.
.png "Running Manual Tests in Different Test Suites")
Manual tests are run locally using the Test Runner that lets you record the outcome of each test step and save the results of your test every time that you run it.
You can use Microsoft Test Manager to set up test settings to determine how you run your tests and select what data and diagnostics you collect when you are running your tests. You can collect data and diagnostic information locally, or remotely by using test environments. When you run your tests, you can save this data and diagnostics with your results, and if you need to you can use it to create detailed bugs.
Typically, you would use an environment when you are testing a more complex application. An environment consists of a set of roles. A role specifies the purpose of a computer in the environment. For example, a specific role could be called 'Web Site for Customer Data Store'.
The environment enables you to run tests, gather data or perform system actions on machines for each specific role. A machine can be a physical computer or a virtual machine. For example, you could run your tests on one machine and gather system information about a machine that has the Web server installed for your application. Alternatively, you could run your tests on an environment that uses multiple machines and collect test impact data on those machines, and then you can also perform network emulation on the machine that is running the Web client for your application.
Three examples of scenarios for how you can set your test settings with a test plan to run your tests are shown in the following illustration.
.png "Test Settings Concepts")
Use the following topics to help you run your tests:
Tasks |
Associated Topics |
|---|---|
Setting up test machines to run tests or collect data: You can create test settings to define the roles that you require for your application under test and how to collect data and diagnostics for each role. You can use a physical or virtual environment that contains the roles in your test settings to assign the actual machines that will be used when you run your tests. You require a test controller to create physical and virtual environments. You can only create virtual environments using Visual Studio Lab Management. |
|
Running manual tests from a test plan: You can run manual tests from your test plan using Test Runner to record if each step passes or fails. The test outcome and any data that is collected when you run the test can be saved. |
|
Speeding up manual testing: You can record the UI actions that you take when you run a manual test. When you run the test again, you can use this action recording to fast forward by playing back the action recording to the test step that you need to perform to verify a bug. |
|
Running automated tests: You can run tests directly from Microsoft Visual Studio 2010, from Team Build, or from the command line. You can associate an automated test with a test case from Microsoft Visual Studio 2010 and run that test case as part of your test plan using Microsoft Test Manager so the test results from your automated tests can be tracked with any manual tests. |
|
Analyzing test results: You can analyze the test results for your automated tests based on a specific test run. You can file bugs based on your results. You can also review the code coverage results to check that your tests are really testing as much of your application as possible. |
|
Performing exploratory testing: If you want to run some exploratory tests without specific test steps, you can create a test case with a single exploratory step. Then you can use that as a basis to explore the functionality of your application and keep a record of what you have tested using the action log and video recording. You can also log exploratory bugs and select a specific section of your action log to use in the bug based on elapsed time. |
Track
You can now track your testing effort defined in your test plan. You can check whether there are more builds for your plan, and view the recommended tests to run for these builds based on code changes. You can run standard reports, or run your own customized queries, to track the quality of the application under test.
Use the following topics to help you track your testing effort:
Key Tasks |
Associated Topics |
|---|---|
View reports to help you track your testing progress: you can view reports on your test case readiness and the testing progress for your test plan. |
|
Finding tests to rerun based on code changes: You can compare builds to view which tests are recommended to be rerun, based on changes to your application under test. |
|
Triaging Bugs: You can review your bugs and decide the next steps for the bug. You can also reassign bugs as necessary. |
|
Use standard and custom queries for reporting: You can use existing queries to report on test cases and bugs. You can also create your own custom queries for reporting. |