After you create test definitions in the Test Management Framework, you define test runs. The test runs are collections of test definitions. To test the target application, the test runs are executed.

Concepts: Test Execution Objects

In TMF, performing tests involves three inter-related building blocks. The following diagram illustrates the objects that comprise a test run, their relationship, and their work object ID prefixes.

Test Run

The parent set (work folder) of executed test suites and scenarios. Test runs are categorized by application and version.

Test Suite Execution

A work folder that represents the execution of a test suite during the test run. A test suite execution contains one or more executed scenarios.

Scenario Execution

A work object that represents the execution of a scenario within the test suite. Test scenarios can include and execute more than one segment.

Creating a Test Run

Creating a test run involves specifying which test suites should be executed in that test run and the order in which they should be executed.

To create a test run:

Click New > Test Run. This launches the guided entry process in which you create the test run. An object ID prefixed by RUN- is assigned to the test run and displayed in the work tab at the top of the workspace.

* indicates a required field

1. Enter the run detail.
   • Enter a name for the test run. The default value uses the RUN- prefix followed by the current date and time.
   • Select the test system and application and version in that system the test run applies to. The values default from your current in-context testing application.
   • Optionally enter a description for the run.
   • Check the Send Email Notification To box if you want to receive email notification when this test run is completed. This field appears if you have a valid operator email address.
2. Click Next>> to move to the suite selection screen.

This screen displays a list of test suites defined in TMF for the application. Click >> to expand a row to display a description of the suite.

1. Select the test suites you want to include in the test run.
2. Use the Generate Scenario Executionsdrop-down list to choose whether to generate the run now or at a scheduled time:

   Now

   Select to have TMF run the tests upon your completion of the guided entry process on this test run. As the tests are run, the display shows a tree grid of the test suites and their underlying scenarios and segments and the progress of their executions.

   If you select Now, you can optionally select to run the tests in the background. If you make this selection, TMF runs the tests in the background and displays a confirmation message.

   At Scheduled Time

   Select to have TMF run the tests at a later time. The Scheduled Test Time field appears so you can specify the time. When you complete the guided entry process, TMF routes the test run to a workbasket and holds it pending the scheduled execution time.

3. If you select more than one suite and want to set their order of execution, you can check the Order for Execution box. After clicking Next >>, a screen displays where you can set the order of the test suites.
4. Click Next >> to confirm your entry.

If you selected to order the test suites for execution, when you click Next > >, a screen displays where you can specify the order. Otherwise, TMF runs the test suites (if you selected Now) or saves the test run for future execution (if you selected At Scheduled Time).

Running Tests and Managing Test Runs

Test runs are managed from the Test Executions workspace of the test portal where you can:

• Initiate a test run
• Select test runs from a worklist of test executions that are in progress for your in-context application and version
• Display a snapshot of the number of test runs and scenario executions and their status for the in-context application and version

When you complete defining a test run, TMF builds a test package. If the test run is specified to run now, the test package is built immediately; if the run is specified to run at a scheduled time, the test packages is built at that scheduled time.

To create the test package, TMF generates a suite execution object (prefix ESTE-) for every test suite included in the test run and generates a scenario execution object (prefix ESCN-) for every test scenario included in the included test suites.

Resolved test runs

Test runs resolve automatically when all of the scenarios included in the run are complete. If any executed scenario object (ESCN-) is still open, the test run is not resolved.

For test runs that involve automated unit testing, if the following items are true:

• The test run contains test suites and scenarios solely based on automated unit test suites and test cases
• The segments based on those automated unit test cases are set up to automatically execute
• The automated unit test case does not require manual intervention (for example, because the test case was blocked)

Then the scenarios can complete and the test run automatically goes to Resolved status after it is initiated.

To see the results of completed runs, click on a value in the Resolved row of the Test Executions Inventory on the Test Executions workspace and then select a run, or use the reports in the Test Execution Reports section of the Reports workspace. (See TMF - Reporting test progress and results).

In progress test runs

Test runs in progress are routed to workbaskets. After defining a test run using the Now option, a review screen displays where you can see the generation of the underlying scenario work objects.

After the generation process is complete, a Run link displays for those scenarios that have items that must be tested manually to complete the scenario.

If you scheduled your test run for a later time, you can access it by clicking on a value in the In Progress row of the Test Executions Inventory on the Test Executions workspace and then select the run. (See TMF - Reporting test progress and results).

To select an open test run and complete the pending scenarios:

1. Go to the Test Executions view in the test portal.
2. In the Scenario Executions column of the Test Executions Inventory, click the value in the In Progress row. A list of test scenarios available to run displays.

1. Expand the row for the test run and select the scenario execution that you want to complete. Summary information about the scenario appears in a section below the list.

   To review details about this scenario, click Review. To view, add, or edit any attachments on this scenario, click Attachments.

6. To perform the test, click link in the In Flowfield:
   • If the scenario has an automated unit test case that is not set up for automatic execution or that requires manual intervention because the test case was blocked, the link is labeled Execute Auto Test
   • If it is a manual test, it is labeled Execute Test.

Running a Manual Test

If the scenario execution is a manual test (for example, with segments of type Manual Use Case or Freeform), this form displays:

1. Click to open a session in the test environment.
2. Run the test. The input script and expected outcome information on the form provide online reference material for the tester.
3. Log the result on the Record Test Resultstab:
   • Choose a result in the Results field. Options are Passed, Failed, or Blocked.
   • Enter a Case ID if one has been created by the test.
   • Enter any notes about the test that you want to include.
4. If you want to take screen captures and attach them to the test results or attach any additional information about the test that you performed, use the Add Attachments tab.
5. Click Submit to complete this test.

From this point, you can:

• Test the next available test segment in the scenario execution (using Run Next Test).
• Review the test suite (using Review test suite).
• Close the form and return to the test portal (using Close).

Running an Automated Unit Test Case

If the scenario execution involves an automated unit test case that is either not set up to be automatically run or that needs manual intervention, this form displays:

1. Click to run the automated unit test case. When the test is run, the SOAP interface to the test environment automatically returns the results in a separate pop-up window.

   The window displays the test results and the Case ID created by the test. In this window, you can click Update Scenario with Results to populate the test results form with the result, Case ID, and differences found in the run. If a test fails, the View Differences button is available to use to see a list of differences found in the execution of the automated unit test.

2. Enter test notes in the Record Test Results tab.
3. If you want to take screen captures and attach them to the test results or attach any additional information about the test that you performed, use the Add Attachments tab.
4. Click Submit to resolve this scenario.

From this point, you can:

• Test the next available test segment in the scenario execution (using Run Next Test).
• Review the test suite (using Review test suite).
• Close the form and return to the test portal(using Close).

Logging Bugs

When a test fails, the test results tab displays two fields in the Qualification section.

You can indicate whether to have:

• A bug item created
• The test scenario definition updated

If you specify to have a bug item created and you click Submit, one of two sections appears:

• If you have not specified a Bug Tracking system for the testing application and version in the environment record, a form displays to enter a bug item reference. Although the framework can record that a bug has been logged through its reference ID, the actual bug report and detail must be entered and worked in the external tracking system.

• If you have specified a Bug Tracking environment for the testing application and version, a form displays for you to enter information about the bug to be created in that Bug Tracking system. Submitting this screen displays a confirmation screen with the Bug ID that was entered for the issue.

Additional information

• Defining scenarios and test suites for testing applications

Back to About the Test Management Framework