Testim Documentation

Welcome to the Testim documentation. You'll find comprehensive guides to help you get started quickly, example code snippets, and tips for being more productive. You will also find details about advanced features that allow you to customize Testim to fit your environment or special testing conditions. Don't worry, we'll support you if you get stuck. Let's jump right in!

Guides    Changelog

Test Status

Add statuses to your tests to help manage which test needs to be worked on

You can add a status to your tests to help manage which test needs to be worked on. The statuses are displayed as one of the columns in the Test Library list and in the Test Editor.

By default all statuses are labeled as "Draft". After the feature is enabled, Testim automatically identifies tests that were run in the last 30 days and labels them as "Active". All other statuses are applied manually, as described below.

By managing the tests' statuses you can enjoy the following benefits:

  • Easily quarantine flaky/failing tests without taking them out of the CI/suite.
  • Easily notice tests that are not connected to the CI (not active).
  • Gradually add tests to the CI without failing the entire run.
  • Improve project visibility by filtering tests by status.

📘

This is a PRO feature

This feature is only open to projects on our professional plan. To learn more about our professional plan, see here.

🚧

Note

In order to use this feature, you'll need to upgrade your CLI version to be at least v3.135.0, read here about CLI installation

Test Statuses

Each test can have one of the following statuses:

Status

Definition

Running as part of the CI/Scheduler

Failing scheduler/CI

Draft

Test is still in-progress

Yes*

Yes

Evaluating

Test ready, but its stability should be validated

Yes

No

Active

Test is ready and stable

Yes

Yes

Quarantine

The test doesn’t conform to the definition of a stable test and is waiting to be fixed

No

No

*The best practice is to add a test to the CI/Scheduler only after the test is ready, and not in Draft status

Viewing the test status

The tests' statuses are displayed in the following screens:
Test Library screen - Test Lists -> Tests

Test Editor

Filtering tests by status

To filter test by status:

  1. Go to Test List --> Tests
  2. Click the Filter button.
  1. In the Filter Test pane, select the checkboxes of the relevant statuses.

Modifying the status of a test

Statuses can be modified manually. All changes to the test status will appear In the revision history.

To change the test status through the Test Library:

  1. Go to Test List --> Tests
  2. In the Status column, select the relevant status.

📘

As part of the test status, we also added an option to view your flaky tests and decide how to manage their status. To read more about the flaky tests, see here.

To change the test status through the Testim's Editor:

  1. Open the test in the editor.
  2. Change the status from the top left corner

📘

When changing the status the test is saved as a revision. If you want to merge it back to Master, you will have to resolve it as a 3 way merge conflict, to read more see here

Using the test status

On test runs

After running the tests, the statuses will be reflected in the following way:

  • Draft tests that run will appear as before. It is recommended to change these tests' status to Active.
  • Evaluating tests will appear in the test runs, but in a case of failure there will be an indication that the failure was ignored.
  • Active tests will appear as before.
  • Quarantine tests will not run (they will not appear in the test runs).

On suite runs

On suite runs, the statuses will be reflected in the following way:

  • Draft tests that run will appear as before. It is recommended to change these tests' status to Active.
  • Evaluating tests will appear in the test runs, but in a case of failure, there will be an indication that they did not fail the CI.
  • Active tests will appear as before.
  • Quarantine tests will not run (they will appear in the suite run with a quarantine indication)

CLI runs

  • Evaluating failed tests will appear in the CLI run summary as FAILED-EVALUATING
  • Evaluating failed tests will be added to the run XML report with a new status "failure-evaluating", to read more about the CLI XML report, see here
  • Quarantine tests will be added to the run XML with a "Skipped" flag.

📘

Quarantine tests can be run in the CLI by using the bypassed “--run-quarantined-tests” flag

Updated 2 months ago

Test Status


Add statuses to your tests to help manage which test needs to be worked on

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.