Chapter 2. Introduction

Table of Contents

2.1. What is Klaros-Test­management
2.2. Test Cases - The Basis of Testing
2.3. Test Segments - Reusable Test Case Steps
2.4. Test Suites - Organizing Test Cases
2.5. Test Runs - Executed Tests with their Results
2.6. Test Environment and System under Test - Influences of Test Results
2.7. Iterations - Subdividing your project into phases
2.8. Requirements – Meeting Quality Standards
2.9. Jobs - Planning the test process
2.10. User Roles - Every User has his Rights
2.11. Overview of Artifacts
2.12. Issue Management Systems

2.1. What is Klaros-Test­management

Klaros-Test­management is a modern web application which helps you to organize and manage your test process. It documents all test relevant data by storing it in a database and gives an overview about the test progress of a project or piece of software at any time. A single item of test relevant data will be called Artifact throughout this document.

The main artifact is called Project. It contains all the other artifacts that are needed to define, execute and evaluate tests. For more details on defining projects see Section 6.1.

2.2. Test Cases - The Basis of Testing

Test Case Structure

Figure 2.1. Test Case Structure

The central artifact of a test project is the Test Case which represents a single test. A test case can be executed in an automated or manual fashion. An automated test case refers to a piece of software code that gets executed and in return delivers a test result, usually in form of a test result file. A manual test case contains beside other entries one or more Test Case Steps which hold detailed written instructions and test conditions for the testers executing the test case.

Test Case Step

Figure 2.2. Test Case Step

A test case step consists of a Precondition (the state of the System under Test before the test starts), an Action (explains what the user has to do/test/try out). After the action the outcome should be the Expected Result, and a Postcondition (the state of the system under test after the action). Klaros-Test­management guides the user through the test case step by step. During the test execution every step will be assigned a Test Case Step Result which contains the verdict of the test case step. Passed, Failed, Error or Skipped are the possible verdicts which are available to the user.

If the precondition, postcondition or expected result does not match the description, the User should choose Failure. If an unexpected error in the system prevents the user from carrying out the step (for example, a button is missing) they should choose Error. If one of the test case step result verdicts is marked as Error or Failure, the whole test case result is also marked with that particular verdict. Only if every test case step is marked as passed, the encapsulating test case result is marked as passed.

Automated test cases usually have no test case steps because they are executed with an automated test program (e.g. JUnit). These test case results can be imported and stored with the other test related data into the database.

Requirements are another type of artifact in Klaros-Test­management which represent a condition or capability of the system under test. Each test case can relate to one or more requirements and vice-versa. More information regarding requirements is shown in ( Section 2.8).

2.3. Test Segments - Reusable Test Case Steps

Test Segment Structure

Figure 2.3. Test Segment Structure

Test Segments are grouped, predefined and encapsulated Test Case Steps that can be created separately and reused in Test Cases .

When defining test cases, specific test case steps can be exactly the same in multiple test cases. To make sure these test case step sequences are identical in each test case, it is possible to encapsulate test case steps into test segments. These are inserted into a test case like normal test case steps and can be reused in different test cases within the same project.

They have the same attributes like normal test case steps mentioned above. The only difference is, that they can only be used as is and not be further edited when referenced in test cases.

2.4. Test Suites - Organizing Test Cases

A Test Suite

Figure 2.4. A Test Suite

Multiple test cases can be combined into another artifact that is called Test Suite. When executing a test suite, all test cases that are part of this test suite are executed in a row by the tester.

The same test cases can be part of multiple test suites and changes to a test case will automatically be reflected in all test suites referencing it.

2.5. Test Runs - Executed Tests with their Results

Test Run with Test Results

Figure 2.5. Test Run with Test Results

Every time a tester executes test cases or test suites, a Test Run containing the corresponding Test Case Results is created. The test run consists of all test case results created during the execution of a test case or test suite and stores additional information about the circumstances of the test run shown in the next section.

The simplest case is the execution of a single test case. This generates a test run with a single result (the test case result). If a test suite is executed, there will usually be more than one test result (one result for each test case). The test suite result lists all the test results of the executed test suite.

Test runs can be paused and continued at a later time. This means that it is possible that not all results that are stored in the database already have a known verdict, in this case the verdict is listed as Unknown.

2.6. Test Environment and System under Test - Influences of Test Results

Whenever a test case or test suite is executed, it needs to be executed in a defined Test Environment, and one executes it against a defined System under Test.

The result of a test is influenced by the version of the system being tested (system under test / SUT), since issues are detected in a particular software version, but may then be fixed in a later version.

Test environments represent external conditions which may have impact on the test result, like e.g. the operating system the system under test in running in. To document this properly, every test run is linked to a particular system under test and test environment.

2.7. Iterations - Subdividing your project into phases

An Iteration groups systems under test, test environments and requirements within a project for a selectable time frame. Every iteration represents a selected phase of the project and its artifacts including all jobs and test results executed during this phase. Grouping the artifacts like this makes it easier to identify different test cycles in a project. This allows for better integration of the test process especially with agile software development practices.

2.8. Requirements – Meeting Quality Standards

To confirm if the system under test meets its quality standards it is needed to document which of the system requirements are covered by tests ( Coverage) and how many of them have been executed with a passed result ( Compliance).

To enable this traceability, it is needed to link requirements to one or more test cases and vice versa.

From this set of information Klaros-Test­management is able to automatically determine coverage and compliance information for the whole project or selected artifacts once the first test results are available.

2.9. Jobs - Planning the test process

Many users may execute tests in Klaros-Test­management simultaneously. To orchestrate this process it is recommended to plan the testing activities of the users. The planned execution of a test case or a test suite by a user is defined in the form of a Job (which is also an artifact). Jobs usually reference a test case or test suite. The intended test environment and system under test may be defined beforehand by the creator of a job. Jobs also contain a start date (when the testing should begin), a due date (a deadline for the execution of the test), and may be assigned a priority (how urgent the test execution is). Available priorities are: Trivial, Minor, Major, Critical and Blocker. Jobs may also contain status information like Resolved, Reopened, Closed, In Progress or New to document their progress.

After a job has been created it can be assigned to a user. This user then can execute the job and set its results. A job may be repeated and executed multiple times, so one job can refer to more than one test run.


Figure 2.6. Jobs

In addition to test execution other job types exists allowing to schedule reviews of test cases or requirements and specify arbitrary tasks in a textual form.

Jobs can be nested and individually assigned to users. Therefore it is possible to track progress of larger, distributed test activities.

2.10. User Roles - Every User has his Rights

Klaros-Test­management defines four roles available to users: Administrator, Manager, Tester and Guest. Only a user with the Administrator role is able to create and assign other users.

Guests are only able to view artifacts and reports, they are not able to change anything in the system.

Testers have more permissions than those in the role of guest. They are not only able to view artifacts but in addition they are allowed to execute jobs, test cases and test suites and edit test results.

Managers inherit all rights of a tester and also possess additional rights to create and update artifacts like test cases, requirements and more.

Administrators have all available rights in the system. In addition to the rights a manager has they are able to create and delete projects and users as well as configuring system-wide parameters.

Each users has a defined default role which is assigned by an administrator when creating a user account. The Manager, Tester and Guest roles can also be assigned to users on a per-project basis providing a project based security scheme.

Detailed information about the role permissions can be found in Appendix A).

2.11. Overview of Artifacts

Artifacts Overview

Figure 2.7. Artifacts Overview

2.12. Issue Management Systems

Klaros-Test­management can be integrated with issue tracking systems like Atlassian Jira, Bugzilla and many more (see Section 12.5). If an error of the system under test is detected in one of the test case steps, the user can easily create an entry in the issue tracking system from within Klaros-Test­management which is automatically linked and tracked inside both systems.