Close search
Hoa

Hack book of Hoa\Test

Ensuring software quality is not easy. The Hoa\Test library provides several tools to write and to execute tests, to generate test data or tests themselves etc. This is the basis to test all Hoa's libraries. Research papers have been published and implemented inside Hoa\Test and related libraries (like Hoa\Praspel, Hoa\Realdom etc.).

Table of contents

  1. Introduction
    1. Nature of tests
    2. Test framework
    3. Research papers
  2. Writing tests
    1. Automated unit tests
      1. Automatic test data generation
    2. Automatically generated tests
  3. Run tests
    1. Interpreting the test result
    2. Debugging mode
    3. Select tests to run
    4. Choose the PHP virtual machine
    5. Test execution engine
  4. Virtual file system
    1. Files or directories
    2. Permissions
    3. Access-, change- and modify-time
  5. Conclusion

Introduction

Software quality is a vast topic in Computer Science. There are many strategies, a lot of metrics… and a lot of confusion. One way to ensure quality in Hoa is based on the testing strategy. Let's introduce this chapter by some definitions and by presenting the tools provided by the Hoa\Test library.

Nature of tests

Informally, a test is executed on a System Under Test, abbreviated SUT, and is composed of two parts:

  1. Test data to execute the SUT, and
  2. An oracle, establishing the test verdict: Are the result of the execution and the state of the SUT after its execution the ones we expect or not?

The values of the verdict are: Success, fail or inconclusive. “Success” means that the result is what we expected, “fail” means that the result was not as expected and inconclusive means that it was not possible to determine if the result was either a success or failure. The important part is: “what we expect”. By extension, we understand a test does not check the SUT is verified.

Testing shows the presence, not the absence of bugs.

A test only checks the SUT is valid. The difference is that a valid system has no bug for some non exhaustive executions, but a verified system does what it is designed to do. Despite this, tests do not prove the absence of bugs, they are useful (easy to write, easy to read, not all system can be proven etc.). Nevertheless, depending on the chosen test data, tests can be close to proofs. As an example, a test with exhaustive test data can be considered as a proof.

A test can be written manually, for instance by a test engineer. A SUT is provided, the engineer uses test data to execute it and the test verdict is established by the engineer. In the case of an automated test, the execution of the SUT and the test verdict are computed by the machine. This is the role of the “xUnit” frameworks, such as atoum or PHPUnit. The engineer will use the frameworks to write tests and the machine will execute them. Finally, automatic tests are generated and executed by the machine. The machine may rely on a specification, like a contract, to automatically generate the tests (please, see the Hoa\Praspel library or the research papers below that we have published about it to learn more about the Design-by-Contract and Contract-based Testing paradigms). Automatic tests have two sub-categories: Online and offline tests. Online means the tests are generated and executed in one-pass, while offline means the tests are generated and then executed later (with the test data or not).

In Hoa, we have both automated and offline automatic tests.

This is a high classification of tests. A thinner one is described by the following schema explaining what kind of tests to write:

Dimensions of the test universe is represented by 3 axis.

The “Conception support” axis describes the visibility we have on the System Under Test: Either black box or white box. Their meanings are different according to the context but we can consider a system as a white box if we look at and use what is inside the box, else it is a black box. For instance, with unit tests, let's say with a function as the SUT, black box testing will consider only the inputs and the outputs of the function without considering its body (even if it reads or writes global variables, streams etc.). On the other hand, white box testing will consider the body of the function and the testing strategy will be totally different. What we expect from a black box or a white box system is not the same.

The “System size" axis describes at what level (or altitude) the system will be tested: From the lowest one, unit (a function or a method) to the highest one, the whole system, which includes components (a set of functions) and the integration of these components inside other ones, which together form the system. All levels are important: While unit tests ensure that a function does what it is supposed to, they do not ensure that a function integrates correctly with others in the system.

Finally, the “Type of test” axis describes the goal of the tests: Either functional (“everything works as expected and that's it”), robustness (“it is not possible to put the system into an error state if we do something unexpected”), performance (“the system supports heavy loads and behaves as expected under these conditions”) etc.

In Hoa, we have white box and black box, unit and functional tests. We also include performance tests in some libraries.

Remember that each point in this test universe may imply different tools and different development practices. When talking about tests, it is important to have this schema in mind and to consider where our problematic lies.

Test, test case and test suite

Formally, a test of a SUT is a set of data (called test data) that fixes the values of the arguments (also called inputs) of the SUT. A test case is a pair composed of a state (the context in which the test is executed) and a test. A test suite is a set of test cases. Therefore, there are two states from the test point of view: A pre-state —before the execution of the SUT— and a post-state —after the execution—.

Building the pre-state is something crucial: The goal is to put the SUT into a specific (pre-)state in order to test it. The part of the test case responsible for creating the pre-state is called the preamble. Thus, the preambler is code that puts the SUT into a specific state.

So, when writing a test case, we will have 3 items to consider:

  1. The preamble, to put the SUT into a specific state,
  2. The execution, to run the SUT and get a result,
  3. The oracle, to compute the test verdict (based on the execution result and the post-state).

The oracle consists of a sequence of assertions: “Is this value equal to…?”, “Is this value changed this way…?” etc. It is supposed to be able to check the form of some data as much as comparing other data between the pre-state and the post-state.

In a test case, we represent and introduce the preamble, the execution and the oracle respectively by the given, when and then keywords. This will be detailed hereinafter.

Test framework

Hoa did not develop its own test framework for several reasons. The main one is that this is a laborious task. It implies more development, more tools to maintain, less people focused on Hoa etc. Instead, we chose to rely on a good project and contribute to it. We chose atoum, “a simple, modern and intuitive test framework”.

Thus, Hoa\Test is the basis used to test all of Hoa's libraries. It wraps atoum and it aims at being a bridge between Hoa's features and atoum's features. It also adds some features when needed. This is more than an abstract layer on top of atoum. Hoa\Test also provides its own command-line to run the tests with everything pre-packaged. Consequently, Hoa\Test provides a ready-to-use test environment targeting Hoa.

One shall not be surprised to see some contributors of Hoa as contributors of atoum. The two communities are strongly linked. Hoa also provides official contributions to atoum like the atoum/praspel-extension extension and some other extensions are based on Hoa, like the atoum/ruler-extension extension which is based on the Hoa\Ruler library, also used by Hoa\Test. Even some atoum's extensions are tested with atoum and Hoa (with atoum/praspel-extension). The loop is complete.

Research papers

Several research papers (articles, journal and a PhD thesis) have been published about Hoa\Test, Hoa\Praspel and Hoa\Realdom:

These papers are about test data generation and validation, including Design-by-Contract and Contract-based Testing in PHP, along with Grammar-based Testing and Solver-based Testing.

Writing tests

Each library provides a Test/ directory at the root of the repository. This is where tests are located. Automated unit tests can be found in the Test/Unit/ directory while automatically generated unit tests can be found in the Test/Praspel/Unit/ directory (not by default, they must be generated first).

Automated unit tests

In Hoa, a test suite is represented by a file, containing a class. A test case is a method in this class, expressing the preamble, the execution of the SUT and the oracle. The template of a test suite is then the following (let's say for the Test/Unit/Bar.php test suite of the Hoa\Foo library):

namespace Hoa\Foo\Test\Unit;

use Hoa\Foo\Bar as SUT;
use Hoa\Test;

class Bar extends Test\Unit\Suite
{
    public function case_classic()
    {
        // test case.
    }
}

We define an alias SUT to the current System Under Test. In a unit test context, this is a method or a class. We can also declare the CUT alias, standing for Class Under Test, and LUT for Library Under Test. LUT is useful if we need to access to other classes in the same library. Because they are unit tests, we should not use other classes but remember that a test case is composed of a preamble and it needs to be build. In this case, LUT could be defined to Hoa\Foo.

The Bar class extends the Hoa\Test\Unit\Suite class, which defines a unit test suite. It offers the complete test API needed to write tests.

Test cases are public methods. They do not use a camel case notation but an underscore lower-cased notation. They must be prefixed by case_. The template of a test case must follow the preamble, execution and oracle principle, respectively represented by the given, when and then keywords, and thus must be written as follows:

public function case_classic()
{
    $this
        ->given(
            …,
            …,
            …
        )
        ->when(
            …,
            $result = …
        )
        ->then
            ->…;
}

The given and when “control flow structures” are function calls with an unbounded arity. They are void methods, they do nothing. The goal is not to pass something to them but to declare variables, to establish the preamble and to execute the test. For instance:

public function case_sum()
{
    $this
        ->given(
            $x = 1,
            $y = 2
        )
        ->when(
            $result = $x + $y
        )
        ->then
            ->integer($result)
                ->isEqualTo(3);
}

This is strictly equivalent to the following:

public function case_sum()
{
    // Given:
    $x = 1;
    $y = 2;
    // When:
    $result = $x + $y;
    // Then:
    $this->integer($result)->isEqualTo(3);
}

We use, however, the first form. It is clearer: Indentation, special keywords (through method names) and it forces us to clearly separate these 3 parts: Preamble, test execution and oracle.

The $result variable is special. It must always hold the result of the test execution, i.e. the result of the SUT's return statement. This is a convention.

One test case can contain multiple test executions with multiple oracles (for instance if a SUT returns different results according to the time). Then, the test case must be written as follows:

public function case_classic()
{
    $this
        ->given(…)
        ->when($result = …)
        ->then
            ->…

        ->when($result = …)
        ->then
            ->…

        ->when($result = …)
        ->then
            ->…;
}

Assertions are always used after the then part and most of the time has the form assertion-group->assertion. We can link assertion groups too. For instance:

public function case_sum()
{
    $this
        ->given(
            $x = 1,
            $y = 2
        )
        ->when(
            $result = $x + $y
        )
        ->then
            ->integer($result)
                ->isLowerThan(4)
                ->isGreaterThan(2)
            ->string((string) $result)
                ->isNotEmpty();
}

The list of all assertions can be find on atoum's documentation. Even if it is pretty natural to write assertions because it is close to “human conventions”, sometimes it is useful to discover that a specific assertion already exists, like float->isNearlyEqualTo which compares two floats as expected most of the time. So feel free to check atoum's documentation often!

Automatic test data generation

Hoa\Test includes the atoum/praspel-extension extension. This extension includes the Hoa\Praspel library and the Hoa\Realdom library inside atoum. When doing manual or automated tests, this extension can be used to automatically generate test data.

All we need is to describe a realistic domain (see the Hoa\Realdom library) with the realdom assertion group and then use the sample “assertion” (internally this is a handler but the syntax is the same) to generate one value or sampleMany to generate many values. For instance, to automatically generate an integer in the interval 7 to 13 or 42 to 153:

public function case_sum()
{
    $this
        ->given(
            $_x =
                $this
                    ->realdom
                        ->boundinteger(7, 13)
                        ->or
                        ->boundinteger(42, 153),
            $x = $this->sample($_x)
        )
        ->when(…)
        ->then
          ->…;
}

Actually, atoum/praspel-extension provides 3 asserters to generate data and 1 asserter to validate data:

  1. realdom to create a a realistic domain disjunction (see the or in the previous example),
  2. sample to generate one datum from a realistic domains disjunction,
  3. sampleMany to generate several data,
  4. predicate to validate a datum against a realistic domain disjunction.

Hoa\Realdom provides a standard realistic domain collection, which includes very useful ones, like regex that describes a regular expression (in the PCRE format). Thus, we are able to automatically generate a datum against or validate data matching regular expressions. The following example generate one string similar to an email address:

$this
    ->given(
        $email = $this->sample(
            $this->realdom->regex('/[\w\-_]+(\.[\w\-\_]+)*@\w\.(net|org)/')
        ),
        …
    )
    ->…

The following example declare a realistic domain representing a date with the d/m H:i format, between yesterday and the next Monday:

$this
    ->given(
        $_date = $this->realdom->date(
            'd/m H:i',
            $this->realdom->boundinteger(
                $this->realdom->timestamp('yesterday'),
                $this->realdom->timestamp('next Monday')
            )
        )
    )
    ->…

Then, to generate one date:

$this->sample($_date)

Or to sample 42 dates:

$this->sampleMany($_date, 42)

Similarly, booleans, arrays, classes, colors, strings based on grammars etc. can be generated… but also validated! The predicate asserter is a real one, contrary to realdom, sample and sampleMany. By nature, it computes a boolean: Either true or false. It can be used on a realistic domain disjunction, for instance:

$this
    ->given(
        $_date = $this->realdom->date(
            'd/m H:i',
            $this->realdom->boundinteger(
                $this->realdom->timestamp('yesterday'),
                $this->realdom->timestamp('next Monday')
            )
        ),
        $stuff = …
    )
    ->when($result = SUT::compute($stuff)
    ->then
        ->predicate($_date, $result);

In the above example, the _date realistic domain is described, not to generate a datum but to validate the result from the SUT.

To get more information about Grammar-based Testing, please, see the Hoa\Compiler library. To get more information about Solver-based Testing, please, see the the Hoa\Realdom library. In all cases, please, see our research papers.

Automatically generated tests

This section has been addressed by a PhD thesis and several research papers. However, tools are not stable yet and documentation is under-writing. Please, refer to the Hoa\Praspel library to get more information.

Run tests

Most of the time, we will run tests from inside a library repository. Composer will be required to install dependencies of the library, which includes Hoa\Test as a development dependency. Thus, from the root of the library repository:

$ composer install

By default it will install development dependencies. If not, force it by adding the --dev option.

A vendor/ directory is then created. Inside this directory, we will find the bin/hoa command (please, see the Hoa\Cli library). Executing this command without any option will list all available sub-commands for this current installation. We should see the test commands. To execute tests, we will use the test:run sub-command. We must provide the directory where tests are stored, which is always Test/. Thus:

$ vendor/bin/hoa test:run --directories Test/

Note that the --directories option has a plural form. This is because we can specify more than one directory name by using a comma.

At this point, we will see test executing themselves.

Interpreting the test result

The CLI test reporter will use colours to indicate the test verdict: green for success, red for fail or inconclusive. In case of a success, we should see:

Success (S test suites, c/C test cases, V void test cases, K skipped test cases, A assertions)!

where:

In case of a failure, we should see:

Failure (S test suites, c/C test cases, V void test cases, K skipped test cases, …, F failure, E error, X exception)!
> There is F failure:
…

For each failure description, a “diff” will be computed, i.e. a textual differential representation of what we expect and what we got.

Debugging mode

In some cases it is possible that debugging information are accessible but hidden for a regular running. To enable the outputs, we need to use the --debug option; thus:

$ vendor/bin/hoa test:run --directories Test/ --debug

The order of the option does not matter, as usual with the hoa command.

To produce debugging information, we should use the dump handler while writing the test case. For instance:

public function case_sum()
{
    $this
        ->given(
            $x = 1,
            $y = 2
        )
        ->when(
            $result = $x + $y
        )
        ->then
            ->dump($result)
            ->integer($result)
                ->isEqualTo(3);
}

Select tests to run

Selecting tests to run enables a faster feedback and a shorter “test loop”. We previously used the --directories option to pick some directories: Either the test root directory Test/ or one or many sub-directories (with --directories Test/Foo/,Test/Bar/,Test/Baz/Qux/ for instance).

We also have the --files option to select one or many specific files to run. For instance:

$ vendor/bin/hoa test:run --files Test/Unit/Foo.php,Test/Unit/Bar.php

Finally, Hoa\Test requires the atoum/ruler-extension extension to be installed. This extension allows to precisely filter tests to run. We access to it with the --filter option followed by an expression. Fun fact: Expressions are based on the Hoa\Ruler library!

The following variables are available in a filter expression:

All standard operators from the Hoa\Ruler library can be used. So for instance, to only run the Hoa\Foo\Test\Unit\Bar test suite, which is represented by a class, we will write:

$ vendor/bin/hoa test:run --directories Test/ --filter 'class = "Hoa\Foo\Test\Unit\Bar"'

Or, to run only two specific test cases, let's say case_sum and case_baz:

$ vendor/bin/hoa test:run --directories Test/ --filter 'method = "case_sum" or method = "case_baz"'

The previous example is strictly equivalent to:

$ vendor/bin/hoa test:run --directories Test/ --filter 'method in ["case_sum", "case_baz"]'

Another example to filter by test suites' name: Only the ones that end by Qux:

$ vendor/bin/hoa test:run --directories Test/ --filter 'class matches "Qux$"'

The matches operator requires a regular expression to the PCRE format.

Finally, test suites and test cases can hold one or more tags, thanks to the @tags annotation. An annotation is a comment of kind /** */ located on top of a class, interface, method, function etc. Tags allow to have a transversal classification of test cases. For instance:

/**
 * @tags featureA featureB
 */
public function case_sum()
{
    // …
}

To run only test cases with the tag featureA, we will use the following command-line:

$ vendor/bin/hoa test:run --directories Test/ --filter '"featureA" in tags'

Here, the variable tags contains an array of strings, representing tag names.

Choose the PHP virtual machine

When testing, it is also useful to be able to select one specific PHP virtual machine, like a specific version of PHP (the default virtual machine) or the latest version of HHVM for instance. There exist several PHP virtual machines nowadays and we cannot ignore them.

We select a virtual machine by using the --php-binary option and by providing a path to the virtual CLI binary; thus, for instance:

$ vendor/bin/hoa test:run --directories … --php-binary /usr/bin/php

We can imagine that the --php-binary option value is determined by a global variable. In this case we could re-use the same script to execute tests against several PHP virtual machines, like:

$ PHP_BIN=/usr/bin/php vendor/bin/hoa test:run --directories … --php-binary $PHP_BIN

Please, remember to use PHP syntax and features described in the PHP specification as much as possible.

Test execution engine

atoum provides several test execution engines, such as:

When running tests, isolation is really important: No memory conflicts, no execution conflicts, the state of the System Under Test is reset each time etc. The test verdict does not depend of a previous run and is therefore deterministic and unambiguous (in as far as the SUT is).

By default, Hoa\Test will use the concurrent test execution engine. All test cases are not executed at the same time; in fact the number of test cases to run is defined by the number of processes to use. By default, this is either 2 or 4 depending of the platform but we can specify it by using the --concurrent-processes option. Thus, to force the use of 1024 processes when running the tests (assuming that we have a monster computer):

$ vendor/bin/hoa test:run --directories Test/ --concurrent-processes 1024

Because the inline test execution engine is not relevant for Hoa's usecases and contexts and because the isolate test execution engine has more cons than pros compared to the concurrent one (mainly the latter is faster than the former), we cannot change the test execution engine. However, to emulate the isolate test engine, we could force --concurrent-processes to 1.

Virtual file system

The Hoa\Test library provides a virtual file system over its hoa:// protocol to ease testing of files or directories.

The root of the virtual file system is hoa://Test/Vfs/. Everything added after this root will be a path to a virtual file or a directory. Additional query strings can be present to specify more information, such as permissions, access time etc. The parent-child relation is automatically created, it is not necessary to specify that a file is a child of a directory, this is deduced from the path.

Note: The virtual file system is only accessible from inside a test case.

Files or directories

A virtual file Foo can be represented by the following path: hoa://Test/Vfs/Foo?type=file. In this case, either we open it with regular stream functions like fopen, fread, fwrite, file_put_content etc. or, with the Hoa\File library, thus:

$file = new Hoa\File\ReadWrite('hoa://Test/Vfs/Foo?type=file');
$file->writeAll('Hello world!');

var_dump($file->readAll());

/**
 * Will output:
 *     string(12) "Hello world!"
 */

To create a file or a directory, we must use the type query string. Using type=file creates a file, while type=directory creates a directory; thus:

$directory = new Hoa\File\Directory('hoa://Test/Vfs/Foo?type=directory');

This is due to the fact that we cannot deduce stream type based on its name. This query string is only necessary when creating the file.

For the rest, it works like any regular files or directories, nothing different.

Permissions

To change the permissions, either the specific stream permission functions can be used or we can set them directly by using the permissions query string followed by an octal value; thus:

$file = new Hoa\File\ReadWrite('hoa://Test/Vfs/Foo?type=file&permissions=0644');
var_dump($file->getReadablePermissions());

/**
 * Will output:
 *     string(10) "-rw-r--r--"
 */

As any query strings, we can concatenate key-values by using the & symbol. For instance, type=directory&permissions=0644 represents a directory with the 0644 permissions.

Access-, change- and modify-time

This can also be useful to define access-, change- and modify-time by respectively using the atime, ctime and mtime query strings. The expected values are an integer representing a timestamp (a number of seconds). Thus, to represent a file that has been accessed in the future (in 1 minute):

$aTime = time() + 60;
$file  = new Hoa\File\ReadWrite('hoa://Test/Vfs/Foo?type=file&atime=' . $aTime);

Conclusion

The Hoa\Test library is a set of tools to make white box or black box, unit or functional, automated and automatic unit tests. It is more than a wrapper around atoum: It provides a test structure and conventions, a virtual file system and extensions to automatically generate test data, automatically generate tests or to get a powerful test filtering system. Grammar-based Testing, Solver-based Testing, Random-based Testing, Contract-based Testing… all these paradigms live in Hoa\Test. Several research papers have been published in major test literature and conferences.

An error or a suggestion about the documentation? Contributions are welcome!

Comments

menu