Integration of TAT functionalities inside DejaGnu

The TAT functionalities that have been considered worth to be integrated inside the DejaGnu mainstream are the following:

• the ability to provide a list of tests within a file via a --testlist option : this way you can launch tests in a certain order inside a directory and subdirectories. Those test can be expect scripts (.exp as currently supported by dejagnu) or any other kind of scripts (.sh, .csh, .pl, ...) or executables.
  
• the ability to launch regression tests : a test or a list of tests (scripts or executables) are executed, the outputs stored inside a file and compared to a reference file. If the outputs match, we send a PASS command and if it fails a FAIL one. It can be launched with --reg MyList [--g]. The --g option will create the references files.

The test list :

In Dejagnu, to launch a list of tests, you have to specify them in the command line making it possible to select the one you want and ignore the others. Tat is based on a TestList file that contains the tests to launch in a specific order.

The runtest options have been enriched so that is possible to launch a set of tests by specifying the name of the file containing the tests list. If the test name is starting with a #, it will be ignored.

runtest --testlist mylists

For example the file mylist could contain:

toto.exp
essai.exp
Pack1.sh

In order to execute only a single test within the test list the option --sin has been added so it could be possible to type:

runtest –-testlist mylist --sin Pack1.sh

(This additional --sin is needed since the current dejagnu is dealing only with test developed in expect so with an .exp extension)

These mentioned options can also be combined with already existing mechanism. For instance :

runtest –-testlist mylist toto.exp

Will only run the toto.exp tests inside this testfile.

In the same way, is  also possible to use the –ignore command so as not to launch some specific tests.

Regression tests :

In order to perform regression tests, some TAT source code has been added inside DejaGnu. Basically, when running a regression test, you have to specify a file containing the names of the tests to launch. When the tests will be executed, their outputs will be filtered and compared to a reference file.

The command is :

runtest --reg mytests [--g]

The –-g option makes it possible to (re)generate the reference file for a test or a set of tests.

Supposing this the content the file mytests :

#BBBB
bin
Pack1.exp
script.sh
toto.exp

It is possible to create the reference with the –-g  option. In every directory containing a test to execute, this option will create a ref/ subdirectory to store the reference files.

runtest --reg mytests --g

Here the four tests have their references created, so we can launch the regression test and if the outputs do not change, the tests should be successful :

runtest --reg mytests --all

The four tests have generated an output corresponding to the reference so they all pass.

In order to have time or configuration independent tests, the output is filtered. It is possible to substitute values by placing some sed command in the sed file and lines can be removed thanks to the grep file.

Here is an example of grep file :

Potential memory leak
invisible
Replacing

The lines inside the output file that will contain this patterns will be removed

Here is an example of sed file :

s/on/off/
s/111/000/

Every sed expression will be performed on the test output.

Naming convention:

When using the –reg option, if the name of the testlist used is foo, then the grep file it exists will be named foo.grep and the sed file foo.sed.