Wrapping TAT from DejaGnu

So as to run a test suite with DejaGnu, you basically have to specify the name of the tool you want to launch and the name of the source directory (directory containing the directories with tests).
A tool is nothing but the name used as a prefix by the directories containing the test suites that you want to launch : for instance if you specify 'foo' as the tool option for DejaGnu, inside the source directory, it will launch all the test files inside the directories whose names are prefixed by 'foo'.
Inside this directory, a directory whose name start by the name of the tool option must be created (for instance if the name of the tool you pass to runtest is mytool, the directory name must be referred to as mytool*, like mytool.test). The tests that will be stored inside this directory must be expect scripts (.exp) and will be automatically launched by DejaGnu.

Let's use the example of Dy :

Inside the test directory we create the dejatest and the dejatest/DyTest directory as well as the  dejatest/DyTest/DyTest1.exp file.

Now, to launch the test suite by calling DejaGnu instead of TAT we have to be in the test directory (exactly like for a TAT call) and run the command :

runtest --all --tool Dy --srcdir dejatest

The site.exp file

It is possible to avoid typing all the options with the runtest command by specifying them in the site.exp file. For instance, the site.exp file for the previous test would look like this :

set all_flag 1
set tool Dy
set srcdir /virgoDev/dejagnu_tests/dejatest

The site.exp must be in the directory from where you launch runtest with the single command :

runtest

The DEJAGNU environment variable

DejaGnu makes it possible to define the site.exp file you will use by default if you don't put one in the directory from where you call runtest. The only thing to do is to specify the site.exp file inside the DEJAGNU environement variable.

The wrapping file

In order to wrap TAT calls with DejaGnu, we will use the functionnalities of DejaGnu. We have to launch an expect script located in a specific folder that will execute TAT, read its outputs and translate them in DejaGnu outputs.
This wrapping file is nothing but a special test file that we can call using a specific 'tool' name according to the name of the directory where the file is stored.

A directory tree has to be created, for instance the wrapper can be stored in, it will be done by the install script of CMT, but we will use the following example to illustrate the way it works.

/virgoDev/dejagnu_tests/test_include/wrap.exp

and the main site.exp file in :

/virgoDev/dejagnu_tests/site.exp

The site.exp file will contain these lines :

set all_flag 1
set tool test
set srcdir /virgoDev/dejagnu_tests

The DEJAGNU environment must be set up too :

setenv DEJAGNU /virgoDev/dejagnu_tests/site.exp

Runtest is the main script of DejaGnu in charge of launching the test suites. In this case the --all options makes it possible to have the PASS outputs as well as the FAIL ones (the default is not to show the PASS ones). The --tool option specifies the name of the tool and will look inside the --srcdir to launch tests it finds in the directories whose names start with the tool option.

The Expect script will read the TestFile to count the number of tests it has to do and read TAT outputs so as to send the correct  PASS or FAIL outputs from dejagnu.

To run a TAT test, just go inside the directory where you would have launched TAT and execute runtest. The wrapper will be called, launching TAT and waiting for outputs and translate it in DejaGnu instructions.

For instance, here is the output of 3 tests on Dy :

PASS: dyTest1
FAIL: dyTest4
FAIL: dyTest7
                === Dy Summary ===

# of expected passes            1
# of unexpected failures        2

If you want to pass some extra options to TAT through the wrapper, use the TAT_OPTIONS Tcl variable when calling runtest.

For instance, if you want to pass the '-g' option so as to generate the referrence files of TAT, just execute :

runtest TAT_OPTIONS="-g"

In the end you will get 2 output files : the toolname.log which is exactly the output you've seen and the toolname.sum which summarizes what happened without warnings and so on.

Mixing TAT and DejaGnu tests

It would be interesting to be able to launch both former TAT and new DejaGnu tests with a single DejaGnu call an to have a common output for both test suites.
To perform this action we have to deal with DejaGnu limitations :

It is not possible to specify multiple source directories inside DejaGnu, which means that the test directories for the new DejaGnu tests and the directory containing the wrapper must be inside the same source directory.

Launching multiple testsuites with DEJATOOL and the runtest.sh script

It is possible to launch multiple testsuites by specifying multiple tools : it is not possible to do it by the command line or with the site.exp file.
For this reason, just comment the line for the tool in the site.exp file :

set all_flag 1
#set tool test
set srcdir /virgoDev/dejagnu_tests

Multiple test suites can be launched using the DEJATOOL environment variable.

The script runtest.sh has been written so that the tool name must be the name of the package, then if you want to launch the TAT tests of Dy, the name of the DejaGnu tool to use will be Dy and a directory tree testsuite/Dy.test/ will have to be created inside the test directory of the package to contain the new DejaGnu scripts.

This will be performed by the shell script runtest.sh. Basically it will create a symbolic link to the wrapper directory near the DejaGnu testsuite of the tool we want to test, launch runtest with the DEJATOOL variable containing the names of both the wrapper tool and the tool we want to test.

It will also display a final message informing if the global test suite failed or passed (a test suite fails if a least one test fails).

[ ... ]

    ===  Summary ===
# of expected passes            1
# of unexpected failures        1

********************
* Testsuite FAILED *
********************

If you want to pass an extra option to TAT (trough runtest, through runtest.sh), just specify argument on the command line to runtest.sh, all the arguments will be passed to runtest :

runtest.sh TAT_OPTIONS="-g"

Launching multiple testsuites without DEJATOOL

It is possible to avoid such a tricky work by modifying a little bit the directory tree.
The tests for a package are located in the tool/version/test directory.

It is possible to launch both TAT and new DejaGnu test suites by adding 2 files :

• Inside the test directory, copy the wrap.exp file as well as the new expect test scripts for DejaGnu
  
• in the parent directory of the test directory, copy a site.exp file like this one :
  

set all_flag 1
set tool test

And now, from the parent directory, just call runtest, it will lauch every expect script inside the test directory, scriupt after script, and when it will run the wrap.exp file, this one will execute the tat scripts.
With this method you can launch both TAT and DejaGnu scripts and have a single common output result for the both test suites.

The main problem with this solution is that it is requirred to create the site.exp and the wrap.exp files for each package you want to test and if the wrap..exp script is modified you have to copy again the file everywhere. But you can create a symbolic link to these files making it possible update the wrapping file for instance without having to copy it again everywhere.

This solution has the big advantage to be more maintainable because of more simple. It uses no extra script and just needs the basic functionalities of DejaGnu.