µOCCAM

CS3 Individual Programming Project
2000-2001

Test Script

NAME

ipp -- test CS3 individual programming project work.

SYNOPSIS

ipp [-1|-2] [options]
ipp1 [options]
ipp2 [options]

DESCRIPTION

The ipp program will check a µOCCAM compiler written for the CS3 individual programming project, applying tests provided by the user. To enable the script, extend your search path using the command

setpath /home/cs3/ipp/bin

The program makes the following preliminary checks.

Source code for the compiler must be in a directory that is protected against access by others; no read, write or execute permissions for group or other.
Executing make in the source directory must run without error.
This should create an executable file named "occam".

If any of these fails, ipp prints an appropriate error message and terminates. Otherwise, it runs the created µOCCAM compiler on sample files provided in a separate test directory. These files have the following different forms, indicated by the extension part of their names.

`<test>.accept`	Valid occam code that the compiler should accept.
`<test>.reject`	Illegal occam code that the compiler should reject.
`<test>.io`	A valid occam program to be compiled and run.
`<test>.in`	Input for the corresponding `<test>.io` program.
`<test>.out`	Correct output for `<test>.io`
`<test>.comment`	Brief text describing the `<test>` program.

Every <test>.io. file must have matching <test>.in and <test>.out files. The <test>.comment files are optional.

While running, ipp writes a two-line report to standard output.

ipp2: ++**---//---//-/!!?!
ipp2: Passed 12 of 20 tests

Here "ipp2" indicates the directory containing the occam program. The first line has one character for the outcome of each test:

+ valid program correctly accepted
* valid program incorrectly rejected
- invalid program correctly rejected
/ invalid program incorrectly accepted
! I/O program produced correct output
? I/O program produced incorrect output
% I/O program incorrectly rejected
^ compiler aborted or exceeded time limit.

The second line summarises the results. A separate log file ipplog records detailed information about performance on all the tests.

OPTIONS

Several options exist to control how the program operates. Most have default values, which may also be given by environment variables. If an option is repeated, the last occurrence takes priority.

-1 -2 --stage <n>

Determines the relevant stage of the practical. This can also be fixed using the name by which the program is invoked: ipp1 or ipp2. There is no default value: if no stage can be determined, ipp terminates with an error message.

For stage 1, the occam compiler is expected to read a program from its standard input and performs syntactic checking only. None of the I/O tests are used. In stage 2, the occam compiler is given a file name, and I/O tests are included. For simple accept/reject tests in stage 2 the compiler is given the "-r" command-line switch to prevent running compiled code.

-d <dir> --directory <dir>

Directory containing source code for the occam compiler, and a suitable Makefile. Default is "ipp1" or "ipp2" depending on the stage.

-t <testdir> --tests <testdir>

Directory containing all the test files. Default "ipptests".

-l <logfile> --log <logfile>

File to record detailed report on tests. This is in addition to the two-line diagnostic described earlier. Default "ipplog".

-s <seconds> --limit <seconds>

Maximum CPU time in seconds given to the occam compiler for each test. The ulimit function is used to cut off any program from taking too long. Default value 2 seconds.

ENVIRONMENT

Each of the options above has a corresponding environment variable.

IPPSTAGE	Stage of practical, 1 or 2.
IPPDIR	Directory for compiler source code.
IPPTESTS	Directory for test µOCCAM programs.
IPPLOG	Log file.
IPPLIMIT	CPU time limit per test, in seconds.

FILES

`ipp1/`, `ipp2/`	Directories for compiler source code.
`occam`	Executable program in `ipp1/` or `ipp2/` as appropriate.
`ipptests/*`	µOCCAM test programs
`ipplog`	Record of test results.

LOG FILE

The log file contains the following information.

Time, date, option settings.
Report on access protection for source code.
Standard output and standard error from make.
Report on presence of "occam" executable.
For each test:
- Title of test, kind of test, and comment if available.
- Standard output and standard error from running occam on test file.
- Result of test (accept or reject) and whether that is correct.
A summary of how many tests passed and failed of each kind.

EXIT STATUS

If the preliminary checks fail, the program returns the error code 1. Otherwise, it returns 0 to indicate success, regardless of how well the compiler performs on the individual tests.

WARNING

The program makes no check to see whether the test files provided by the user actually are valid or invalid occam programs as claimed.