ANSI Validation (And Validation in general).

[Paul Reger]

I was wondering what experiences anyone has had with test suites for
C.  To date we are using a commercial suite for C but we are finding
the coverage of it lacks in many areas particulary when it comes to
'oddball' expressions found often in kernel code.  For example the
following code construction:

    if ((struct xyz *) 0x2000 ->element == 29)

Is not in the test suite, neither is:

int
func(i)
    register int i;
{
...
    if (((short) i = f(45)) < 0)
  /* the line containing the problem... */


Is not in our suite either.

We are embellishing the suite when we do find the bugs, but the bugs
are not found in a 'scientific' way.  Rather, we just 'build the
world' (Operating system on down), and then testing the built binaries
often reveals bugs in the compiler (arrrgh!!!).

The process is time consuming and therefore costly.

What I would be interested in is responses from your experiences with
testing a C compiler:

1. What suites do you consider the best ones ??

2. Are there automatic tools available which produce and evaluate
random sequences of 'oddball' C expressions (this assumes they also
contain an 'Oracle' which will evaluate the correctness of the code
produced by the compiler...).

3.  How successful are such automatic test case generators ???

             paulr       (Paul Reger)
     Sequent Computer Systems  Beaverton, OR.
 ... {sun,ucbvax!rutgers!ogccse,uunet}!sequent!paulr