Test suites

[Chuck Karish]

In article <5265 at lynx.UUCP> m5 at lynx.UUCP (Mike McNally) wrote:
>2)  Has anyone bought the NBS-PCTS test suite for FIPS 151 (posix)?
>If so, I'd appreciate comments.

	I installed the NBS-PCTS here at Mindcraft last summer, and
	tried to run it.  It compiles all in a batch, and many parts
	won't compile if your system doesn't completely conform to
	POSIX.

	The instructions are terse and the code, based on the SVVS, is
	somewhat idiosyncratic.

	Using our POSIX compatibility tools, we managed to compile and
	run the whole NBS-PCTS on a non-POSIX system.  This gave us an
	early look at what extensions that operating system needed to
	come up to the standard.

	We've run it in parallel with the test suite we developed for
	IBM.  This is a good cross-check, to show us which failures are
	real failures, which are due to problems in one test suite or
	the other, and which are due to differing interpretations of
	the standard.

	Several people I talked to at Uniforum last week pointed
	out that there are some bugs/errors in the NBS-PCTS.  I'm sure
	there are a number of them in the IBM-PCTS, too.  Both
	developers (Mindcraft and the NIST [was: NBS]) are willing and
	able to fix deficiencies in the tests.  AT&T is more
	conservative about changing the SVVS, probably because they
	don't want to make already-certified operating systems
	non-conforming.


	Chuck Karish	karish at denali.stanford.edu
			hplabs!hpda!mindcrf!karish
			(415) 493-7277