Questionable Benchmarks in May 1989 issue of IEEE CG&A

[Dave Ciemiewicz]

When I saw the article "A Set of Benchmarks for Evaluating Engineering
Workstations", May, 1989, issue of IEEE Computer Graphics and
Applications, I was initially furious.  The data is two years old
(gathered from "January 1987 to April 1987").  Some of the hardware is
not available on the market today.  For instance, SGI no longer makes
the IRIS 4D/60G or IRIS 2400T.  I may be wrong but I don't think you can
buy GE Graphicon boards anymore.

Not only is the data presented out of date, but I question the authors'
results.  First, Figure 1, "CPU benchmark performance" shows the
authors' CPU benchmark took 64 seconds to run on an IRIS 2400T.  By the
authors' numbers, the IRIS 2400T should be 47% of a VAX 8600 yet their
graph shows the IRIS 2400T at 30%.  The plots for the HP350SRX and
HP320SRX are at 15% and 10% yet the numbers should be 9% and 7%,
respectively.  This is sloppy both on the authors' part and on the part
of the peer review and editorial staff.  Such errors should never reach
the readers.

Also, the authors quote price/performance ratios but they don't give
any configuration information about the systems involved.  Factors like
the amount of memory and number, types, and capacities of disk drives
will affect the list price of a machine.  Without this information,
their price/performance data is useless.

If the authors are so sloppy in the presentation of their data, I
wonder how sloppy they were in the collection of their data.

The authors seem to think that their benchmark is more useful than
other benchmarks refered to in their article.  Maybe it is more useful
for their particular application.  However, their particular benchmark
may be useless for other computer graphics applications.

The best way you can tell how well your application is going to run a
given architecture is to run your application on that architecture.
This seems to be the gist of IEEE CG&A editor John Standhammer's note
at the beginning of the article.  Maybe Mr. Standhammer waited
approximately two years to publish the article so the description of
the benchmarking process would be the only useful part of the article.
This was a wise (or lucky) move on his part.  I only wish he hadn't
lured people into reading the article by plastering the first page
with the banner "Workstation Benchmarks".

In his note, Mr. Standhammer encourages "others to share their
evaluation procedures and results."  I encourage those people doing
systems evaluations to do so accurately and fairly.  I encourage the
CG&A editorial staff and peer reviewers to subject systems evaluation
articles to a very high degree of scrutiny.  Lastly, I encourage
readers of CG&A to demand this kind of quality.

These opinions are my own and do not reflect those of Silicon Graphics, Inc.
--

David M. Ciemiewicz
Member of the Technical Staff
WorkGroup Products Division
Silicon Graphics, Inc.