Rumour about IBM benchmarks

[pbickers at groucho]

In article <1990Sep14.215517.28056 at world.std.com> madd at world.std.com (jim frost) writes:
>david at torsqnt.UUCP (David Haynes) writes:
>>I have heard a rumour that the benchmark results that IBM posted for
>>their RS6000 system were the results of hand-coded, hand-optimized
>>assembler coding rather than the result of compiling C or FORTRAN
>>code. Can anyone confirm or deny this? 
>
I believe the IBM results but as everybody knows (or ought to know)
numbers like MIPS and MFLOPS are well nigh useless for evaluating a
machine. These rumours may originate in the very patchy performance
of the RISC6000. (See below.)

>Interestingly the machine doesn't `feel' as fast as it tests -- the
>one I have here (RS/6000 model 520 w/ 32Mb RAM and all the graphics
>hardware I'll ever need) feels more sluggish than a 12Mb sparcstation
 
This is interesting!

>running the same kinds of utilities (except things like `grep' which
>go quite fast).  I hear that performance becomes much better once you

I'm surprised by this remark. When we evaluated this machine one of the
things that struck us was that the Unix was slow. We found (on a 32 MB
configuration) that grep and diff were *slower* than on a 12.5 MIPS,
0.5 MFLOP HP9000/375.
>get more than about 40Mb RAM in the thing so I wonder if there's not a
>VM problem.  I won't know until we stuff more memory in it.
>
>sluggishness I notice -- things don't seem to be very well tuned right
>now.
>
One thing to note about this machine is that it gets its 7.4 MFLOP
performance from being *occasionally* able to execute more than one 
instruction at once. Thus it seems able to do very well on Fortran code
that contains some vectorizability. It is not lightning fast on Fortran
code per se -- though in general it is still quick. We run a lot of Fortran
and tested our codes on it. Some did show the 7.4 MFLOP performance relative
to rivals but for most it was within 10% of machines one might have expected
it to thrash e.g. MIPS R3000 and 33 MHz SPARC cpus. On some Fortran codes
the other machines were even faster.

Another thing to note about its floating point performance is that it
depends a lot on the type of operation being performed (true of any machine
I guess).

(It does come with a quite good Fortran compiler. It handles VMS source
quite well and the optimizer does optimize.)

I suppose what I'm trying to say is that the variation in performance for
the RISC6000 is considerably greater than for other workstations from SUN,
DEC, HP etc. -- i.e. its rivals in the market place. Thus while it may be
super fast at some things it is awfully slow at others, much more so than
one would expect. (There is an article in Unix Today, May 14 1990, which
bears further testimony to this.)

Thus anybody buying this machine solely on account of its 27 Dhrystone MIPS
and 7.4 MFLOP performance runs a high risk of being very disappointed. 

Consideration of other benchmarks such as Whetstone or Livermore Loops will
give a better overall but still incomplete picture.

Moral: if you're considering this machine test *your* applications on it (and
remember that you'll be doing editing etc as well as executing Fortran). You
may love it, you may hate it.

Good Luck!

-- 
Paul Bickerstaff                 Internet: pbickers at neon.chem.uidaho.edu
Physics Dept., Univ. of Idaho    Phone:    (208) 885 6809
Moscow ID 83843, USA             FAX:      (208) 885 6173