Poor implementation of TCP/IP or RPC on a SUN3/60 running SUNOS-4.0?

[Carsten Ruseng Jakobsen]

Major change of TCP performance when sending packets with sizes close to
4096 and 8192 bytes.

While evaluating the performance of SUN-RPC, I detected the TCP protocol
slowing dramatical down when sending packets with sizes close to the
magical 4096 and 8192 bytes. The tests I ran was a simple RPC-program with
a client calling it's server with an array of some specified size. The
server routine was a null routine, doing nothing.  When the array-size was
between (4045 - 4952) and (8090 - 8960) the time used for the
communication was 5 - 10 times slower than expected.  It was only the TCP
protocol that suddenly had this overhead, the UDP protocol seemed
unaffected, which makes me suspect the implementation of TCP. I tried to
figure out what happened, by monitoring the net with "traffic". I observed
that

 	1] there was NO collisions on the ethernet (10Mbit)
 	2] there were NO errors detected.
 	3] none of the cpu's were working very much
 	4] none of the machines were swapping
 	5] the number of context-switches weren't very high.
 	6] #packets/sec was 25% - 50% of what I saw when
 	   sending eg 512 bytes.

>From a profile of the program it can be seen that it is in the
function _write called from write_tcp that 78% of the time is used. In
the "normal" cases i expected 10% there.

The experiment was performed on 3 sun3/60 workstations running
SUNOS-4.0.1. They were totally isolated from anything else in the world.
They were not running yp, neither syslog or cron.  Except from the
operating system, the only processes running was the server and the client
processes.

The tests I performed was to measure the time used for 100 RPC calls.  For
each argument size the time for the 100 calls were measured 20 times, and
an average calculated. The RPC-call was made using TCP and UDP protocols.
Further was it performed with server and client on the same machine, and
server and client on two different machines.  The client called the server
with an argument of N bytes.  The results I got was:

- --------------------------------------------------------------
argument size|	      T C P         |		   UDP
	N    |	local	   remote   |    local		remote
- --------------------------------------------------------------
        0    |  6,9	   7,1     |||   5,6		5,8
      512    |  8,2        9,1     |||	 6,7	 	7,6
     1024    | 10,6       11,6     |||   7,9            9,1
     2048    | 14,9       15,7     |||  10,7	       12,8
     4000    | 23,6       24,2     |||  15,6           20,0
*    4096    |174,5      199,8     |||  16,1           20,3
     8000    | 40,2       40,8     |||  26,2           34,5
*    8192    |182,8      199,9     |||  26,8           35,1
             |                     |||  
- -----------|---------------------|||-------------------------------------

local = server-process and client-process reside on the same machine.
remote= server and client reside on 2 different nodes.

If you have any suggestions which explains these results, then please
*please* e-mail them to me.