Why is Pyramid TCP so fast?

[Robert H. Hyerle]

We need to run a simulation based on two or more processes running on
different machines.  In particular, one process must run on a MIP's
(RiscOS) box and another on a SPARC.  These processes need to
synchronously exchange information periodically (such as every simulated
clock tick).  The data exchanged runs into 10's of bytes, 10's of times
per second.  So this is not a heavy load.

To test the rate at which we could expect to exchange data, we wrote some
test programs.  One starts a "responder" program on one machine that sets
up two sockets for stream communication, establishes connections, waits
for a packet (that is, some 10's of bytes from the "receive" socket) by
doing a "read" system call, upon returning from the "read" issues a
"write" on the "send" socket.  This repeats.  One starts, at the same
time, a "sender" process which likewise sets up two sockets and makes the
corresponding connections with the "responder," "writes" a packet of
information (sending it to the responder), then waits on a "read" on the
other socket for the return packet of the "responder."  This repeats.  In
pictures:

    Sender:			|	|    Responder:
	while ( ! done ) {	|	|	while ( ! done ) {
	    write(snd,buf,128);	| ----> |	    read(rcv,rbuf,128);
	    read(rcv,rbuf,128);	| <----	|	    write(snd,buf,128);
	}			|	|	}

Between two Pyramid machines, the two processes manage to send about 500
128 byte messages per second.  Between a Pyramid and a SPARC, a Pyramid
and a MIPS, a SPARC and a SPARC, a MIPS and a MIPS, a SPARC and a MIPS,
... the rate goes down to 5 messages per second.  Even if both processes
are on the same host (e.g. both on a SPARC), the rate is 5 messages per
second!

So, the question is "Why is Pyramid so fast?", "What is different about
Pyramid's TCP?", and really "How do I make the SPARC's and MIP's machines
go faster?"  The best guesses so far are that some unfortunate buffering
or flow control or delayed "ACK" or something is constipating the
connections.

We have tried packet sizes from 1 to 1000 bytes with no effect.

Yes, we could write our own little protocol using UDP, but we would much
prefer the simplicity of a TCP virtual circuit.

Here is one clue:  I have set the buffer sizes with the "setsockopt" call
(SO_SNDBUF, SO_RCVBUF) to exactly match the number of bytes being written
or read.  This ups the "both processes on the same host" performance to
that of the Pyramid:  ~500 messages per second.  Unfortunately, it has no
effect when actually using the network.  Other hints as to what the
problem might be include something called a "push" bit.  Is it on for
Pyramids and off for everyone else?  How does one turn it on or find out
its setting?

-- robert hyerle; hyerle at pyramid.com