Secure NFS exhausts kernel resource

[Ed Keizer]

While converting from SunOS 3.5 to SunOS 4.0.1 we decided to use the
secure NFS software to protect the staff file systems at our faculty.

The first sign of `something wrong somewhere' was that the server
exporting the Secure NFS system started crashing about once a week
on null pointer derefences in kernel code connected with authorization.
We also had to reboot one of our diskless clients after each crash.
That client had produced the error message:
	vmunix: authget: authdes_create failure
and could not be convinced to perform any further accesses the Secure
file system, not even after rebooting the server.

We did not pursue this in the hope that SUN would have repaired this in
SunOS 4.0.3. We would have started searching if we had the kernel code,
but we don't so we left it at that, although it was somewhat annoying.

Then, one day, our server produced the following error message:
	vmunix: ie0: out of mbufs: output packet dropped
while  more than a few diskless clients produced the message mentioned
earlier.
This was the sign to start a search for an mbuf leak in the kernel.

We found that each unauthorized access to a Secure File System used 10
`mbufs allocated to data' which where never freed.  An unauthorized access
happens when a process with a uid that has a public key in the publickey
data bases tries an access from a client that does not have that users
private key.

Unauthorized accesses happen whenever a user with a key in the publickey
database and his or her home directory on the Secure file system does an
rlogin, reading $HOME/.rhosts, to a client he or she has never used
before. Or, when somebody tries the well-known trick of `su user' after
becoming super-user in order to access that users files over the network
from a client that does not have that users private key.

We often have two of these `Unauthorized access' messages:
	vmunix: NFS getattr failed from server: RPC: Authentication error
when one of the events mentioned above happens.
That means that each event costs us twenty mbufs. Mbufs are are finite
resource. The kernel code limits the amount of memory dedicated to mbufs
to 1 Mb. In practice this means that we have to reboot our Secure NFS servers
every second day. We see the amount of mbufs allocated to data growing
from about 20 to 2924 and higher.

We had our first SunOS 4.0.3 system running yesterday. The bug was still
there.  We reported this problem to SUN through the official channel a few
days ago, but have not yet received an answer.

Ed Keizer

	Vakgroep Informatika
	Vrije Universiteit
	Amsterdam
	The Netherlands

	keie at cs.vu.nl