generation based tty security IDEA

[Keith Muller]

There is another way to handle access checks other than placing the
info in the open file table (that should work on cases where there is
no open file table reference to the tty). A generation based revocation would
eliminate the file table searching for references. (Generation based
protection is a well known general technique).
This idea would require a little more coding, but may be less objectionable
than the crude file table searching vhangup()...

Consider adding a generation field in the tty structure. (A generation
is an integer that increments under controlled conditions). This
has already been done in 4.3 RENO, it is called the t_gen field.  In
Reno this value is incremented in ttyclose() (which is called on
last close). This value is used to catch sleepers in ttysleep(). A simple
description of ttysleep() is that it works by saving the current value
on the per process kernel stack before doing the sleep. After wakeup it
checks the current value to the saved value. If the two values are different,
a value called ERESTART is returned  (an ok return is a 0).  If the return
value of ttyclose() is not zero, then the system call returns without doing
the operation and returns a 0 to the caller. However if last close on the
device can be avoided, the t_gen field is not incremented (allowing improper
access to the device).

This technique can be expanded to revoke access on tty allocation (either on
hardwired ports by getty/init, or by a server that has just obtained
the master side of a pty and needs to assure the slave side is protected).
For each open file table entry (or in the per process fd table) add a
field to store the generation number. This fd field is set on first open
to be the same as the generation value stored in the tty structure it
references only after the file system access checks have granted permission
to open the device. On each system call that operates on a tty device, the
generation number associated with this file descriptor MUST be checked against
the current generation number in the tty structure. If they do not match,
the system call fails. 

The generation number in the tty structure (t_gen) is incremented in two
cases. First by ttyclose() on last device close() and second by the revoke()
system call.

Revoke() is used by getty/login and by all processes that allocate
ptys (processes that open the master side). On ptys, revoke is used on the
slave side after access to open the master side has been granted
(assuming the current scheme of a sucessful open on the master side indicates
an reusable pty).

The action of Revoke(fd) is to increment the generation number and
reset the generation number stored for the fd it was called with to the new
current generation number. All other fd's now have an out of date generation
value and can no longer perform tty operations. No traversals of the open file
table are required (unlike the traditional vhangup()... which is not in RENO).

To summarize use (as root):

1) Open master side (for ptys only)

2) open the tty (or slave side pty) 
	The tty fd (slave side for ptys) ageneration value is assigned the
	CURRENT tty generation value (the t_gen in the tty structure)

3) Set exclusive use on the tty (slave side for ptys) to prevent additional
	opens.

4) Call revoke(fd) (where fd is the tty fd).
	This increments the generation value in the tty structure (t_gen)
	and copies this value to the generation field associated with the
	fd file desriptor. Only this file descriptor has access right to
	this tty at this point. All other fd's have older generation values.

5) Set the modes, owner and permissions on the tty.
	THis sets file system access rights for future opens.

6) Remove the exclusive use on the tty.
	The tty is now secure for use.

The nice thing is that there is no special requirement for where each fd's
generation needs to be stored since they do not have to be globally
accessible.

Keith Muller
University of California
kmuller at ucsd.edu