how do I make a process release its terminal?

[postman# at andrew.cmu.edu.UUCP]

ReSent-To:nntp-xmit#@andrew.cmu.edu
Return-path: <zs01#@andrew.cmu.edu>
To: outnews#ext.nn.comp.unix.wizards at andrew.cmu.edu,
	m5d at bobkat.UUCP (Mike McNally (dlsh)),
	paul at vixie.UUCP
In-Reply-To: <471 at bobkat.UUCP>

I had a lot of fun trying to figure out process groups and control terminals
under BSD 4.2. There was this bug where I couldn't open /dev/tty in a certain
process (ENXIO error). So after spending a year looking for it in my code, I
got access to kernel sources. Here is what I found.

The kernel's idea of a process without a control terminal is one who's
process group is zero. It also has a pointer to the device, if this is NULL,
you don't have a control terminal. The documentation doesn't help here,
either you read the source or you lose. It says that if you have no control
terminal and you open another tty, it will become your control terminal. What
really happens is that if your process group is 0 and you open a tty, it
becomes your control terminal. If your process group is non-zero, and that
pointer is NULL, /dev/tty remains broken.

Getting rid of a control terminal is pretty easy. The following code
(strictly BSD 4.2) will do it:

#include <stdio.h>
#include <sys/file.h>
#include <ioctl.h>

void NukeControlTerminal()
{

    int fd;

    if ((fd = open("/dev/tty", O_RDWR)) >= 0) {
        ioctl(fd, TIOCNOTTY, NULL);
        close(fd);
    }
    else
/* Insert real error handling here... */
        printf("Couldn't open control terminal, errno = %d\n", errno);
}

My bug was caused by setting the process group and then trying to open
/dev/tty. Since I really didn't have a control terminal (i.e. pointer was
NULL), it returned EXNIO. But when I opened another tty, it did not become
/dev/tty (because my process group was not 0).

There are a number of other problems you can get into by changing the process
group of a terminal (or of your process), and then trying to do an ioctl on
it. For example where you wish to fork a child, and have a terminal in the
child's process group but not in yours. If you do the process group switching
in the wrong order, you will not be able to ioctl the terminal into the
target process group...

Sincerely,
Zalman Stern
Information Technology Center
Carnegie Mellon University
zs01 at andrew.cmu.edu