Ultrix 4.1 TIOCSPGRP peculiarities -- is this correct behaviour?

[Ian! D. Allen [CGL]]

You cannot use TIOCSPGRP to set a tty into a process group N that has
any other processes in it, unless your controlling terminal matches the
controlling terminal of some process that is also in process group N.
You can't even do it if you're running as root.  Is this useful?

This means if some user somewhere on the system happens to have a
process in a process group that you want to use, you can't do an ioctl
on a tty to set that process group unless you first yourself create some
dummy process in the process group.

Examine the following.  I have a command that uses TIOCSPGRP to set a tty
process group.  When given a process id, it uses setpgrp on the pid.

    % setpgrp 6612 /dev/ttyq6       -- this doesn't work
    /dev/ttyq6: Not owner
    % sleep 999 &
    [1] 5585
    % setpgrp 6612 5585             -- put the sleep in pgrp 6612
    % setpgrp 6612 /dev/ttyq6       -- this works now (?!)
    % kill 5585
    [1] 5585 Terminated sleep 999
    % setpgrp 6612 /dev/ttyq6       -- now it fails again
    /dev/ttyq6: Not owner

What is happening is that the initial setpgrp on ttyq6 is failing because
the kernel can't find any processes with my current controlling tty and
the 6612 process group.  So, to make the ioctl succeed, I simply create a
silly process (thus, it inherits my current controlling tty) and put it
in the 6612 process group.  I re-issue the setpgrp against ttyq6, the kernel
finds a match for process and control tty, and lets the ioctl succeed.

Is this useful?  Since it is trivial to bypass the "security" this
provides, why restrict the ioctl at all?

RISC/os (UMIPS) 4.51 and BSD4.3 do not restrict the ioctl, and
programs I've ported from these systems don't work unmodified on Ultrix.

SunOS 4.1 has some even stranger rules about TIOCSPGRP.

This has been around since at least Ultrix 3.1C.  I posted this
observation last December; now we have kernel sources so I know what is
going on, and it's still going on in Ultrix 4.1.

#include <standard wish for an electronic SPR form>

Note:
    Anyone can still use TIOCSPGRP to set any readable or writable tty to
    any *unused* process group.  This can really foul up your friends who
    leave their terminals unprotected; but then, so can "stty 0".
-- 
-IAN! (Ian! D. Allen) idallen at watcgl.uwaterloo.ca idallen at watcgl.waterloo.edu
 [129.97.128.64]  Computer Graphics Lab/University of Waterloo/Ontario/Canada