Mysterious pty behavior - lotsa ^Ds (was Re: rlogins hanging...)

[Guy Harris]

If you get a pseudo-tty into the state where it acts as if "every
character typed is a ^D", or "it runs billions of 'ls'es", or something
like that, or get it into a state where it isn't in "raw" or "no-echo"
mode but doesn't echo what you type, under 4.0, it's probably because it's
gotten into an invalid "half-remote mode" state.

Basically, "cmdtool" puts it into "remote mode", which is described (to
some degree) in PTY(4).  If you ran something from the "cmdtool" in the
background, and left it around after you quit from the "cmdtool", it holds
the "slave" side of the pty in remote mode, but not the master side.

The "remote" mode explains the seeming "raw, no-echo" state; the "every
character typed is a ^D" stuff seems to come from the bizarre stuff the C
shell does when you do "set filec", and the "billions of 'ls'es" are just
the C shell's response to an end-of-file (^D) when not at the end of the
line - it's showing you the options for the file name you're trying to
complete.

There is a fix; I've mentioned it to Sun.  (For those of you with source
and an adventurous spirit, you want to make the pseudo-tty
master/controller open routine "ptcopen" send an M_CTL streams message
with an MC_DOCANON code upstream on the slave side if it's open and has a
stream; check out the code in "ptcioctl" that handles a TIOCREMOTE "ioctl"
with a zero argument.  I've not tried this, but I think it should fix it;
if it blows up and scribbles "I am a duck" all over your root file system,
don't ask me about it.)

There's also a workaround: the attached program will take a pseudo-tty
master out of remote mode, which will also unwedge its slave side.  You
run it with the "/dev" name of the *master* side as an argument; this
means that if the tty that's wedged is "/dev/ttyp0", you give it
"/dev/ptyp0" - "pty", *NOT* "tty" - as an argument.  Since pseudo-tty
masters are "exclusive-use" devices, this will only work if no process has
the master side open, so you can't run it from the wedged terminal. 

__________cut here__________

#include <stdio.h>

extern int	fprintf();

#include <fcntl.h>

extern int	open();

#include <sys/termio.h>

extern int	ioctl();

#include <errno.h>

extern void	perror();
extern void	exit();

static char	*strerror();

int
main(argc, argv)
	int argc;
	char **argv;
{
	register int fd;
	int zero = 0;

	if (argc != 2) {
		(void) fprintf(stderr, "Usage: fixup pseudo-tty\n");
		return 1;
	}

	if ((fd = open(argv[1], O_RDWR)) < 0) {
		(void) fprintf(stderr, "fixup: Can't open %s: %s\n", argv[1],
		    strerror(errno));
		return 1;
	}

	if (ioctl(fd, TIOCREMOTE, &zero) < 0) {
		(void) fprintf(stderr, "fixup: Can't fix %s: %s\n", argv[1],
		    strerror(errno));
		return 1;
	}

	return 0;
}

static char *
strerror(errnum)
	int errnum;
{
	extern int sys_nerr;
	extern char *sys_errlist[];
	static char msg[6+10+1];	/* "Error "+number+'\0' */

	if (errnum < 0 || errnum > sys_nerr) {
		(void) sprintf(msg, "Error %d", errnum);
		return msg;
	} else
		return sys_errlist[errnum];
}