Saving stderr output in memory

[Larry Wall]

In article <14020086 at hpisod2.HP.COM> decot at hpisod2.HP.COM (Dave Decot) writes:
: Hi.  I am writing a C program, say prog1, that wants to work on System V.3
: or later and BSD 4.2 or later.
: 
: It wants to run an arbitrary shell command and collect all of that
: command's standard output into a buffer in memory, and all of its standard
: error output into another buffer.  I need to know the exit status of the
: shell command to determine what to do next.  I need to use prog1's 
: original standard output and standard error for other purposes later.
: I don't know in advance how long either of the standard output or
: standard error of the command will be, but it is usually under 80 bytes.
: I don't want to use temporary files if at all possible.
: 
: I used popen(cmd, "r") to run the command and fgets() to retrieve the
: output into a buffer.  I shifted and masked the return value of popen()
: to get the exit status of the command, and then call pclose() to wait
: for the child process and get rid of the popen() stream.  All of this is
: very straightforward.
: 
: However, capturing the *standard error* output of the command was harder,
: since popen() makes no allowances for that, and I didn't want to use
: a temporary file.
: 
: Here's what I tried.  Prior to calling popen(), prog1 saves the original
: standard error file by duplicating it to another file descriptor, then
: creates a pipe, then uses dup2() to duplicate the write end of the pipe
: to file descriptor 2 (standard error).  If you follow the above, prog1
: is now all ready to read from the read end of the pipe whatever is
: written to its own (and any forked child's) standard error output.  So,
: as described above, prog1 now calls popen(), reads the standard output
: from the popen() stream, and finally calls pclose() to wait for the
: child process and destroy the popen() stream.
: 
: OK, now.  The problem that arises is that the child process created by
: popen() cannot write any more than a pipe-ful bytes to the standard error
: pipe, because nobody reads any of it from the other end until after the
: child process has terminated and been waited for by the pclose() call.
: 
: I thought of keeping the popen() stream open until after reading from
: the standard error pipe, but then I cannot determine when I have read all
: of the standard error output, since the child process may still be running.
: 
: Has anyone any brilliant suggestions?

You need a separate process to collect the stderr while your original process
is collecting stdout.  This can be a child process which runs the actual prog1.
When prog1 dies, the child appends the collected error messages to stdout,
plus the exit status.  Your program then looks for the magic lines at the
end of the ordinary stdout output.  You can do it something like this:

#include <stdio.h>
main()
{
    char stdoutbuf[8192];               /* or whatever */
    char stderrbuf[8192];
    int exitstatus;
    FILE *fp;
    char *bufptr,*bufend;

    fp = popen("cachestderr 'prog1'", "r");	/* prog1 is any shell command */
    bufptr = stdoutbuf;
    bufend = bufptr + sizeof stdoutbuf;
    while (fgets(bufptr,bufend-bufptr, fp)) {
        if (strcmp(bufptr,"*** HERE ARE THE ERRORS ***\n") == 0)
            break;
	bufptr += strlen(bufptr);
    }
    *bufptr = '\0';
    bufptr = stderrbuf;
    bufend = bufptr + sizeof stderrbuf;
    while (fgets(bufptr,bufend-bufptr, fp)) {
        if (strncmp(bufptr,"*** EXIT STATUS WAS",19) == 0)
            break;
	bufptr += strlen(bufptr);
    }
    *bufptr = '\0';
    exitstatus = atoi(bufptr+20);
    pclose(fp);

    printf("The stdout was:\n");
    fputs(stdoutbuf,stdout);
    printf("The stderr was:\n");
    fputs(stderrbuf,stdout);
    printf("The exit status was %d, signal %d\n",
      exitstatus / 256, exitstatus % 256);
}

Of course, then you have to write cachestderr.  The following is one way:

#!/usr/bin/perl
pipe(IN,OUT);			# pipe() requires 3.0 patchlevel 9
open(STDERR,">&OUT");		# hook stderr to the write side of the pipe
close OUT;			# this filehandle not needed now
fork || (close IN, exec @ARGV);	# run command as a child
close STDERR;			# make sure only child has write side open.
@cache = <IN>;			# gets EOF when child dies
wait;				# harvest status
print "*** HERE ARE THE ERRORS ***\n", @cache, "*** EXIT STATUS WAS $? ***\n";

Note that we do nothing with stdout except keep it open till prog1 is done,
then scribble on it ourselves and exit.

Here's a test prog1 for you, while we're at it:

#!/usr/bin/perl
print "Line 1 to stdout\n";
warn "Line 1 to stderr\n";
print "Line 2 to stdout\n";
warn "Line 2 to stderr\n";
exit 123;

Larry Wall
lwall at jpl-devvax.jpl.nasa.gov