longjmp() out of nested signal handlers

[weiser.pa at xerox.com]

Well, what does the standard say about the values of register variables after a
longjmp is taken?  The 4.3bsd Vax implementation of longjmp carefully unwinds
the stack and restores the registers to their values when control left the
procedure doing the setjmp, which has the effect of preserving assignments to
register variables.  This makes register variables in the setjmping procedure
behave like other variables in the setjmping procedure.

The 3.X SunOS 68020 longjmp does not restore register values (although the 3.X
SunOS Sparc longjmp does).  We needed a longjmp that would restore register
values for our port of Cedar to the Suns.  So we wrote a stack unwinding
longjmp.  This was not easy, because...

The calling convention used on the 68020 by Sun (and I presume others) does not
place onto the stack anyplace information about how many registers were saved,
or where.  The registers are not always saved at the same offset from the frame
pointer, for instance.  So, our unwinder has to, for each procedure in the
stack: (1)  find the PC value (fortunately this is in a standard place), (2)
figure out the first instruction in the procedure from the current PC, (3)
examine the first few instructions to figure out what registers were saved
where, (4) recover the register values.

The hard step?  Number 2.  It requires keeping around for each procedure the
starting and ending PC values.  We considered trying to look at the call
instruction to figure out what it was calling, but the 68020 instruction set
doesn't enable us to unambiguously distinguish between a call through a register
value and a call immediate.

Now, what happens if there are signal handlers on the stack, and one wants to do
this unwinding?  Well, signal handlers don't look like procedure calls exactly:
in SunOS, at least, the kernel throws some junk on the stack, and then proceeds
to build a valid procedure frame.  One cannot just return through that junk.  So
our unwinder notices when it is about to unwind through _sigtramp, picks a
couple of registers out of the junk and skips the rest, and then keeps going. 

It could be complications like this which cause the longjmp standard to punt on
returning through nested signal handlers.  But I'd prefer to see a standard say
that it must be possible to do it, because it is clearly a good thing to be able
to do.

-mark