* Borislav Petkov:

quoted

One possibility is that the sigaltstack size check prevents application
from running which work just fine today because all they do is install a
stack overflow handler, and stack overflow does not actually happen.

So sigaltstack(2) says in the NOTES:

       Functions  called  from  a signal handler executing on an alternate signal stack
       will also use the alternate signal stack.  (This also applies  to  any  handlers
       invoked for other signals while the process is executing on the alternate signal
       stack.)  Unlike the standard stack, the system does not automatically extend the
       alternate  signal  stack.   Exceeding the allocated size of the alternate signal
       stack will lead to unpredictable results.

quoted

So if sigaltstack fails and the application checks the result of the
system call, it probably won't run at all. Shifting the diagnostic to
the pointer where the signal would have to be delivered is perhaps the
only thing that can be done.

So using the example from the same manpage:

       The most common usage of an alternate signal stack is to handle the SIGSEGV sig‐
       nal that is generated if the space available for the normal process stack is ex‐
       hausted: in this case, a signal handler for SIGSEGV cannot  be  invoked  on  the
       process stack; if we wish to handle it, we must use an alternate signal stack.

and considering these "unpredictable results" would it make sense or
even be at all possible to return SIGFAIL from that SIGSEGV signal
handler which should run on the sigaltstack but that sigaltstack
overflows?

I think we wanna be able to tell the process through that previously
registered SIGSEGV handler which is supposed to run on the sigaltstack,
that that stack got overflowed.

Just to be clear, I'm worried about the case where an application
installs a stack overflow handler, but stack overflow does not regularly
happen at run time.  GNU m4 is an example.  Today, for most m4 scripts,
it's totally fine to have an alternative signal stack which is too
small.  If the kernel returned an error for the sigaltstack call, m4
wouldn't start anymore, independently of the script.  Which is worse
than memory corruption with some scripts, I think.

Or is this use case obsolete and this is not what people do at all?

It's widely used in currently-maintained software.  It's the only way to
recover from stack overflows without boundary checks on every function
call.

Does the alternative signal stack actually have to contain the siginfo_t
data?  I don't think it has to be contiguous.  Maybe the kernel could
allocate and map something behind the processes back if the sigaltstack
region is too small?

And for the stack overflow handler, the kernel could treat SIGSEGV with
a sigaltstack region that is too small like the SIG_DFL handler.  This
would make m4 work again.

Thanks,
Florian