On Tue, May 26, 2026 at 5:39 PM William Roberts
[off-list ref] wrote:

Hello,

I am trying to understand the motivation behind having
security_task_prctl only continue if the return value is -ENOSYS. This
seems to be very different from other LSM hooks I have investigated.
For example, in other hooks, the value from SE Linux avc_has_perms is
used directly. This essentially means that a 0 will cause the check to
pass, and anything < 0 usually an error.

In commit:
----
commit d84f4f992cbd76e8f39c488cf0c5d123843923b1 ("CRED: Inaugurate COW
credentials")

(8) security_task_prctl() and cap_task_prctl().

         security_task_prctl() has been modified to return -ENOSYS if it doesn't
         want to handle a function, or otherwise return the return
value directly
         rather than through an argument.

         Additionally, cap_task_prctl() now prepares a new set of
credentials, even
         if it doesn't end up using it.
----

The check in kernel/sys.c is currently:
        error = security_task_prctl(option, arg2, arg3, arg4, arg5);
        if (error != -ENOSYS)
                return error;

Should this be something like, "error && error != -ENOSYS"?

I ask because I am looking to leverage this hook in SE Linux, and it's
annoying to have to coerce all 0 returns to -ENOSYS.

Of course after hours of banging my head and one email sent, it's more clear to
me now WHY. This hook isn't meant for making yes or no decisions on an operation
but rather to also handle special prctl flags for an LSM in question.

I guess with the said, do we want this interface to be used for both
a, let the lsm handle
this prctl flag directed to me, as well as a yes/no security decision
or do we want to split
this out into two hooks?

Thanks,
Bill