Re: [PATCH 1/5] glibc: Perform rseq(2) registration at C startup and thread creation (v10)

----- On May 31, 2019, at 4:06 AM, Florian Weimer fweimer@redhat.com wrote:

* Mathieu Desnoyers:

quoted

I found that it's because touching a __thread variable from
ld-linux-x86-64.so.2 ends up setting the DF_STATIC_TLS flag
for that .so, which is really not expected.

Even if I tweak the assert to make it more lenient there,
touching the __thread variable ends up triggering a SIGFPE.

Sorry, I got distracted at this critical juncture.  Yes, I forgot that
there isn't TLS support in the dynamic loader today.

quoted

So rather than touching the TLS from ld-linux-x86-64.so.2,
I've rather experimented with moving the rseq initialization
for both SHARED and !SHARED cases to a library constructor
within libc.so.

Are you aware of any downside to this approach ?

The information whether the kernel supports rseq would not be available
to IFUNC resolvers.  And in some cases, ELF constructors for application
libraries could run before the libc.so.6 constructor, so applications
would see a transition from lack of kernel support to kernel support.

quoted

+static
+__attribute__ ((constructor))
+void __rseq_libc_init (void)
+{
+  rseq_init ();
+  /* Register rseq ABI to the kernel.   */
+  (void) rseq_register_current_thread ();
+}

I think the call to rseq_init (and the __rseq_handled variable) should
still be part of the dynamic loader.  Otherwise there could be confusion
about whether glibc handles the registration (due the constructor
ordering issue).

Let's break this down into the various sub-issues involved:

1) How early do we need to setup rseq ? Should it be setup before:
   - LD_PRELOAD .so constructors ?
     - Without circular dependency,
     - With circular dependency,
   - audit libraries initialization ?
   - IFUNC resolvers ?
   - other callbacks ?
   - memory allocator calls ?

We may end up in a situation where we need memory allocation to be setup
in order to initialize TLS before rseq can be registered for the main
thread. I suspect we will end up needing a fallbacks which always work
for the few cases that would try to use rseq too early in dl/libc startup.

2) Do we need to setup __rseq_handled and __rseq_abi at the same stage of
   startup, or is it OK to setup __rseq_handled before __rseq_abi ?

3) Which shared object owns __rseq_handled and __rseq_abi ?
   - libc.so ?
   - ld-linux-*.so.2 ?
   - Should both symbols be owned by the same .so ?
   - What about the !SHARED case ? I think this would end up in libc.a in all cases.

4) Inability to touch a TLS variable (__rseq_abi) from ld-linux-*.so.2
   - Should we extend the dynamic linker to allow such TLS variable to be
     accessed ? If so, how much effort is required ?
   - Can we find an alternative way to initialize rseq early during
     dl init stages while still performing the TLS access from a function
     implemented within libc.so ?

So far, I got rseq to be initialized before LD_PRELOADed library constructors
by doing the initialization in a constructor within libc.so. I don't particularly
like this approach, because the constructor order is not guaranteed.

One possible solution would be to somehow expose a rseq initialization function
symbol from libc.so, look it up from ld-linux-*.so.2, and invoke it after libc.so
has been loaded. It would end up being similar to a constructor, but with a
fixed invocation order.

I'm just not sure we have everything we need to do this in ld-linux-*.so.2
init stages.

Thoughts ?

Thanks,

Mathieu


-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com