PIE binaries are no longer mapped below 4 GiB on ppc64le

We tried to use Go to build PIE binaries, and while the Go toolchain is
definitely not ready (it produces text relocations and problematic
relocations in general), it exposed what could be an accidental
userspace ABI change.

With our 4.10-derived kernel, PIE binaries are mapped below 4 GiB, so
relocations like R_PPC64_ADDR16_HA work:

21f00000-220d0000 r-xp 00000000 fd:00 36593493                           /root/extld
220d0000-220e0000 r--p 001c0000 fd:00 36593493                           /root/extld
220e0000-22100000 rw-p 001d0000 fd:00 36593493                           /root/extld
22100000-22120000 rw-p 00000000 00:00 0
264b0000-264e0000 rw-p 00000000 00:00 0                                  [heap]
c000000000-c000010000 rw-p 00000000 00:00 0
c41ffe0000-c420300000 rw-p 00000000 00:00 0
3fff8c000000-3fff8c030000 rw-p 00000000 00:00 0
3fff8c030000-3fff90000000 ---p 00000000 00:00 0
3fff90000000-3fff90030000 rw-p 00000000 00:00 0
3fff90030000-3fff94000000 ---p 00000000 00:00 0
3fff94000000-3fff94030000 rw-p 00000000 00:00 0
3fff94030000-3fff98000000 ---p 00000000 00:00 0
3fff98000000-3fff98030000 rw-p 00000000 00:00 0
3fff98030000-3fff9c000000 ---p 00000000 00:00 0
3fff9c000000-3fff9c030000 rw-p 00000000 00:00 0
3fff9c030000-3fffa0000000 ---p 00000000 00:00 0
3fffa2290000-3fffa22d0000 rw-p 00000000 00:00 0
3fffa22d0000-3fffa22e0000 ---p 00000000 00:00 0
3fffa22e0000-3fffa2ae0000 rw-p 00000000 00:00 0
3fffa2ae0000-3fffa2af0000 ---p 00000000 00:00 0
3fffa2af0000-3fffa32f0000 rw-p 00000000 00:00 0
3fffa32f0000-3fffa3300000 ---p 00000000 00:00 0
3fffa3300000-3fffa3b00000 rw-p 00000000 00:00 0
3fffa3b00000-3fffa3b10000 ---p 00000000 00:00 0
3fffa3b10000-3fffa4310000 rw-p 00000000 00:00 0
3fffa4310000-3fffa4320000 ---p 00000000 00:00 0
3fffa4320000-3fffa4bb0000 rw-p 00000000 00:00 0
3fffa4bb0000-3fffa4da0000 r-xp 00000000 fd:00 34316081                   /usr/lib64/power9/libc-2.28.so
3fffa4da0000-3fffa4db0000 r--p 001e0000 fd:00 34316081                   /usr/lib64/power9/libc-2.28.so
3fffa4db0000-3fffa4dc0000 rw-p 001f0000 fd:00 34316081                   /usr/lib64/power9/libc-2.28.so
3fffa4dc0000-3fffa4df0000 r-xp 00000000 fd:00 34316085                   /usr/lib64/power9/libpthread-2.28.so
3fffa4df0000-3fffa4e00000 r--p 00020000 fd:00 34316085                   /usr/lib64/power9/libpthread-2.28.so
3fffa4e00000-3fffa4e10000 rw-p 00030000 fd:00 34316085                   /usr/lib64/power9/libpthread-2.28.so
3fffa4e10000-3fffa4e20000 rw-p 00000000 00:00 0
3fffa4e20000-3fffa4e40000 r-xp 00000000 00:00 0                          [vdso]
3fffa4e40000-3fffa4e70000 r-xp 00000000 fd:00 874114                     /usr/lib64/ld-2.28.so
3fffa4e70000-3fffa4e80000 r--p 00020000 fd:00 874114                     /usr/lib64/ld-2.28.so
3fffa4e80000-3fffa4e90000 rw-p 00030000 fd:00 874114                     /usr/lib64/ld-2.28.so
3ffff3000000-3ffff3030000 rw-p 00000000 00:00 0
[stack]

With a 4.18-derived kernel (with the hashed mm), we get this instead:

120e60000-121030000 rw-p 00000000 fd:00 102447141                        /root/extld
121030000-121060000 rw-p 001c0000 fd:00 102447141                        /root/extld
121060000-121080000 rw-p 00000000 00:00 0 
7fffb5b00000-7fffb5cf0000 r-xp 00000000 fd:00 67169871                   /usr/lib64/power9/libc-2.28.so
7fffb5cf0000-7fffb5d00000 r--p 001e0000 fd:00 67169871                   /usr/lib64/power9/libc-2.28.so
7fffb5d00000-7fffb5d10000 rw-p 001f0000 fd:00 67169871                   /usr/lib64/power9/libc-2.28.so
7fffb5d10000-7fffb5d40000 r-xp 00000000 fd:00 67169875                   /usr/lib64/power9/libpthread-2.28.so
7fffb5d40000-7fffb5d50000 r--p 00020000 fd:00 67169875                   /usr/lib64/power9/libpthread-2.28.so
7fffb5d50000-7fffb5d60000 rw-p 00030000 fd:00 67169875                   /usr/lib64/power9/libpthread-2.28.so
7fffb5d60000-7fffb5d70000 r--p 00000000 fd:00 67780267                   /etc/ld.so.cache
7fffb5d70000-7fffb5d90000 r-xp 00000000 00:00 0                          [vdso]
7fffb5d90000-7fffb5dc0000 r-xp 00000000 fd:00 1477                       /usr/lib64/ld-2.28.so
7fffb5dc0000-7fffb5de0000 rw-p 00020000 fd:00 1477                       /usr/lib64/ld-2.28.so
7fffff6c0000-7fffff6f0000 rw-p 00000000 00:00 0                          [stack]

There are fewer mappings because the loader detects a relocation
overflow and aborts (“error while loading shared libraries:
R_PPC64_ADDR16_HA reloc at 0x0000000120f0983c for symbol `' out of
range”), so I had to recover the mappings externally.  Disabling ASLR
does not help.

The Go program looks like this:

package main

import (
	"fmt"
	"io/ioutil"
	"os"
)

// #include <gnu/libc-version.h>
import "C"

func main() {
	// Force external linking against glibc.
	fmt.Printf("%#v\n", C.GoString(C.gnu_get_libc_version()))

	maps, err := os.Open("/proc/self/maps")
	if err != nil {
     		panic(err)
	}
	defer maps.Close()
	contents, err := ioutil.ReadAll(maps)
	if err != nil {
		panic(err)
	}
	_, err = os.Stdout.Write(contents)
	if err != nil {
		panic(err)
	}
}

And it needs to be built with:

  go build -ldflags=-extldflags=-pie extld.go

I'm not entirely sure what to make of this, but I'm worried that this
could be a regression that matters to userspace.

Thanks,
Florian

On Wed, 31 Oct 2018 18:20:56 +0100
Florian Weimer [off-list ref] wrote:

We tried to use Go to build PIE binaries, and while the Go toolchain
is definitely not ready (it produces text relocations and problematic
relocations in general), it exposed what could be an accidental
userspace ABI change.

With our 4.10-derived kernel, PIE binaries are mapped below 4 GiB, so
relocations like R_PPC64_ADDR16_HA work:

...

There are fewer mappings because the loader detects a relocation
overflow and aborts (“error while loading shared libraries:
R_PPC64_ADDR16_HA reloc at 0x0000000120f0983c for symbol `' out of
range”), so I had to recover the mappings externally.  Disabling ASLR
does not help.

...

And it needs to be built with:

  go build -ldflags=-extldflags=-pie extld.go

I'm not entirely sure what to make of this, but I'm worried that this
could be a regression that matters to userspace.

I encountered the same when trying to build go on ppc64le. I am not
familiar with the internals so I just let it be.

It does not seem to matter to any other userspace. Maybe it would be
good idea to generate 64bit relocations on 64bit targets?

Thanks

Michal

Hi Florian,

Florian Weimer [off-list ref] writes:

We tried to use Go to build PIE binaries, and while the Go toolchain is
definitely not ready (it produces text relocations and problematic
relocations in general), it exposed what could be an accidental
userspace ABI change.

With our 4.10-derived kernel, PIE binaries are mapped below 4 GiB, so
relocations like R_PPC64_ADDR16_HA work:

21f00000-220d0000 r-xp 00000000 fd:00 36593493                           /root/extld
220d0000-220e0000 r--p 001c0000 fd:00 36593493                           /root/extld
220e0000-22100000 rw-p 001d0000 fd:00 36593493                           /root/extld

...

With a 4.18-derived kernel (with the hashed mm), we get this instead:

120e60000-121030000 rw-p 00000000 fd:00 102447141                        /root/extld
121030000-121060000 rw-p 001c0000 fd:00 102447141                        /root/extld
121060000-121080000 rw-p 00000000 00:00 0 

I assume that's caused by:

  47ebb09d5485 ("powerpc: move ELF_ET_DYN_BASE to 4GB / 4MB")

Which did roughly:

  -#define ELF_ET_DYN_BASE	0x20000000
  +#define ELF_ET_DYN_BASE		(is_32bit_task() ? 0x000400000UL : \
  +					   0x100000000UL)

And went into 4.13.

...
I'm not entirely sure what to make of this, but I'm worried that this
could be a regression that matters to userspace.

It was a deliberate change, and it seemed to not break anything so we
merged it. But obviously we didn't test widely enough.

So I guess it clearly can matter to userspace, and it used to work, so
therefore it is a regression.

But at the same time we haven't had any other reports of breakage, so is
this somehow specific to something Go is doing? Or did we just get lucky
up until now? Or is no one actually testing on Power? ;)

cheers

On Thu, Nov 01, 2018 at 02:55:34PM +1100, Michael Ellerman wrote:

Hi Florian,

Florian Weimer [off-list ref] writes:

quoted

We tried to use Go to build PIE binaries, and while the Go toolchain is
definitely not ready (it produces text relocations and problematic
relocations in general), it exposed what could be an accidental
userspace ABI change.

With our 4.10-derived kernel, PIE binaries are mapped below 4 GiB, so
relocations like R_PPC64_ADDR16_HA work:

21f00000-220d0000 r-xp 00000000 fd:00 36593493                           /root/extld
220d0000-220e0000 r--p 001c0000 fd:00 36593493                           /root/extld
220e0000-22100000 rw-p 001d0000 fd:00 36593493                           /root/extld

...

quoted

With a 4.18-derived kernel (with the hashed mm), we get this instead:

120e60000-121030000 rw-p 00000000 fd:00 102447141                        /root/extld
121030000-121060000 rw-p 001c0000 fd:00 102447141                        /root/extld
121060000-121080000 rw-p 00000000 00:00 0 

I assume that's caused by:

  47ebb09d5485 ("powerpc: move ELF_ET_DYN_BASE to 4GB / 4MB")

Which did roughly:

  -#define ELF_ET_DYN_BASE	0x20000000
  +#define ELF_ET_DYN_BASE		(is_32bit_task() ? 0x000400000UL : \
  +					   0x100000000UL)

And went into 4.13.

quoted

...
I'm not entirely sure what to make of this, but I'm worried that this
could be a regression that matters to userspace.

It was a deliberate change, and it seemed to not break anything so we
merged it. But obviously we didn't test widely enough.

So I guess it clearly can matter to userspace, and it used to work, so
therefore it is a regression.

But at the same time we haven't had any other reports of breakage, so is
this somehow specific to something Go is doing? Or did we just get lucky
up until now? Or is no one actually testing on Power? ;)

Mapping PIEs above 4G should be fine.  It works for gcc C and C++
after all.  The problem is that ppc64le Go is generating code not
suitable for a PIE.  Dynamic text relocations are evidence of non-PIC
object files.

Quoting Lynn Boger [off-list ref]:
"When building a pie binary with golang, they should be using
-buildmode=pie and not just pass -pie to the linker".

-- 
Alan Modra
Australia Development Lab, IBM

Alan Modra [off-list ref] writes:

On Thu, Nov 01, 2018 at 02:55:34PM +1100, Michael Ellerman wrote:

quoted

Hi Florian,

Florian Weimer [off-list ref] writes:

quoted

We tried to use Go to build PIE binaries, and while the Go toolchain is
definitely not ready (it produces text relocations and problematic
relocations in general), it exposed what could be an accidental
userspace ABI change.

With our 4.10-derived kernel, PIE binaries are mapped below 4 GiB, so
relocations like R_PPC64_ADDR16_HA work:

21f00000-220d0000 r-xp 00000000 fd:00 36593493                           /root/extld
220d0000-220e0000 r--p 001c0000 fd:00 36593493                           /root/extld
220e0000-22100000 rw-p 001d0000 fd:00 36593493                           /root/extld

...

quoted

With a 4.18-derived kernel (with the hashed mm), we get this instead:

120e60000-121030000 rw-p 00000000 fd:00 102447141                        /root/extld
121030000-121060000 rw-p 001c0000 fd:00 102447141                        /root/extld
121060000-121080000 rw-p 00000000 00:00 0 

I assume that's caused by:

  47ebb09d5485 ("powerpc: move ELF_ET_DYN_BASE to 4GB / 4MB")

Which did roughly:

  -#define ELF_ET_DYN_BASE	0x20000000
  +#define ELF_ET_DYN_BASE		(is_32bit_task() ? 0x000400000UL : \
  +					   0x100000000UL)

And went into 4.13.

quoted

...
I'm not entirely sure what to make of this, but I'm worried that this
could be a regression that matters to userspace.

It was a deliberate change, and it seemed to not break anything so we
merged it. But obviously we didn't test widely enough.

So I guess it clearly can matter to userspace, and it used to work, so
therefore it is a regression.

But at the same time we haven't had any other reports of breakage, so is
this somehow specific to something Go is doing? Or did we just get lucky
up until now? Or is no one actually testing on Power? ;)

Mapping PIEs above 4G should be fine.  It works for gcc C and C++
after all.  The problem is that ppc64le Go is generating code not
suitable for a PIE.  Dynamic text relocations are evidence of non-PIC
object files.

Quoting Lynn Boger [off-list ref]:
"When building a pie binary with golang, they should be using
-buildmode=pie and not just pass -pie to the linker".

Thanks Alan.

So this isn't a kernel bug per se, but the the old behaviour falls in
the category of "shouldn't have worked but did by accident", and so the
question is just how wide spread is the userspace breakage.

At least so far it seems not very wide spread, so we'll leave things as
they are for now. As Florian said we can always add a personality flag
in future if we need to.

cheers