From: Andy Lutomirski <luto@kernel.org> Date: 2018-12-11 01:23:54
Hi all-
I'm seriously considering sending a patch to remove x32 support from
upstream Linux. Here are some problems with it:
1. It's not entirely clear that it has users. As far as I know, it's
supported on Gentoo and Debian, and the Debian popcon graph for x32
has been falling off dramatically. I don't think that any enterprise
distro has ever supported x32.
2. The way that system calls work is very strange. Most syscalls on
x32 enter through their *native* (i.e. not COMPAT_SYSCALL_DEFINE)
entry point, and this is intentional. For example, adjtimex() uses
the native entry, not the compat entry, because x32's struct timex
matches the x86_64 layout. But a handful of syscalls have separate
entry points -- these are the syscalls starting at 512. These enter
through the COMPAT_SYSCALL_DEFINE entry points.
The x32 syscalls that are *not* in the 512 range violate all semblance
of kernel syscall convention. In the syscall handlers,
in_compat_syscall() returns true, but the COMPAT_SYSCALL_DEFINE entry
is not invoked. This is nutty and risks breaking things when people
refactor their syscall implementations. And no one tests these
things. Similarly, if someone calls any of the syscalls below 512 but
sets bit 31 in RAX, then the native entry will be called with
in_compat_set().
Conversely, if you call a syscall in the 512 range with bit 31
*clear*, then the compat entry is set with in_compat_syscall()
*clear*. This is also nutty.
Finally, the kernel has a weird distinction between CONFIG_X86_X32_ABI
and and CONFIG_X86_X32, which I suspect results in incorrect builds if
the host doesn't have an x32 toolchain installed.
I propose that we make CONFIG_X86_X32 depend on BROKEN for a release
or two and then remove all the code if no one complains. If anyone
wants to re-add it, IMO they're welcome to do so, but they need to do
it in a way that is maintainable.
--Andy
On Mon, Dec 10, 2018 at 5:23 PM Andy Lutomirski [off-list ref] wrote:
I'm seriously considering sending a patch to remove x32 support from
upstream Linux. Here are some problems with it:
I talked to Arnd (I think - we were talking about all the crazy ABI's,
but maybe it was with somebody else) about exactly this in Edinburgh.
Apparently the main real use case is for extreme benchmarking. It's
the only use-case where the complexity of maintaining a whole
development environment and distro is worth it, it seems. Apparently a
number of Spec submissions have been done with the x32 model.
I'm not opposed to trying to sunset the support, but let's see who complains..
Linus
On December 10, 2018 5:40:33 PM PST, Linus Torvalds [off-list ref] wrote:
On Mon, Dec 10, 2018 at 5:23 PM Andy Lutomirski [off-list ref]
wrote:
quoted
I'm seriously considering sending a patch to remove x32 support from
upstream Linux. Here are some problems with it:
I talked to Arnd (I think - we were talking about all the crazy ABI's,
but maybe it was with somebody else) about exactly this in Edinburgh.
Apparently the main real use case is for extreme benchmarking. It's
the only use-case where the complexity of maintaining a whole
development environment and distro is worth it, it seems. Apparently a
number of Spec submissions have been done with the x32 model.
I'm not opposed to trying to sunset the support, but let's see who
complains..
Linus
The use case aside, I need to address the technical issues in this post; some of the behaviors that Andy is pointing out area quite intentional, even if they are perhaps somewhat confusing at first glance. That being said, some were due to tradeoffs that might have been wrong.
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
On Mon, Dec 10, 2018 at 5:23 PM Andy Lutomirski [off-list ref] wrote:
Hi all-
I'm seriously considering sending a patch to remove x32 support from
upstream Linux. Here are some problems with it:
1. It's not entirely clear that it has users. As far as I know, it's
supported on Gentoo and Debian, and the Debian popcon graph for x32
has been falling off dramatically. I don't think that any enterprise
distro has ever supported x32.
2. The way that system calls work is very strange. Most syscalls on
x32 enter through their *native* (i.e. not COMPAT_SYSCALL_DEFINE)
entry point, and this is intentional. For example, adjtimex() uses
the native entry, not the compat entry, because x32's struct timex
matches the x86_64 layout. But a handful of syscalls have separate
This becomes less an issue with 64-bit time_t.
entry points -- these are the syscalls starting at 512. These enter
throuh the COMPAT_SYSCALL_DEFINE entry points.
The x32 syscalls that are *not* in the 512 range violate all semblance
of kernel syscall convention. In the syscall handlers,
in_compat_syscall() returns true, but the COMPAT_SYSCALL_DEFINE entry
is not invoked. This is nutty and risks breaking things when people
refactor their syscall implementations. And no one tests these
things. Similarly, if someone calls any of the syscalls below 512 but
sets bit 31 in RAX, then the native entry will be called with
in_compat_set().
Conversely, if you call a syscall in the 512 range with bit 31
*clear*, then the compat entry is set with in_compat_syscall()
*clear*. This is also nutty.
This is to share syscalls between LP64 and ILP32 (x32) in x86-64 kernel.
Finally, the kernel has a weird distinction between CONFIG_X86_X32_ABI
and and CONFIG_X86_X32, which I suspect results in incorrect builds if
the host doesn't have an x32 toolchain installed.
X86-64 binutils and GCC can be used to build x86-64 kernel with x32 support.
I propose that we make CONFIG_X86_X32 depend on BROKEN for a release
or two and then remove all the code if no one complains. If anyone
wants to re-add it, IMO they're welcome to do so, but they need to do
it in a way that is maintainable.
From: Andy Lutomirski <luto@kernel.org> Date: 2018-12-11 05:35:19
On Mon, Dec 10, 2018 at 7:15 PM H.J. Lu [off-list ref] wrote:
On Mon, Dec 10, 2018 at 5:23 PM Andy Lutomirski [off-list ref] wrote:
quoted
Hi all-
I'm seriously considering sending a patch to remove x32 support from
upstream Linux. Here are some problems with it:
1. It's not entirely clear that it has users. As far as I know, it's
supported on Gentoo and Debian, and the Debian popcon graph for x32
has been falling off dramatically. I don't think that any enterprise
distro has ever supported x32.
Right. My question wasn't whether x32 had developers -- it was
whether it had users. If the only users are a small handful of people
who keep the toolchain and working and some people who benchmark it,
then I think the case for keeping it in upstream Linux is a bit weak.
quoted
2. The way that system calls work is very strange. Most syscalls on
x32 enter through their *native* (i.e. not COMPAT_SYSCALL_DEFINE)
entry point, and this is intentional. For example, adjtimex() uses
the native entry, not the compat entry, because x32's struct timex
matches the x86_64 layout. But a handful of syscalls have separate
This becomes less an issue with 64-bit time_t.
quoted
entry points -- these are the syscalls starting at 512. These enter
throuh the COMPAT_SYSCALL_DEFINE entry points.
The x32 syscalls that are *not* in the 512 range violate all semblance
of kernel syscall convention. In the syscall handlers,
in_compat_syscall() returns true, but the COMPAT_SYSCALL_DEFINE entry
is not invoked. This is nutty and risks breaking things when people
refactor their syscall implementations. And no one tests these
things. Similarly, if someone calls any of the syscalls below 512 but
sets bit 31 in RAX, then the native entry will be called with
in_compat_set().
Conversely, if you call a syscall in the 512 range with bit 31
*clear*, then the compat entry is set with in_compat_syscall()
*clear*. This is also nutty.
This is to share syscalls between LP64 and ILP32 (x32) in x86-64 kernel.
I tried to understand what's going on. As far as I can tell, most of
the magic is the fact that __kernel_long_t and __kernel_ulong_t are
64-bit as seen by x32 user code. This means that a decent number of
uapi structures are the same on x32 and x86_64. Syscalls that only
use structures like this should route to the x86_64 entry points. But
the implementation is still highly dubious -- in_compat_syscall() will
be *true* in such system calls, which means that, if someone changes:
SYSCALL_DEFINE1(some_func, struct some_struct __user *, ptr)
{
/* x32 goes here, but it's entirely non-obvious unless you read the
x86 syscall table */
native impl;
}
COMPAT_SYSCALL_DEFINE1(some_func, struct compat_some_struct __user *, ptr)
{
compat impl;
}
to the Obviously Equivalent (tm):
SYSCALL_DEFINE1(some_func, struct some_struct __user *, ptr)
{
struct some_struct kernel_val;
if (in_compat_syscall()) {
get_compat_some_struct(&kernel_val, ptr);
} else {
copy_from_user(&kernel_val, ptr, sizeof(struct some_struct));
}
do the work;
}
then x32 breaks.
And I don't even know how x32 is supposed to support some hypothetical
syscall like this:
long sys_nasty(struct adjtimex *a, struct iovec *b);
where one argument has x32 and x86_64 matching but the other has x32
and x86_32 matching.
This whole thing seems extremely fragile.
From: Christian Brauner <hidden> Date: 2018-12-11 05:46:23
On Mon, Dec 10, 2018 at 05:23:39PM -0800, Andy Lutomirski wrote:
Hi all-
I'm seriously considering sending a patch to remove x32 support from
upstream Linux. Here are some problems with it:
1. It's not entirely clear that it has users. As far as I know, it's
supported on Gentoo and Debian, and the Debian popcon graph for x32
has been falling off dramatically. I don't think that any enterprise
distro has ever supported x32.
2. The way that system calls work is very strange. Most syscalls on
x32 enter through their *native* (i.e. not COMPAT_SYSCALL_DEFINE)
entry point, and this is intentional. For example, adjtimex() uses
the native entry, not the compat entry, because x32's struct timex
matches the x86_64 layout. But a handful of syscalls have separate
entry points -- these are the syscalls starting at 512. These enter
through the COMPAT_SYSCALL_DEFINE entry points.
The x32 syscalls that are *not* in the 512 range violate all semblance
of kernel syscall convention. In the syscall handlers,
in_compat_syscall() returns true, but the COMPAT_SYSCALL_DEFINE entry
is not invoked. This is nutty and risks breaking things when people
refactor their syscall implementations. And no one tests these
things. Similarly, if someone calls any of the syscalls below 512 but
sets bit 31 in RAX, then the native entry will be called with
in_compat_set().
Conversely, if you call a syscall in the 512 range with bit 31
*clear*, then the compat entry is set with in_compat_syscall()
*clear*. This is also nutty.
Finally, the kernel has a weird distinction between CONFIG_X86_X32_ABI
and and CONFIG_X86_X32, which I suspect results in incorrect builds if
the host doesn't have an x32 toolchain installed.
I propose that we make CONFIG_X86_X32 depend on BROKEN for a release
or two and then remove all the code if no one complains. If anyone
Based on the discussion we had at the beginning of the pidfd_send_signal
syscall patchset I think this is a good idea. For once, the complex
compat handling can make adding new syscalls that need to rely on compat
types because of precedent established by older syscalls icky.
wants to re-add it, IMO they're welcome to do so, but they need to do
it in a way that is maintainable.
--Andy
On Tue, Dec 11, 2018 at 6:35 AM Andy Lutomirski [off-list ref] wrote:
On Mon, Dec 10, 2018 at 7:15 PM H.J. Lu [off-list ref] wrote:
quoted
On Mon, Dec 10, 2018 at 5:23 PM Andy Lutomirski [off-list ref] wrote:
Right. My question wasn't whether x32 had developers -- it was
whether it had users. If the only users are a small handful of people
who keep the toolchain and working and some people who benchmark it,
then I think the case for keeping it in upstream Linux is a bit weak.
+1
quoted
quoted
Conversely, if you call a syscall in the 512 range with bit 31
*clear*, then the compat entry is set with in_compat_syscall()
*clear*. This is also nutty.
This is to share syscalls between LP64 and ILP32 (x32) in x86-64 kernel.
I tried to understand what's going on. As far as I can tell, most of
the magic is the fact that __kernel_long_t and __kernel_ulong_t are
64-bit as seen by x32 user code. This means that a decent number of
uapi structures are the same on x32 and x86_64. Syscalls that only
use structures like this should route to the x86_64 entry points. But
the implementation is still highly dubious -- in_compat_syscall() will
be *true* in such system calls,
I think the fundamental issue was that the intention had always been
to use only the 64-bit entry points for system calls, but the most
complex one we have -- ioctl() -- has to use the compat entry point
because device drivers define their own data structures using 'long'
and pointer members and they need translation, as well as
matching in_compat_syscall() checks. This in turn breaks down
again whenever a driver defines an ioctl command that takes
a __kernel_long_t or a derived type like timespec as its argument.
which means that, if someone changes:
...
where one argument has x32 and x86_64 matching but the other has x32
and x86_32 matching.
This whole thing seems extremely fragile.
It definitely is. We have lots of workarounds specifically for
x32 in device drivers, but in the time_t conversion for y2038
I still found ones that had not been caught earlier, and for each
y2038 conversion that someone did to a driver or syscall, we have
to make sure that it doesn't break x32 in the process.
Arnd
From: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> Date: 2018-12-11 10:29:45
Hello!
I'm Debian's principal maintainer of the x32 port.
On 12/11/18 2:23 AM, Andy Lutomirski wrote:
1. It's not entirely clear that it has users. As far as I know, it's
supported on Gentoo and Debian, and the Debian popcon graph for x32
has been falling off dramatically. I don't think that any enterprise
distro has ever supported x32.
There are definitely some users of this port. I don't know the actual
number, but I hear from users from time to time. As for the popcon
curve, I wouldn't say it has dropped dramatically as it was never
high in the first place.
It seems that the highest number of recorded users was 18 and it's
now down to 7. Keep in mind though that popcon participation is
opt-in, so the actual number of users should be higher. According
to popcon, there are also only 172331 Debian installations on x86_64:
As for the enterprise support, this seems to be correct. I don't know
of any enterprise distribution with x32 support either.
2. The way that system calls work is very strange. Most syscalls on
x32 enter through their *native* (i.e. not COMPAT_SYSCALL_DEFINE)
entry point, and this is intentional. For example, adjtimex() uses
the native entry, not the compat entry, because x32's struct timex
matches the x86_64 layout. But a handful of syscalls have separate
entry points -- these are the syscalls starting at 512. These enter
through the COMPAT_SYSCALL_DEFINE entry points.
The x32 syscalls that are *not* in the 512 range violate all semblance
of kernel syscall convention. In the syscall handlers,
in_compat_syscall() returns true, but the COMPAT_SYSCALL_DEFINE entry
is not invoked. This is nutty and risks breaking things when people
refactor their syscall implementations. And no one tests these
things. Similarly, if someone calls any of the syscalls below 512 but
sets bit 31 in RAX, then the native entry will be called with
in_compat_set().
Conversely, if you call a syscall in the 512 range with bit 31
*clear*, then the compat entry is set with in_compat_syscall()
*clear*. This is also nutty.
I can't say anything about the syscall interface. However, what I do know
is that the weird combination of a 32-bit userland with a 64-bit kernel
interface is sometimes causing issues. For example, application code usually
expects things like time_t to be 32-bit on a 32-bit system. However, this
isn't the case for x32 which is why code fails to build.
Additionally, x32 support in many applications is either rudimentary
or broken. For example, while LLVM has support for x32, the backend
isn't really stable on this target meaning that compilers like clang
or Rust are partially broken or crash. I'm not sure whether anyone
is interested in fixing this.
It's also that the performance benefits of x32 are often eaten up by
the fact that none of the scripted languages that I know of provide
a JIT that supports x32. Thus, things like Javascript are either
unsupported or slow on x32.
I propose that we make CONFIG_X86_X32 depend on BROKEN for a release
or two and then remove all the code if no one complains. If anyone
wants to re-add it, IMO they're welcome to do so, but they need to do
it in a way that is maintainable.
I'm not terribly opposed to this change. I'm usually for keeping support
for things that people are using, but the maintenance is a huge burden
to upstream projects, I'm fine with letting it go.
There are other architectures in the kernel like Alpha, HPPA, M68K, PowerPC,
SH and SPARC that I care much more about than x32.
If x32 is eventually to be removed, we should also take care of removing
x32 support from userland code. From the top of my head, this would at least
concern:
* OpenJDK
* LLVM
* gcc
* glibc
* Rust
* binutils
I can take care of these once I know about the decision regarding the kernel.
Usually, it's a matter of grepping the commit history for "x32" and revert
the corresponding commits.
Thanks,
Adrian
--
.''`. John Paul Adrian Glaubitz
: :' : Debian Developer - glaubitz@debian.org
`. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
`- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913
On Tue, Dec 11, 2018 at 10:02:45AM +0100, Arnd Bergmann wrote:
On Tue, Dec 11, 2018 at 6:35 AM Andy Lutomirski [off-list ref] wrote:
quoted
I tried to understand what's going on. As far as I can tell, most of
the magic is the fact that __kernel_long_t and __kernel_ulong_t are
64-bit as seen by x32 user code. This means that a decent number of
uapi structures are the same on x32 and x86_64. Syscalls that only
use structures like this should route to the x86_64 entry points. But
the implementation is still highly dubious -- in_compat_syscall() will
be *true* in such system calls,
I think the fundamental issue was that the intention had always been
to use only the 64-bit entry points for system calls, but the most
complex one we have -- ioctl() -- has to use the compat entry point
because device drivers define their own data structures using 'long'
and pointer members and they need translation, as well as
matching in_compat_syscall() checks. This in turn breaks down
again whenever a driver defines an ioctl command that takes
a __kernel_long_t or a derived type like timespec as its argument.
With arm64 ILP32 we tried to avoid the ioctl() problem by having
__kernel_long_t 32-bit, IOW mimicking the arm32 ABI (compat). The
biggest pain point is signals where the state is completely different
from arm32 (more, wider registers) and can't be dealt with by the compat
layer.
Fortunately, we haven't merge it yet as we have the same dilemma about
real users and who's going to regularly test the ABI in the long run. In
the meantime, watching this thread with interest ;).
--
Catalin
Note: please keep me in Cc because I am not subscribed.
Linus Torvalds dixit:
I'm not opposed to trying to sunset the support, but let's see who complains..
I will hereby complain.
I’m using Debian/x32 on my main desktop at work, and do
occasionally help out with porting issues. It’s a good
way to make more out of 64-bit machines without going
all 64 bit; it’s also helped me find bugs in software.
It’s a nice architectural idea, and a way forward for
things that are constricted to 32 bits while opening
up stuff like 64-bit time_t without taking up half the
available CPU registers (while more than doubling the
number of the available CPU registers, too).
I was also considering investing a nontrivial amount of
work into porting klibc to x32, since hpa does not wish
to do it himself. Thankfully I have only done a bit yet.
Furthermore, x32 was the first of the many *64ilp32
architectures; I know I’ve seen amd64ilp32 and at least
one other I don’t recall. It will have prototyped many
of the problems users of these will run in, and I’d prefer
to keep it (completely selfish because I don’t wish to
have to crossgrade a whole system yet again).
Thanks for your consideration,
//mirabilos
--
When he found out that the m68k port was in a pretty bad shape, he did
not, like many before him, shrug and move on; instead, he took it upon
himself to start compiling things, just so he could compile his shell.
How's that for dedication. -- Wouter, about my Debian/m68k revival
I can't say anything about the syscall interface. However, what I do know
is that the weird combination of a 32-bit userland with a 64-bit kernel
interface is sometimes causing issues. For example, application code usually
Yes, but more and more ${foo}64ilp32 architectures are popping up.
Additionally, x32 support in many applications is either rudimentary
If a signal is sent that this kind of architectures will stay, some
people might be convinced to fix that.
It's also that the performance benefits of x32 are often eaten up by
the fact that none of the scripted languages that I know of provide
Non-JITted languages like yours truly’s shell do benefit from it,
though. (mksh works just fine on LP64 but its internal structures
pack massively better on ILP32, for example.)
If x32 is eventually to be removed, we should also take care of removing
x32 support from userland code. From the top of my head, this would at least
I don’t think so. The patches also contain
– stuff to support 64-bit time_t on 32-bit architectures, e.g:
- bugfixes like printf("%lld", (long long)timet_value) instead
of assuming time_t fits into a long (also important for other
operating systems…)
- generally switching from generic types like long to specific
types like size_t, ptrdiff_t, etc.
- there was one more but after having written two eMails I forgot it
- oh and, of course, they lay the base for e.g. amd64ilp32 support
bye,
//mirabilos
--
FWIW, I'm quite impressed with mksh interactively. I thought it was much
*much* more bare bones. But it turns out it beats the living hell out of
ksh93 in that respect. I'd even consider it for my daily use if I hadn't
wasted half my life on my zsh setup. :-) -- Frank Terbeck in #!/bin/mksh
From: Andy Lutomirski <luto@kernel.org> Date: 2018-12-11 23:23:01
On Tue, Dec 11, 2018 at 2:14 PM Thorsten Glaser [off-list ref] wrote:
Note: please keep me in Cc because I am not subscribed.
Linus Torvalds dixit:
quoted
I'm not opposed to trying to sunset the support, but let's see who complains..
I will hereby complain.
I’m using Debian/x32 on my main desktop at work, and do
occasionally help out with porting issues. It’s a good
way to make more out of 64-bit machines without going
all 64 bit; it’s also helped me find bugs in software.
It’s a nice architectural idea, and a way forward for
things that are constricted to 32 bits while opening
up stuff like 64-bit time_t without taking up half the
available CPU registers (while more than doubling the
number of the available CPU registers, too).
Thanks for responding!
I suppose the question is: are you enough of a user to justify the
continued maintenance effort.
I was also considering investing a nontrivial amount of
work into porting klibc to x32, since hpa does not wish
to do it himself. Thankfully I have only done a bit yet.
Furthermore, x32 was the first of the many *64ilp32
architectures; I know I’ve seen amd64ilp32 and at least
one other I don’t recall. It will have prototyped many
of the problems users of these will run in, and I’d prefer
to keep it (completely selfish because I don’t wish to
have to crossgrade a whole system yet again).
it kind of seems like arm64's lesson is "don't do it like x32".
There's some effort going on right now to make it possible to add
syscalls without having to muck with every single architecture. I
don't really want x32 to derail that effort. I suppose we could say
that x32 stays but that it simply gets no new syscalls, but that seems
a bit lame. Unfortunately, on x86, x32 really is a third ABI that is
not compatible in a structure-memory-layout sense with the other two.
What happens if someone adds a struct like:
struct nasty_on_x32 {
__kernel_long_t a;
void * __user b;
};
On x86_64, that's two 8-byte fields. On x86_32, it's two four-byte
fields. On x32, it's an 8-byte field and a 4-byte field. Now what?
I'm sure we could have some magic gcc plugin or other nifty tool that gives us:
copy_from_user(struct struct_name, kernel_ptr, user_ptr);
where it automatically generates code for all possible ABIs to copy
over the struct and dispatches dynamically based on the current
syscall ABI, but I have trouble imagining anyone volunteering to
actually do this work. Instead we get ad hoc fixes for each syscall,
along the lines of preadv64v2(), which get done when somebody notices
a problem.
Linus, any advice here?
--Andy
On Tue, Dec 11, 2018 at 09:59:48PM +0000, Thorsten Glaser wrote:
John Paul Adrian Glaubitz dixit:
quoted
I can't say anything about the syscall interface. However, what I do know
is that the weird combination of a 32-bit userland with a 64-bit kernel
interface is sometimes causing issues. For example, application code usually
Yes, but more and more ${foo}64ilp32 architectures are popping up.
quoted
Additionally, x32 support in many applications is either rudimentary
If a signal is sent that this kind of architectures will stay, some
people might be convinced to fix that.
quoted
It's also that the performance benefits of x32 are often eaten up by
the fact that none of the scripted languages that I know of provide
Non-JITted languages like yours truly’s shell do benefit from it,
though. (mksh works just fine on LP64 but its internal structures
pack massively better on ILP32, for example.)
gcc also benefits massively, up to 2x or more performance improvement
from ILP32, just because you can double your -jN to make. The "or
more" comes from cache utilization also improving. Actually I often
use i486 gcc even on 64-bit systems for this reason, and have observed
this order of improvement despite the lack of registers and worse call
ABI. Presumably x32 would be better, but I've avoided it because of
how buggy it's been in the past.
quoted
If x32 is eventually to be removed, we should also take care of removing
x32 support from userland code. From the top of my head, this would at least
I don’t think so. The patches also contain
– stuff to support 64-bit time_t on 32-bit architectures, e.g:
- bugfixes like printf("%lld", (long long)timet_value) instead
of assuming time_t fits into a long (also important for other
operating systems…)
- generally switching from generic types like long to specific
types like size_t, ptrdiff_t, etc.
- there was one more but after having written two eMails I forgot it
- oh and, of course, they lay the base for e.g. amd64ilp32 support
I would not classify any of that as part of "removing x32 support",
but rather as "reintroducing broken assumptions". I think "removing
x32 support" was intended to mean places where there's
explicitly-x32-specific code or tooling, like asm functions/fragments
in libc, x32 versions of the dispatch asm in mesa, x32 code
generation, etc. Obviously (at least I hope it's obvious) nonportable
assumptions should not be reintroduced regardless of whether x32 is
dropped.
FWIW x32 is nice as a test case for catching such nonportable
assumptions and getting them fixed.
Rich
On Tue, Dec 11, 2018 at 03:22:43PM -0800, Andy Lutomirski wrote:
On Tue, Dec 11, 2018 at 2:14 PM Thorsten Glaser [off-list ref] wrote:
quoted
I was also considering investing a nontrivial amount of
work into porting klibc to x32, since hpa does not wish
to do it himself. Thankfully I have only done a bit yet.
Furthermore, x32 was the first of the many *64ilp32
architectures; I know I’ve seen amd64ilp32 and at least
one other I don’t recall. It will have prototyped many
of the problems users of these will run in, and I’d prefer
to keep it (completely selfish because I don’t wish to
have to crossgrade a whole system yet again).
it kind of seems like arm64's lesson is "don't do it like x32".
There's some effort going on right now to make it possible to add
syscalls without having to muck with every single architecture. I
don't really want x32 to derail that effort. I suppose we could say
that x32 stays but that it simply gets no new syscalls, but that seems
a bit lame. Unfortunately, on x86, x32 really is a third ABI that is
not compatible in a structure-memory-layout sense with the other two.
What happens if someone adds a struct like:
struct nasty_on_x32 {
__kernel_long_t a;
void * __user b;
};
On x86_64, that's two 8-byte fields. On x86_32, it's two four-byte
fields. On x32, it's an 8-byte field and a 4-byte field. Now what?
I'm sure we could have some magic gcc plugin or other nifty tool that gives us:
copy_from_user(struct struct_name, kernel_ptr, user_ptr);
where it automatically generates code for all possible ABIs to copy
over the struct and dispatches dynamically based on the current
syscall ABI, but I have trouble imagining anyone volunteering to
actually do this work. Instead we get ad hoc fixes for each syscall,
along the lines of preadv64v2(), which get done when somebody notices
a problem.
For what it's worth, this is exactly what I plan to do if/when I ever
get to doing my kernel with multiple/universal userspace ABI support:
pure-data markup for the data structures (and syscall argument
arrangements) userspace is speaking, with generic marshalling code.
But I don't think it's easy to retrofit to Linux, and probably
wouldn't be welcome there.
Rich
What happens if someone adds a struct like:
struct nasty_on_x32 {
__kernel_long_t a;
void * __user b;
};
On x86_64, that's two 8-byte fields. On x86_32, it's two four-byte
fields. On x32, it's an 8-byte field and a 4-byte field. Now what?
Yes, that’s indeed ugly. I understand. But don’t we already have
this problem with architectures which support multiple ABIs at the
same time? An amd64 kernel with i386 userspace comes to mind, or
the multiple MIPS ABIs.
I'm sure we could have some magic gcc plugin or other nifty tool that
gives us:
copy_from_user(struct struct_name, kernel_ptr, user_ptr);
Something like that might be useful. Generate call stubs, which
then call the syscall implementation with the actual user-space
struct contents as arguments. Hm, that might be too generic to
be useful. Generate macros that can read from or write specific
structures to userspace?
I think something like this could solve other more general problems
as well, so it might be “nice to have anyway”. Of course it’s work,
and I’m not involved enough in Linux kernel programming to be able
to usefully help with it (doing too much elsewhere already).
actually do this work. Instead we get ad hoc fixes for each syscall,
along the lines of preadv64v2(), which get done when somebody notices
Yes, that’s absolutely ugly and ridiculous and all kinds of bad.
On the other hand, from my current experience, someone (Arnd?) noticed
all the currently existing baddies for x32 already and fixed them.
New syscalls are indeed an issue, but perhaps something generating
copyinout stubs could help. This might allow other architectures
that could do with a new ABI but have until now feared the overhead
as well. (IIRC, m68k could do with a new ABI that reserves a register
for TLS, but Geert would know. At the same time, time_t and off_t could
be bumped to 64 bit. Something like that. If changing sizes of types
shared between kernel and user spaces is not something feared…)
Thanks for considering,
//mirabilos
--
„Cool, /usr/share/doc/mksh/examples/uhr.gz ist ja ein Grund,
mksh auf jedem System zu installieren.“
-- XTaran auf der OpenRheinRuhr, ganz begeistert
(EN: “[…]uhr.gz is a reason to install mksh on every system.”)
On Wed, Dec 12, 2018 at 12:38 AM Thorsten Glaser [off-list ref] wrote:
Andy Lutomirski dixit:
quoted
What happens if someone adds a struct like:
struct nasty_on_x32 {
__kernel_long_t a;
void * __user b;
};
On x86_64, that's two 8-byte fields. On x86_32, it's two four-byte
fields. On x32, it's an 8-byte field and a 4-byte field. Now what?
Yes, that’s indeed ugly. I understand. But don’t we already have
this problem with architectures which support multiple ABIs at the
same time? An amd64 kernel with i386 userspace comes to mind, or
the multiple MIPS ABIs.
The main trouble that I see with x32 is in how it's different
from all the other compat architectures. We have a number
of drivers that support not only two ABIs (native and compat)
but also a third one specifically for x32, and more drivers
that fail on x32 because they never implemented the third
case.
Note that this is not a problem for MIPS n32 and aarch64ilp32,
which generally stick to the regular compat ABI (each compat
architecture has some minor quirks somewhere, much less so
than x32 does).
quoted
actually do this work. Instead we get ad hoc fixes for each syscall,
along the lines of preadv64v2(), which get done when somebody notices
Yes, that’s absolutely ugly and ridiculous and all kinds of bad.
On the other hand, from my current experience, someone (Arnd?) noticed
all the currently existing baddies for x32 already and fixed them.
I did some, and others did more, but it's hard to ever do a complete
job, and even harder to prevent new bugs from creeping in.
New syscalls are indeed an issue, but perhaps something generating
copyinout stubs could help. This might allow other architectures
that could do with a new ABI but have until now feared the overhead
as well. (IIRC, m68k could do with a new ABI that reserves a register
for TLS, but Geert would know. At the same time, time_t and off_t could
be bumped to 64 bit. Something like that. If changing sizes of types
shared between kernel and user spaces is not something feared…)
Creating a completely new ABI for an existing architecutre is generally
something you want to avoid, as this means all existing user space
has to be recompiled.
All architectures support 64-bit off_t already, and soon will support
64-bit time_t as well, at this point (as of this week's linux-next)
we mostly just need to assign a couple of syscall numbers for each
32-bit architectures.
Arnd
From: Andy Lutomirski <luto@kernel.org> Date: 2018-12-12 02:25:25
On Dec 11, 2018, at 3:35 PM, Thorsten Glaser [off-list ref] wrote:
Andy Lutomirski dixit:
quoted
What happens if someone adds a struct like:
struct nasty_on_x32 {
__kernel_long_t a;
void * __user b;
};
On x86_64, that's two 8-byte fields. On x86_32, it's two four-byte
fields. On x32, it's an 8-byte field and a 4-byte field. Now what?
Yes, that’s indeed ugly. I understand. But don’t we already have
this problem with architectures which support multiple ABIs at the
same time? An amd64 kernel with i386 userspace comes to mind, or
the multiple MIPS ABIs.
That’s the thing, though: the whole generic kernel compat
infrastructure assumes there are at most two ABIs: native and, if
enabled and relevant, compat. x32 breaks this entirely.
quoted
I'm sure we could have some magic gcc plugin or other nifty tool that
gives us:
copy_from_user(struct struct_name, kernel_ptr, user_ptr);
Something like that might be useful. Generate call stubs, which
then call the syscall implementation with the actual user-space
struct contents as arguments. Hm, that might be too generic to
be useful. Generate macros that can read from or write specific
structures to userspace?
I think something like this could solve other more general problems
as well, so it might be “nice to have anyway”. Of course it’s work,
and I’m not involved enough in Linux kernel programming to be able
to usefully help with it (doing too much elsewhere already).
quoted
actually do this work. Instead we get ad hoc fixes for each syscall,
along the lines of preadv64v2(), which get done when somebody notices
Yes, that’s absolutely ugly and ridiculous and all kinds of bad.
On the other hand, from my current experience, someone (Arnd?) noticed
all the currently existing baddies for x32 already and fixed them.
New syscalls are indeed an issue, but perhaps something generating
copyinout stubs could help. This might allow other architectures
that could do with a new ABI but have until now feared the overhead
as well. (IIRC, m68k could do with a new ABI that reserves a register
for TLS, but Geert would know. At the same time, time_t and off_t could
be bumped to 64 bit. Something like that. If changing sizes of types
shared between kernel and user spaces is not something feared…)
Magic autogenerated stubs would be great. Difficult, too, given
unions, multiplexers, cmsg, etc.
I suppose I will see how bad it would be to split out the x32 syscall
table and at least isolate the mess to some extent.
IMO the real right solution would be to push the whole problem to
userspace: get an ILP32 system working with almost or entirely LP64
syscalls. POSIX support might have to be a bit flexible, but still.
How hard would it be to have __attribute__((ilp64)), with an optional
warning if any embedded structs are not ilp64? This plus a wrapper to
make sure that mmap puts everything below 4GB ought to do the trick.
Or something like what arm64 is proposing where the kernel ABI has
32-bit long doesn’t seem too horrible.
That’s the thing, though: the whole generic kernel compat
infrastructure assumes there are at most two ABIs: native and, if
enabled and relevant, compat. x32 breaks this entirely.
MIPS had o32, n32, n64 since like forever.
ARM has old ABI, EABI and now 64-bit.
Other architectures are yet to come.
IMO the real right solution would be to push the whole problem to
userspace: get an ILP32 system working with almost or entirely LP64
Is this a reflex of Linux kernel developers? ;-)
I doubt that userspace is the right place for this, remember
the recent glibc vs. syscalls debate. It would also need to
multiply across various libcs.
How hard would it be to have __attribute__((ilp64)), with an optional
warning if any embedded structs are not ilp64? This plus a wrapper to
You mean LP64. Impossible, because LP64 vs. ILP32 is not the only
difference between amd64 and x32.
bye,
//mirabilos
--
I believe no one can invent an algorithm. One just happens to hit upon it
when God enlightens him. Or only God invents algorithms, we merely copy them.
If you don't believe in God, just consider God as Nature if you won't deny
existence. -- Coywolf Qi Hunt
On Wed, Dec 12, 2018 at 3:38 AM Thorsten Glaser [off-list ref] wrote:
Andy Lutomirski dixit:
quoted
That’s the thing, though: the whole generic kernel compat
infrastructure assumes there are at most two ABIs: native and, if
enabled and relevant, compat. x32 breaks this entirely.
MIPS had o32, n32, n64 since like forever.
o32 and n32 are practically the same, the only difference on the
syscall ABI that I can see are the actual syscall numbers, and
the 'struct sigcontext' definition.
ARM has old ABI, EABI and now 64-bit.
arm64 intentionally did not attempt to support OABI user space
because of this, and as I said the ilp32 ABI follows what MIPS
n32 does using the same data structures as aarch32 (corresponding
to mips o32).
quoted
How hard would it be to have __attribute__((ilp64)), with an optional
warning if any embedded structs are not ilp64? This plus a wrapper to
You mean LP64. Impossible, because LP64 vs. ILP32 is not the only
difference between amd64 and x32.
I think the above is what Intel's compiler does, and similar to what they
do for mixing big-endian and little-endian code (!). Generally possible yes,
but a lot of work, as well as error-prone and not particular appealing for
the GNU toolchain IMHO.
Arnd
From: Andy Lutomirski <luto@kernel.org> Date: 2018-12-12 16:40:11
On Dec 11, 2018, at 6:33 PM, Thorsten Glaser [off-list ref] wrote:
Andy Lutomirski dixit:
quoted
IMO the real right solution would be to push the whole problem to
userspace: get an ILP32 system working with almost or entirely LP64
Is this a reflex of Linux kernel developers? ;-)
I doubt that userspace is the right place for this, remember
the recent glibc vs. syscalls debate. It would also need to
multiply across various libcs.
quoted
How hard would it be to have __attribute__((ilp64)), with an optional
warning if any embedded structs are not ilp64? This plus a wrapper to
You mean LP64. Impossible, because LP64 vs. ILP32 is not the only
difference between amd64 and x32.
I mean LP64. And I'm not suggesting that ILP32 is the only difference
between x32 and x86_64, nor am I suggesting that a technique like this
would implement x32 -- I'm suggesting it would implement something
better than x32.
The kernel, as a practical matter, supports two ABIs on 64-bit builds:
LP64 and ILP32. ILP32 is what the kernel calls "compat". ("compat"
comes with other baggage -- it generally has a 32-bit signal frame,
syscall arguments are mostly limited to 32 bits, etc.) Allowing a
user program that runs in 64-bit mode to issue compat syscalls is not
really a big deal. x86_64 has allowed this forever using int $0x80 --
it's just slow. Adding a faster mechanism would be straightforward.
As I understand it, the arm64 ilp32 proposal involves using a genuine
ILP32 model for user code, so the syscalls will all (except for signal
handling) go through the compat path.
x32 is not this at all. The kernel ABI part of x32 isn't ILP32. It's
IP32, 32-bit size_t, and *64-bit* long. The core kernel doesn't
really support this. The only good things I can think of about it are
that (a) it proves that somewhat odd ABIs are possible, at least in
principle, and (b) three users have come out of the woodwork to say
that they use it.
I'm proposing another alternative. Given that x32 already proves that
the user bitness model doesn't have to match the kernel model (in x32,
user "long" is 32-bit but the kernel ABI "long" is 64-bit), I'm
proposing extending this to just make the kernel ABI be LP64. So
__kernel_size_t would be 64-bit and pointers in kernel data structures
would be 64-bit. In other words, most or all of the kernel ABI would
just match x86_64.
As far as I can tell, the only thing that really needs unusual
toolchain features here is that C doesn't have an extra-wide pointer
type. The kernel headers would need a way to say "this pointer is
still logically a pointer, and user code may assume that it's 32 bits,
but it has 8-byte alignment."
So I guess I'm proposing that people who really like x32 and are
willing to invest effort it in instead invest some effort into getting
something similar to work using the normal x86_64 syscalls. And I'm
hinting to the ARM folks on the thread that they consider this
approach for arm64.
There's an added benefit to my proposal over x32. With my proposal,
an ILP32 program could plausibly call into a true 64-bit C library
using a straightforward FFI translation.
bye,
//mirabilos
--
I believe no one can invent an algorithm. One just happens to hit upon it
when God enlightens him. Or only God invents algorithms, we merely copy them.
If you don't believe in God, just consider God as Nature if you won't deny
existence. -- Coywolf Qi Hunt
On Wed, Dec 12, 2018 at 08:39:53AM -0800, Andy Lutomirski wrote:
quoted
On Dec 11, 2018, at 6:33 PM, Thorsten Glaser [off-list ref] wrote:
Andy Lutomirski dixit:
quoted
quoted
IMO the real right solution would be to push the whole problem to
userspace: get an ILP32 system working with almost or entirely LP64
Is this a reflex of Linux kernel developers? ;-)
I doubt that userspace is the right place for this, remember
the recent glibc vs. syscalls debate. It would also need to
multiply across various libcs.
quoted
How hard would it be to have __attribute__((ilp64)), with an optional
warning if any embedded structs are not ilp64? This plus a wrapper to
You mean LP64. Impossible, because LP64 vs. ILP32 is not the only
difference between amd64 and x32.
I mean LP64. And I'm not suggesting that ILP32 is the only difference
between x32 and x86_64, nor am I suggesting that a technique like this
would implement x32 -- I'm suggesting it would implement something
better than x32.
The kernel, as a practical matter, supports two ABIs on 64-bit builds:
LP64 and ILP32. ILP32 is what the kernel calls "compat". ("compat"
comes with other baggage -- it generally has a 32-bit signal frame,
syscall arguments are mostly limited to 32 bits, etc.) Allowing a
user program that runs in 64-bit mode to issue compat syscalls is not
really a big deal. x86_64 has allowed this forever using int $0x80 --
it's just slow. Adding a faster mechanism would be straightforward.
As I understand it, the arm64 ilp32 proposal involves using a genuine
ILP32 model for user code, so the syscalls will all (except for signal
handling) go through the compat path.
x32 is not this at all. The kernel ABI part of x32 isn't ILP32. It's
IP32, 32-bit size_t, and *64-bit* long. The core kernel doesn't
really support this. The only good things I can think of about it are
that (a) it proves that somewhat odd ABIs are possible, at least in
principle, and (b) three users have come out of the woodwork to say
that they use it.
I'm proposing another alternative. Given that x32 already proves that
the user bitness model doesn't have to match the kernel model (in x32,
user "long" is 32-bit but the kernel ABI "long" is 64-bit), I'm
proposing extending this to just make the kernel ABI be LP64. So
__kernel_size_t would be 64-bit and pointers in kernel data structures
would be 64-bit. In other words, most or all of the kernel ABI would
just match x86_64.
As far as I can tell, the only thing that really needs unusual
toolchain features here is that C doesn't have an extra-wide pointer
type. The kernel headers would need a way to say "this pointer is
still logically a pointer, and user code may assume that it's 32 bits,
but it has 8-byte alignment."
None of this works on the userspace/C side, nor should any attempt be
made to make it work. Types fundamentally cannot have alignments
larger than their size. If you want to make the alignment of some
pointers 8, you have to make their size 8, and then you just have LP64
again if you did it for all pointers.
If on the other hand you tried to make just some pointers "wide
pointers", you'd also be completely breaking the specified API
contracts of standard interfaces. For example in struct iovec's
iov_base, &foo->iov_base is no longer a valid pointer to an object of
type void* that you can pass to interfaces expecting void**. Sloppy
misunderstandings like what you're making now are exactly why x32 is
already broken and buggy (&foo->tv_nsec already has wrong type for
struct timespec foo).
If you wanted to do this and have it work, libc would have to
completely wrap and convert all the types from userspace structures
with the right types matching the specified API contracts to
kernel-interface structures that can be passed to the LP64 syscalls.
This could be done, but it's a big task, and I don't think anyone
wants to do it, especially if it would be yet another new thing that
doesn't exactly match the already-existing x32. The cost-benefit ratio
is just too high.
There's an added benefit to my proposal over x32. With my proposal,
an ILP32 program could plausibly call into a true 64-bit C library
using a straightforward FFI translation.
Unless it's a thin, "pure" library that doesn't need anything from
libc, or needs sufficiently little that it could be satisfied by some
shims, this would necessarily require having two libcs in the same
process, which is not going to work.
Rich
From: Andy Lutomirski <luto@kernel.org> Date: 2018-12-12 18:03:46
On Wed, Dec 12, 2018 at 8:52 AM Rich Felker [off-list ref] wrote:
On Wed, Dec 12, 2018 at 08:39:53AM -0800, Andy Lutomirski wrote:
quoted
quoted
On Dec 11, 2018, at 6:33 PM, Thorsten Glaser [off-list ref] wrote:
Andy Lutomirski dixit:
quoted
quoted
IMO the real right solution would be to push the whole problem to
userspace: get an ILP32 system working with almost or entirely LP64
Is this a reflex of Linux kernel developers? ;-)
I doubt that userspace is the right place for this, remember
the recent glibc vs. syscalls debate. It would also need to
multiply across various libcs.
quoted
How hard would it be to have __attribute__((ilp64)), with an optional
warning if any embedded structs are not ilp64? This plus a wrapper to
You mean LP64. Impossible, because LP64 vs. ILP32 is not the only
difference between amd64 and x32.
I mean LP64. And I'm not suggesting that ILP32 is the only difference
between x32 and x86_64, nor am I suggesting that a technique like this
would implement x32 -- I'm suggesting it would implement something
better than x32.
The kernel, as a practical matter, supports two ABIs on 64-bit builds:
LP64 and ILP32. ILP32 is what the kernel calls "compat". ("compat"
comes with other baggage -- it generally has a 32-bit signal frame,
syscall arguments are mostly limited to 32 bits, etc.) Allowing a
user program that runs in 64-bit mode to issue compat syscalls is not
really a big deal. x86_64 has allowed this forever using int $0x80 --
it's just slow. Adding a faster mechanism would be straightforward.
As I understand it, the arm64 ilp32 proposal involves using a genuine
ILP32 model for user code, so the syscalls will all (except for signal
handling) go through the compat path.
x32 is not this at all. The kernel ABI part of x32 isn't ILP32. It's
IP32, 32-bit size_t, and *64-bit* long. The core kernel doesn't
really support this. The only good things I can think of about it are
that (a) it proves that somewhat odd ABIs are possible, at least in
principle, and (b) three users have come out of the woodwork to say
that they use it.
I'm proposing another alternative. Given that x32 already proves that
the user bitness model doesn't have to match the kernel model (in x32,
user "long" is 32-bit but the kernel ABI "long" is 64-bit), I'm
proposing extending this to just make the kernel ABI be LP64. So
__kernel_size_t would be 64-bit and pointers in kernel data structures
would be 64-bit. In other words, most or all of the kernel ABI would
just match x86_64.
As far as I can tell, the only thing that really needs unusual
toolchain features here is that C doesn't have an extra-wide pointer
type. The kernel headers would need a way to say "this pointer is
still logically a pointer, and user code may assume that it's 32 bits,
but it has 8-byte alignment."
None of this works on the userspace/C side, nor should any attempt be
made to make it work. Types fundamentally cannot have alignments
larger than their size. If you want to make the alignment of some
pointers 8, you have to make their size 8, and then you just have LP64
again if you did it for all pointers.
If on the other hand you tried to make just some pointers "wide
pointers", you'd also be completely breaking the specified API
contracts of standard interfaces. For example in struct iovec's
iov_base, &foo->iov_base is no longer a valid pointer to an object of
type void* that you can pass to interfaces expecting void**. Sloppy
misunderstandings like what you're making now are exactly why x32 is
already broken and buggy (&foo->tv_nsec already has wrong type for
struct timespec foo).
I don't think it's quite that broken. For the struct iovec example,
we currently have:
struct iovec {
void *iov_base; /* Starting address */
size_t iov_len; /* Number of bytes to transfer */
};
we could have, instead: (pardon any whitespace damage)
struct iovec {
void *iov_base; /* Starting address */
uint32_t __pad0;
size_t iov_len; /* Number of bytes to transfer */
uint32_t __pad1;
} __attribute__((aligned(8));
or the same thing but where iov_len is uint64_t. A pointer to
iov_base still works exactly as expected. Something would need to be
done to ensure that the padding is all zeroed, which might be a real
problem.
No one wants to actually type all the macro gunk into the headers to
make this work, but this type of transformation is what I have in mind
when the compiler is asked to handle the headers. Or there could
potentially be a tool that automatically consumes the uapi headers and
spits out modified headers like this.
Realistically, I think a much better model would be to use true ILP32
code, where all the memory layouts in the uapi match i386.
Unless it's a thin, "pure" library that doesn't need anything from
libc, or needs sufficiently little that it could be satisfied by some
shims, this would necessarily require having two libcs in the same
process, which is not going to work.
From: Joseph Myers <hidden> Date: 2018-12-12 18:14:23
On Wed, 12 Dec 2018, Arnd Bergmann wrote:
quoted
MIPS had o32, n32, n64 since like forever.
o32 and n32 are practically the same, the only difference on the
syscall ABI that I can see are the actual syscall numbers, and
the 'struct sigcontext' definition.
And for syscalls that have 64-bit arguments, n32 generally passes those in
a single register (like n64), not pairs of registers (like o32). But,
yes, userspace structure layout for n32 is generally much closer to o32
than to n64.
--
Joseph S. Myers
joseph@codesourcery.com
From: Ivan Ivanov <hidden> Date: 2018-12-12 18:50:41
Please don't drop x86 support from the Linux kernel ! You may not
realize it, but there are plenty of people who care about x86 systems.
I have a good old Pentium 4 based computer which is 32-bit, using it
for some weird Linux experiments + retro gaming, and if I can't run
new Linux kernels there - it would be SAD
ср, 12 дек. 2018 г. в 21:15, Joseph Myers [off-list ref]:
On Wed, 12 Dec 2018, Arnd Bergmann wrote:
quoted
quoted
MIPS had o32, n32, n64 since like forever.
o32 and n32 are practically the same, the only difference on the
syscall ABI that I can see are the actual syscall numbers, and
the 'struct sigcontext' definition.
And for syscalls that have 64-bit arguments, n32 generally passes those in
a single register (like n64), not pairs of registers (like o32). But,
yes, userspace structure layout for n32 is generally much closer to o32
than to n64.
--
Joseph S. Myers
joseph@codesourcery.com
From: Andy Lutomirski <luto@kernel.org> Date: 2018-12-12 19:12:47
On Wed, Dec 12, 2018 at 10:50 AM Ivan Ivanov [off-list ref] wrote:
Please don't drop x86 support from the Linux kernel ! You may not
realize it, but there are plenty of people who care about x86 systems.
I have a good old Pentium 4 based computer which is 32-bit, using it
for some weird Linux experiments + retro gaming, and if I can't run
new Linux kernels there - it would be SAD
Just to clarify: no one is proposing to drop 32-bit hardware support
or normal x86 compatibility support from new kernels.
That being said, you should seriously consider replacing that P4.
Unless you live somewhere with *very* cheap power or you barely use
it, replacing it will pay for itself quite quickly.
From: Ivan Ivanov <hidden> Date: 2018-12-12 19:18:38
My main reason on staying at Pentium 4 is that it does not contain
Intel ME backdoor (it came starting with Core 2 Duo)
ср, 12 дек. 2018 г. в 22:12, Andy Lutomirski [off-list ref]:
On Wed, Dec 12, 2018 at 10:50 AM Ivan Ivanov [off-list ref] wrote:
quoted
Please don't drop x86 support from the Linux kernel ! You may not
realize it, but there are plenty of people who care about x86 systems.
I have a good old Pentium 4 based computer which is 32-bit, using it
for some weird Linux experiments + retro gaming, and if I can't run
new Linux kernels there - it would be SAD
Just to clarify: no one is proposing to drop 32-bit hardware support
or normal x86 compatibility support from new kernels.
That being said, you should seriously consider replacing that P4.
Unless you live somewhere with *very* cheap power or you barely use
it, replacing it will pay for itself quite quickly.
From: Kevin Easton <hidden> Date: 2018-12-13 05:03:19
On Tue, Dec 11, 2018 at 11:29:14AM +0100, John Paul Adrian Glaubitz wrote:
...
I can't say anything about the syscall interface. However, what I do know
is that the weird combination of a 32-bit userland with a 64-bit kernel
interface is sometimes causing issues. For example, application code usually
expects things like time_t to be 32-bit on a 32-bit system. However, this
isn't the case for x32 which is why code fails to build.
OpenBSD and NetBSD both have 64-bit time_t on 32-bit systems and have
had for four or five years at this point.
- Kevin
From: Richard Weinberger <hidden> Date: 2018-12-13 09:05:31
On Thu, Dec 13, 2018 at 6:03 AM Kevin Easton [off-list ref] wrote:
On Tue, Dec 11, 2018 at 11:29:14AM +0100, John Paul Adrian Glaubitz wrote:
...
quoted
I can't say anything about the syscall interface. However, what I do know
is that the weird combination of a 32-bit userland with a 64-bit kernel
interface is sometimes causing issues. For example, application code usually
expects things like time_t to be 32-bit on a 32-bit system. However, this
isn't the case for x32 which is why code fails to build.
OpenBSD and NetBSD both have 64-bit time_t on 32-bit systems and have
had for four or five years at this point.
They can also do flag-day changes and break existing applications, Linux not.
--
Thanks,
//richard
From: Kevin Easton <hidden> Date: 2018-12-13 12:12:20
On Thu, Dec 13, 2018 at 10:05:14AM +0100, Richard Weinberger wrote:
On Thu, Dec 13, 2018 at 6:03 AM Kevin Easton [off-list ref] wrote:
quoted
On Tue, Dec 11, 2018 at 11:29:14AM +0100, John Paul Adrian Glaubitz wrote:
...
quoted
I can't say anything about the syscall interface. However, what I do know
is that the weird combination of a 32-bit userland with a 64-bit kernel
interface is sometimes causing issues. For example, application code usually
expects things like time_t to be 32-bit on a 32-bit system. However, this
isn't the case for x32 which is why code fails to build.
OpenBSD and NetBSD both have 64-bit time_t on 32-bit systems and have
had for four or five years at this point.
They can also do flag-day changes and break existing applications, Linux not.
Sure, but the point is that most widely-used software has probably by
now come in to contact with systems where time_t is bigger than long.
- Kevin
On Wed, Dec 12, 2018 at 10:03:30AM -0800, Andy Lutomirski wrote:
On Wed, Dec 12, 2018 at 8:52 AM Rich Felker [off-list ref] wrote:
quoted
On Wed, Dec 12, 2018 at 08:39:53AM -0800, Andy Lutomirski wrote:
quoted
I'm proposing another alternative. Given that x32 already proves that
the user bitness model doesn't have to match the kernel model (in x32,
user "long" is 32-bit but the kernel ABI "long" is 64-bit), I'm
proposing extending this to just make the kernel ABI be LP64. So
__kernel_size_t would be 64-bit and pointers in kernel data structures
would be 64-bit. In other words, most or all of the kernel ABI would
just match x86_64.
As far as I can tell, the only thing that really needs unusual
toolchain features here is that C doesn't have an extra-wide pointer
type. The kernel headers would need a way to say "this pointer is
still logically a pointer, and user code may assume that it's 32 bits,
but it has 8-byte alignment."
None of this works on the userspace/C side, nor should any attempt be
made to make it work. Types fundamentally cannot have alignments
larger than their size. If you want to make the alignment of some
pointers 8, you have to make their size 8, and then you just have LP64
again if you did it for all pointers.
If on the other hand you tried to make just some pointers "wide
pointers", you'd also be completely breaking the specified API
contracts of standard interfaces. For example in struct iovec's
iov_base, &foo->iov_base is no longer a valid pointer to an object of
type void* that you can pass to interfaces expecting void**. Sloppy
misunderstandings like what you're making now are exactly why x32 is
already broken and buggy (&foo->tv_nsec already has wrong type for
struct timespec foo).
I don't think it's quite that broken. For the struct iovec example,
we currently have:
struct iovec {
void *iov_base; /* Starting address */
size_t iov_len; /* Number of bytes to transfer */
};
we could have, instead: (pardon any whitespace damage)
struct iovec {
void *iov_base; /* Starting address */
uint32_t __pad0;
size_t iov_len; /* Number of bytes to transfer */
uint32_t __pad1;
} __attribute__((aligned(8));
or the same thing but where iov_len is uint64_t. A pointer to
iov_base still works exactly as expected. Something would need to be
done to ensure that the padding is all zeroed, which might be a real
problem.
We looked at this approach briefly for arm64/ILP32 and zeroing the pads
was the biggest problem. User programs would not explicitly zero the pad
and I'm not sure the compiler would be any smarter. This means it's the
kernel's responsibility to zero the pad (around get_user,
copy_from_user), so it doesn't actually simplify the kernel side of the
syscall interface.
If the data flow goes the other way (kernel to user), this approach
works fine.
No one wants to actually type all the macro gunk into the headers to
make this work, but this type of transformation is what I have in mind
when the compiler is asked to handle the headers. Or there could
potentially be a tool that automatically consumes the uapi headers and
spits out modified headers like this.
If the compiler can handle the zeroing, that would be great, though not
sure how (some __attribute__((zero)) which generates a type constructor
for such structure; it kind of departs from what the C language offers).
Realistically, I think a much better model would be to use true ILP32
code, where all the memory layouts in the uapi match i386.
The conclusion we came to on arm64 was that an ILP32 ABI should not
really be any different from a _new_ 32-bit architecture ABI. It differs
from arm32 a bit (different syscall numbers, off_t is 64-bit,
sigcontext) but not significantly as it is still able to use the
majority of the compat_sys_* wrappers.
--
Catalin
From: Olof Johansson <hidden> Date: 2018-12-13 14:39:06
On Tue, Dec 11, 2018 at 9:40 AM Linus Torvalds
[off-list ref] wrote:
On Mon, Dec 10, 2018 at 5:23 PM Andy Lutomirski [off-list ref] wrote:
quoted
I'm seriously considering sending a patch to remove x32 support from
upstream Linux. Here are some problems with it:
I talked to Arnd (I think - we were talking about all the crazy ABI's,
but maybe it was with somebody else) about exactly this in Edinburgh.
Apparently the main real use case is for extreme benchmarking. It's
the only use-case where the complexity of maintaining a whole
development environment and distro is worth it, it seems. Apparently a
number of Spec submissions have been done with the x32 model.
I'm not opposed to trying to sunset the support, but let's see who complains..
I'm just a single user. I do rely on it though, FWIW.
I hadn't finished my benchmarking in Edinburgh, but for my new machine
that does kernel builds 24/7, I ended up going with x32 userspace (in
a container).
Main reason is that it's a free ~10% improvement in runtime over
64-bit. I.e. GCC-as-a-workload is quite a bit faster as x32,
supposedly mostly due to smaller D cache footprints (I ran out of
cycles to tinker with back and forth perf data collection and settled
down on just running it).
Running classic 32-bit (i386? i686? whatever it's called) is about
half as good. I.e. even then I get a ~5% performance win. Less than
x32, but still better than 64-bit userspace.
-Olof
On Thu, Dec 13, 2018 at 9:39 AM Olof Johansson [off-list ref] wrote:
On Tue, Dec 11, 2018 at 9:40 AM Linus Torvalds
[off-list ref] wrote:
quoted
On Mon, Dec 10, 2018 at 5:23 PM Andy Lutomirski [off-list ref] wrote:
quoted
I'm seriously considering sending a patch to remove x32 support from
upstream Linux. Here are some problems with it:
I talked to Arnd (I think - we were talking about all the crazy ABI's,
but maybe it was with somebody else) about exactly this in Edinburgh.
Apparently the main real use case is for extreme benchmarking. It's
the only use-case where the complexity of maintaining a whole
development environment and distro is worth it, it seems. Apparently a
number of Spec submissions have been done with the x32 model.
I'm not opposed to trying to sunset the support, but let's see who complains..
I'm just a single user. I do rely on it though, FWIW.
I hadn't finished my benchmarking in Edinburgh, but for my new machine
that does kernel builds 24/7, I ended up going with x32 userspace (in
a container).
Main reason is that it's a free ~10% improvement in runtime over
64-bit. I.e. GCC-as-a-workload is quite a bit faster as x32,
supposedly mostly due to smaller D cache footprints (I ran out of
cycles to tinker with back and forth perf data collection and settled
down on just running it).
Running classic 32-bit (i386? i686? whatever it's called) is about
half as good. I.e. even then I get a ~5% performance win. Less than
x32, but still better than 64-bit userspace.
-Olof
I'm familiar with two embedded Linux systems using x32 ABI for the
following reasons:
- Significant performance benefits over i386 ABI.
- Smaller memory footprint than x86_64 (pointer-heavy data structures)
- Large legacy software base with many ILP32 assumptions, much
smaller effort to move to x32 than x86_64.
Some examples of (relatively minor) problems encountered with x32:
- Time-related data type mismatch in socket options (fixed)
https://patchwork.ozlabs.org/patch/904254/
- Userspace overrun with select() (patch proposed)
https://patchwork.kernel.org/patch/10245677/
- glibc get_phys_pages() doesn't work correctly with x32
(assumes that struct sysinfo fields are not wider than "long").
So, one small vote for keeping x32 with the hope that support for it
can continue to be improved...
- Lance
On Thu, Dec 13, 2018 at 12:40:25PM +0000, Catalin Marinas wrote:
On Wed, Dec 12, 2018 at 10:03:30AM -0800, Andy Lutomirski wrote:
quoted
On Wed, Dec 12, 2018 at 8:52 AM Rich Felker [off-list ref] wrote:
quoted
On Wed, Dec 12, 2018 at 08:39:53AM -0800, Andy Lutomirski wrote:
quoted
I'm proposing another alternative. Given that x32 already proves that
the user bitness model doesn't have to match the kernel model (in x32,
user "long" is 32-bit but the kernel ABI "long" is 64-bit), I'm
proposing extending this to just make the kernel ABI be LP64. So
__kernel_size_t would be 64-bit and pointers in kernel data structures
would be 64-bit. In other words, most or all of the kernel ABI would
just match x86_64.
As far as I can tell, the only thing that really needs unusual
toolchain features here is that C doesn't have an extra-wide pointer
type. The kernel headers would need a way to say "this pointer is
still logically a pointer, and user code may assume that it's 32 bits,
but it has 8-byte alignment."
None of this works on the userspace/C side, nor should any attempt be
made to make it work. Types fundamentally cannot have alignments
larger than their size. If you want to make the alignment of some
pointers 8, you have to make their size 8, and then you just have LP64
again if you did it for all pointers.
If on the other hand you tried to make just some pointers "wide
pointers", you'd also be completely breaking the specified API
contracts of standard interfaces. For example in struct iovec's
iov_base, &foo->iov_base is no longer a valid pointer to an object of
type void* that you can pass to interfaces expecting void**. Sloppy
misunderstandings like what you're making now are exactly why x32 is
already broken and buggy (&foo->tv_nsec already has wrong type for
struct timespec foo).
I don't think it's quite that broken. For the struct iovec example,
we currently have:
struct iovec {
void *iov_base; /* Starting address */
size_t iov_len; /* Number of bytes to transfer */
};
we could have, instead: (pardon any whitespace damage)
struct iovec {
void *iov_base; /* Starting address */
uint32_t __pad0;
size_t iov_len; /* Number of bytes to transfer */
uint32_t __pad1;
} __attribute__((aligned(8));
or the same thing but where iov_len is uint64_t. A pointer to
iov_base still works exactly as expected. Something would need to be
done to ensure that the padding is all zeroed, which might be a real
problem.
We looked at this approach briefly for arm64/ILP32 and zeroing the pads
was the biggest problem. User programs would not explicitly zero the pad
and I'm not sure the compiler would be any smarter. This means it's the
kernel's responsibility to zero the pad (around get_user,
copy_from_user), so it doesn't actually simplify the kernel side of the
syscall interface.
If the data flow goes the other way (kernel to user), this approach
works fine.
quoted
No one wants to actually type all the macro gunk into the headers to
make this work, but this type of transformation is what I have in mind
when the compiler is asked to handle the headers. Or there could
potentially be a tool that automatically consumes the uapi headers and
spits out modified headers like this.
If the compiler can handle the zeroing, that would be great, though not
sure how (some __attribute__((zero)) which generates a type constructor
for such structure; it kind of departs from what the C language offers).
The compiler fundamentally can't. At the very least it would require
effective type tracking, which requires shadow memory and is even more
controversial than -fstrict-aliasing (because in a sense it's a
stronger version thereof). But even effective type tracking would not
help, since you can have things like:
struct iovec *iov = malloc(sizeof *iov);
scanf("%p %zu", &iov->iov_base, &iov->iov_len);
where no store to the object via the struct type ever happens and the
only stores that do happen are invisible across translation unit
boundaries. (Ignore that scanf here is awful; it's just a canonical
example of a function that would store the members via pointers to
them.)
The kernel-side approach could work if the kernel had some markup for
fields that need to be zero- or sign-extended when copied from user in
a 32-bit process and applied them at copy time. That could also fix
the existing tv_nsec issue. I'm not sure how difficult/costly it would
be though.
Rich
On Tue, Dec 11, 2018 at 11:29:14AM +0100, John Paul Adrian Glaubitz wrote:
I can't say anything about the syscall interface. However, what I do know
is that the weird combination of a 32-bit userland with a 64-bit kernel
interface is sometimes causing issues. For example, application code usually
expects things like time_t to be 32-bit on a 32-bit system. However, this
isn't the case for x32 which is why code fails to build.
I don't see any basis for this claim about expecting time_t to be
32-bit. I've encountered some programs that "implicitly assume" this
by virtue of assuming they can cast time_t to long to print it, or
similar. IIRC this was an issue in busybox at one point; I'm not sure
if it's been fixed. But any software that runs on non-Linux unices has
long been corrected. If not, 2038 is sufficiently close that catching
and correcting any such remaining bugs is more useful than covering
them up and making the broken code work as expected.
Rich
From: Richard Purdie <hidden> Date: 2018-12-13 16:11:45
On Mon, 2018-12-10 at 17:40 -0800, Linus Torvalds wrote:
On Mon, Dec 10, 2018 at 5:23 PM Andy Lutomirski [off-list ref]
wrote:
quoted
I'm seriously considering sending a patch to remove x32 support
from
upstream Linux. Here are some problems with it:
I talked to Arnd (I think - we were talking about all the crazy
ABI's,
but maybe it was with somebody else) about exactly this in Edinburgh.
Apparently the main real use case is for extreme benchmarking. It's
the only use-case where the complexity of maintaining a whole
development environment and distro is worth it, it seems. Apparently
a
number of Spec submissions have been done with the x32 model.
I'm not opposed to trying to sunset the support, but let's see who
complains..
FWIW, the Yocto Project has working x32 support which it regularly
builds, tests and has as a supported configuration. We did help
identify and fix many of the places userspace didn't support it.
As with open source in general, its hard to know how many people are
actually using it. I don't think there are huge numbers but we do
occasionally get bug reports.
Cheers,
Richard
On Tue, Dec 11, 2018 at 11:29:14AM +0100, John Paul Adrian Glaubitz wrote:
quoted
I can't say anything about the syscall interface. However, what I do know
is that the weird combination of a 32-bit userland with a 64-bit kernel
interface is sometimes causing issues. For example, application code usually
expects things like time_t to be 32-bit on a 32-bit system. However, this
IMHO this just historically grown (as in "it has been forever that way"
- it sounds way better in Viennese dialect though;-).
quoted
isn't the case for x32 which is why code fails to build.
I don't see any basis for this claim about expecting time_t to be
32-bit. I've encountered some programs that "implicitly assume" this
by virtue of assuming they can cast time_t to long to print it, or
similar. IIRC this was an issue in busybox at one point; I'm not sure
if it's been fixed. But any software that runs on non-Linux unices has
long been corrected. If not, 2038 is sufficiently close that catching
and correcting any such remaining bugs is more useful than covering
them up and making the broken code work as expected.
Yup, unconditionally providing 64bit
time_t/timespec/timeval/...-equivalents with libc and syscall support
also for 32bit architectures (and deprecating all 32bit versions) should
be the way to go.
FWIW I have
---- snip ----
#if defined __x86_64__
# if defined __ILP32__ // x32
# define PRI_time_t "lld" // for time_t
# define PRI_nsec_t "lld" // for tv_nsec in struct timespec
# else // x86_64
# define PRI_time_t "ld" // for time_t
# define PRI_nsec_t "ld" // for tv_nsec in struct timespec
# endif
#else // i[3-6]68
# define PRI_time_t "ld" // for time_t
# define PRI_nsec_t "ld" // for tv_nsec in struct timespec
#endif
---- snip ----
in my userspace code for printf() and friends - I don't know how libc's
react to such a patch (and I don't care for the name of the macros as
long it's obviously clear for which type they are).
I assume/fear we won't get additional modifiers into the relevant
standards for libc types (as they are far more like pid_t, uid_t etc.).
And casting to u/intmaxptr_t to get a defined printf()-modifier doesn't
look appealing to me to "solve" such issues.
MfG,
Bernd
--
"I dislike type abstraction if it has no real reason. And saving
on typing is not a good reason - if your typing speed is the main
issue when you're coding, you're doing something seriously wrong."
- Linus Torvalds
From: David Laight <hidden> Date: 2018-12-14 14:38:49
From: Richard Weinberger
Sent: 13 December 2018 09:05
On Thu, Dec 13, 2018 at 6:03 AM Kevin Easton [off-list ref] wrote:
quoted
On Tue, Dec 11, 2018 at 11:29:14AM +0100, John Paul Adrian Glaubitz wrote:
...
quoted
I can't say anything about the syscall interface. However, what I do know
is that the weird combination of a 32-bit userland with a 64-bit kernel
interface is sometimes causing issues. For example, application code usually
expects things like time_t to be 32-bit on a 32-bit system. However, this
isn't the case for x32 which is why code fails to build.
OpenBSD and NetBSD both have 64-bit time_t on 32-bit systems and have
had for four or five years at this point.
They can also do flag-day changes and break existing applications, Linux not.
Not true at all.
The binary compatibility in NetBSD is probably better than that in Linux
and goes back a long way.
For the time_t changes new system calls numbers were assigned where needed.
The system headers and libc were updated so that recompiled code would
use the new system calls.
The only real advantage that NetBSD has is that its libc (and standard
utilities) are released with the kernel making it much easier to get
applications to use the new features.
This was also done a very long time ago when file offsets were extended
to 64 bits.
Some of the system calls have quite a few 'compatibility' versions.
As well as the ones for emulations of other operating systems.
It has been possible to run copies of firefox compiled for Linux
under NetBSD.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)
From: Richard Weinberger <richard@nod.at> Date: 2018-12-14 15:18:08
Am Freitag, 14. Dezember 2018, 15:38:53 CET schrieb David Laight:
From: Richard Weinberger
quoted
Sent: 13 December 2018 09:05
On Thu, Dec 13, 2018 at 6:03 AM Kevin Easton [off-list ref] wrote:
quoted
On Tue, Dec 11, 2018 at 11:29:14AM +0100, John Paul Adrian Glaubitz wrote:
...
quoted
I can't say anything about the syscall interface. However, what I do know
is that the weird combination of a 32-bit userland with a 64-bit kernel
interface is sometimes causing issues. For example, application code usually
expects things like time_t to be 32-bit on a 32-bit system. However, this
isn't the case for x32 which is why code fails to build.
OpenBSD and NetBSD both have 64-bit time_t on 32-bit systems and have
had for four or five years at this point.
They can also do flag-day changes and break existing applications, Linux not.
Not true at all.
The binary compatibility in NetBSD is probably better than that in Linux
and goes back a long way.
For the time_t changes new system calls numbers were assigned where needed.
The system headers and libc were updated so that recompiled code would
use the new system calls.
The only real advantage that NetBSD has is that its libc (and standard
utilities) are released with the kernel making it much easier to get
applications to use the new features.
This was also done a very long time ago when file offsets were extended
to 64 bits.
Some of the system calls have quite a few 'compatibility' versions.
As well as the ones for emulations of other operating systems.
It has been possible to run copies of firefox compiled for Linux
under NetBSD.
I stand corrected, I was under the impression that NetBSD went the same
path as OpenBSD did. Thanks for pointing this out.
Thanks,
//richard
On Fri, Dec 14, 2018 at 03:13:10PM +0100, Bernd Petrovitsch wrote:
On 13/12/2018 17:02, Rich Felker wrote:
quoted
On Tue, Dec 11, 2018 at 11:29:14AM +0100, John Paul Adrian Glaubitz wrote:
quoted
I can't say anything about the syscall interface. However, what I do know
is that the weird combination of a 32-bit userland with a 64-bit kernel
interface is sometimes causing issues. For example, application code usually
expects things like time_t to be 32-bit on a 32-bit system. However, this
IMHO this just historically grown (as in "it has been forever that way"
- it sounds way better in Viennese dialect though;-).
quoted
quoted
isn't the case for x32 which is why code fails to build.
I don't see any basis for this claim about expecting time_t to be
32-bit. I've encountered some programs that "implicitly assume" this
by virtue of assuming they can cast time_t to long to print it, or
similar. IIRC this was an issue in busybox at one point; I'm not sure
if it's been fixed. But any software that runs on non-Linux unices has
long been corrected. If not, 2038 is sufficiently close that catching
and correcting any such remaining bugs is more useful than covering
them up and making the broken code work as expected.
Yup, unconditionally providing 64bit
time_t/timespec/timeval/...-equivalents with libc and syscall support
also for 32bit architectures (and deprecating all 32bit versions) should
be the way to go.
FWIW I have
---- snip ----
#if defined __x86_64__
# if defined __ILP32__ // x32
# define PRI_time_t "lld" // for time_t
# define PRI_nsec_t "lld" // for tv_nsec in struct timespec
# else // x86_64
# define PRI_time_t "ld" // for time_t
# define PRI_nsec_t "ld" // for tv_nsec in struct timespec
# endif
#else // i[3-6]68
# define PRI_time_t "ld" // for time_t
# define PRI_nsec_t "ld" // for tv_nsec in struct timespec
#endif
---- snip ----
in my userspace code for printf() and friends - I don't know how libc's
react to such a patch (and I don't care for the name of the macros as
long it's obviously clear for which type they are).
I assume/fear we won't get additional modifiers into the relevant
standards for libc types (as they are far more like pid_t, uid_t etc.).
And casting to u/intmaxptr_t to get a defined printf()-modifier doesn't
look appealing to me to "solve" such issues.
This is all useless (and wrong since tv_nsec is required to have type
long as part of C and POSIX, regardless of ILP32-vs-LP64; that's a bug
in glibc's x32). Just do:
printf("%jd", (intmax_t)t);
Saving 2 or 3 insns (for sign or zero extension) around a call to
printf is not going to make any measurable difference to performance
or any significant difference to size, and it's immeasurably more
readable than the awful PRI* macros and the
adjacent-string-concatenation they rely on.
Rich
On Fri, Dec 14, 2018 at 03:13:10PM +0100, Bernd Petrovitsch wrote:
[..]
quoted
FWIW I have
---- snip ----
#if defined __x86_64__
# if defined __ILP32__ // x32
# define PRI_time_t "lld" // for time_t
# define PRI_nsec_t "lld" // for tv_nsec in struct timespec
# else // x86_64
# define PRI_time_t "ld" // for time_t
# define PRI_nsec_t "ld" // for tv_nsec in struct timespec
# endif
#else // i[3-6]68
# define PRI_time_t "ld" // for time_t
# define PRI_nsec_t "ld" // for tv_nsec in struct timespec
#endif
---- snip ----
in my userspace code for printf() and friends - I don't know how libc's
react to such a patch (and I don't care for the name of the macros as
long it's obviously clear for which type they are).
I assume/fear we won't get additional modifiers into the relevant
standards for libc types (as they are far more like pid_t, uid_t etc.).
And casting to u/intmaxptr_t to get a defined printf()-modifier doesn't
look appealing to me to "solve" such issues.
This is all useless (and wrong since tv_nsec is required to have type
long as part of C and POSIX, regardless of ILP32-vs-LP64; that's a bug
Thanks. OK, I didn't know that - and 32bit is enough to represent 1E9
(as a second won't have more nanosecs).
Hmm, can we fix that in the x32 world?
Sry, I'm not the expert on glibc vs ABI va syscall interface vs breakage
there though.
in glibc's x32). Just do:
printf("%jd", (intmax_t)t);
Saving 2 or 3 insns (for sign or zero extension) around a call to
printf is not going to make any measurable difference to performance
Until someone comes up with hardware with ASIC support for 1k bit int's
and (ab)uses intmax_t for that. SCNR ....
or any significant difference to size, and it's immeasurably more
readable than the awful PRI* macros and the
adjacent-string-concatenation they rely on.
One gets used to the PRI_* macros over time (and there no calculated
format strings in my world) - and type casts are not better in my eyes ...
MfG,
Bernd
--
"I dislike type abstraction if it has no real reason. And saving
on typing is not a good reason - if your typing speed is the main
issue when you're coding, you're doing something seriously wrong."
- Linus Torvalds
From: Thomas Schöbel-Theuer <hidden> Date: 2018-12-14 21:22:45
On 12/11/18 02:23, Andy Lutomirski wrote:
I'm seriously considering sending a patch to remove x32 support from
upstream Linux.
I am downstream maintainer of several self-patched kernels at 1&1 Ionos.
The kernels are rolled out to several tenthousands of production servers
running in several datacenters and in multiple continents.
Currently, we have a few thousands of servers relying on 32bit ABIs in
some thousands of VMs and/or containers of various types (LXC, OpenVZ, etc).
Here is my private opinion, not speaking for 1&1: at some point the
future, 32bit userspace support needs to be dropped anyway, somewhen in
future. This is inevitable in the very long term.
Thus the discussion should be about _timing_ / _roadmaps_, but not about
the fact as such.
My suggestion:
1) please release / declare a new LTS kernel, with upstream support for
at least 5 years (as usual). Currently, only 4.4 and 4.9 are marked as
LTS. Either mark another existing stable kernel as LTS, or a future one.
2) please _announce_ _now_ that after the _next_ LTS kernel (whichever
you want to declare as such), you will _afterwards_ drop the legacy
32bit support for 64 kernels (I am deliberately using "management speak"
here).
=> result: the industry should have to fair chance to deal with such a
roadmap. Yes, it will hurt some people, but they will have enough time
for their migration projects.
Example: I know that out of several millions of customers of webhosting,
a very low number of them have some very old legacy 32bit software
installed in their webspace. This cannot be supported forever. But the
number of such cases is very small, and there just needs to be enough
time for finding a solution for those few customers.
3) the next development kernel _after_ that LTS release can then
immediately drop the 32bit support. Enterprise users should have enough
time for planning, and for lots of internal projects modernizing their
infrastructure. Usually, they will need to do this anyway in the long term.
A roadmap should be _reliable_ for planning, and there should be no
"unexpected surprises". That's the most important requirements.
Notice: 5 years is what I know will very likely work for 1&1 Ionos. I
cannot speak for other large-scale enterprise users, but I think most of
them should be able to deal with suchalike intervals in a clear roadmap.
Addendum / side note: I know of cases where _critical_ / sometimes even
proprietary 32bit enterprise software needs to run for several
_decades_. Please don't drop KVM/qemu support for 32bit guests using
_old_ (unchanged) 32bit kernels, which are just happen to run on 64bit
hypervisors. But AFAICS that was not the intention of this initiative.
Just keep the latter in mind as another (independent) requirement.
Cheers,
Thomas
From: Andy Lutomirski <luto@kernel.org> Date: 2018-12-14 21:24:44
On Fri, Dec 14, 2018 at 1:22 PM Thomas Schöbel-Theuer
[off-list ref] wrote:
On 12/11/18 02:23, Andy Lutomirski wrote:
quoted
I'm seriously considering sending a patch to remove x32 support from
upstream Linux.
I am downstream maintainer of several self-patched kernels at 1&1 Ionos.
The kernels are rolled out to several tenthousands of production servers
running in several datacenters and in multiple continents.
Currently, we have a few thousands of servers relying on 32bit ABIs in
some thousands of VMs and/or containers of various types (LXC, OpenVZ, etc).
I should clarify again: I am not suggesting that we drop 32-bit
support in the forseeable future. (Well, I might eventually suggest
that we drop support for 32-bit *hardware* at some point, but not for
32-bit compat software.) Linux's compat code is quite robust and is
even fairly maintainable.
I'm talking about x32, which is a different beast.
From: Thomas Schöbel-Theuer <hidden> Date: 2018-12-14 21:41:28
On 12/14/18 22:24, Andy Lutomirski wrote:
I should clarify again: I am not suggesting that we drop 32-bit
support in the forseeable future. (Well, I might eventually suggest
that we drop support for 32-bit *hardware* at some point, but not for
32-bit compat software.) Linux's compat code is quite robust and is
even fairly maintainable.
I'm talking about x32, which is a different beast.
OK, this does not really make a big difference for large-scale
enterprise users. Some day in the very distant future any legacy system
will disappear, eventually.
So from my viewpoint the mentioned roadmap / timing requirements will
remain the same, whatever you are dropping.
Enterprise-critical use cases will probably need to be migrated to
KVM/qemu together with their old kernel versions, anyway (because the
original hardware will be no longer available in a few decades).
x32 is not this at all. The kernel ABI part of x32 isn't ILP32. It's
IP32, 32-bit size_t, and *64-bit* long. The core kernel doesn't
Yeah, I was looking at this from userspace PoV, as I said I’m not
a Linux kernel programmer.
In BSD we have register_t which is probably the equivalent to your
__kernel_long_t? Maybe removing the “long” from the name helps.
But yes, x32 is just a (second to i386) ILP32 userspace API in an
amd64 kernel. This does imply mapping on the userspace (x32) to
kernel (amd64) boundary and back. I would have thought full struct
member mapping, as dalias described, to be the most robust.
something similar to work using the normal x86_64 syscalls. And I'm
But those would require the longer structs etc. and therefore
lose all the benefits of x32…
bye,
//mirabilos
--
„Cool, /usr/share/doc/mksh/examples/uhr.gz ist ja ein Grund,
mksh auf jedem System zu installieren.“
-- XTaran auf der OpenRheinRuhr, ganz begeistert
(EN: “[…]uhr.gz is a reason to install mksh on every system.”)
From: Thomas Schoebel-Theuer <hidden> Date: 2018-12-15 07:42:14
On 12/14/18 22:41, Thomas Schöbel-Theuer wrote:
On 12/14/18 22:24, Andy Lutomirski wrote:
quoted
I'm talking about x32, which is a different beast.
So from my viewpoint the mentioned roadmap / timing requirements will
remain the same, whatever you are dropping.
Enterprise-critical use cases will probably need to be migrated to
KVM/qemu together with their old kernel versions, anyway (because the
original hardware will be no longer available in a few decades).
Here is a systematic approach to the problem.
AFAICS legacy 32bit userspace code (which exists in some notable masses)
can be executed at least in the following ways:
1) natively on 32bit-capable hardware, under 32bit kernels. Besides
legacy hardware, this also encompasses most current Intel / AMD 64bit
hardware in 32bit compatibility mode.
2) under 64bit kernels, using the 32bit compat layer from practically
any kernel version.
3) under KVM/qemu.
When you just drop 1), users have a fair chance by migrating to any of
the other two possibilities.
As explained, a time frame of ~5 years should work for the vast majority.
If you clearly explain the migration paths to your users (and to the
press), I think it will be acceptable.
[side note: I know of a single legacy instance which is now ~20 years
old, but makes a revenue of several millions per month. These guys have
large quantities of legacy hardware in stock. And they have enough money
to hire a downstream maintainer in case of emergency.]
Fatal problems would only arise if you would drop all three
possibilities in the very long term.
In ~100 years, possibility 3) should be sufficient for handling use
cases like preservation of historic documents. The latter is roughly
equivalent to running binary-only MSDOS, Windows NT, and similar, even
in 100 years, and even non-natively under future hardware architectures.
From: Andy Lutomirski <luto@amacapital.net> Date: 2018-12-15 15:30:48
On Dec 14, 2018, at 11:41 PM, Thomas Schoebel-Theuer [off-list ref] wrote:
quoted
On 12/14/18 22:41, Thomas Schöbel-Theuer wrote:
quoted
On 12/14/18 22:24, Andy Lutomirski wrote:
I'm talking about x32, which is a different beast.
So from my viewpoint the mentioned roadmap / timing requirements will remain the same, whatever you are dropping.
Enterprise-critical use cases will probably need to be migrated to KVM/qemu together with their old kernel versions, anyway (because the original hardware will be no longer available in a few decades).
Here is a systematic approach to the problem.
This is way off topic. Let’s not discuss it on this thread, please.
From: Enrico Weigelt, metux IT consult <hidden> Date: 2019-01-22 13:34:56
On 14.12.18 22:16, Thomas Schöbel-Theuer wrote:
Hi,
Currently, we have a few thousands of servers relying on 32bit ABIs in> some thousands of VMs and/or containers of various types (LXC,
OpenVZ,> etc).
Similar w/ my clients, but in Industrie/Embedded/i4.0 world. We have
thousands of devices in the field, that need support for at least a
decade.
Right now, I'm working on a new device, that - even having a 64bit cpu -
will still run 32bit userland for various reasons (memory footprint
is one of them).
Even on the data center side, where we have *a lot* of containers
(eg. container-based microservices on minimal distros like alpine
or ptxdist - as well as many different, often highly specialized,
legacy applications), 32bit userland indeed is a major factor.
Dropping it would have huge economic consequences:
* massive costs for re-deployment / migration of thousands of different
applications
* field roll of thousands of different embedded / industrial devices,
as a major cpu arch
* huge increase in memory consumption and io load in container-
based microservice environments
* loosing x86 for a huge portion of the embedded / small devices world
I don't even dare to estimate the economic damage. (not even speaking
about reputation loss of the Linux community)
Here is my private opinion, not speaking for 1&1: at some point the> future, 32bit userspace support needs to be dropped anyway, somewhen
in> future. This is inevitable in the very long term.
Very, very, long term. Just the devices I'm coping w/ will have
remaining lifetime of at least 10..15 years. That's a really long time
in IT world. Nobody knows whether x86 at all will play a big role then.
1) please release / declare a new LTS kernel, with upstream support for
at least 5 years (as usual). Currently, only 4.4 and 4.9 are marked as
LTS. Either mark another existing stable kernel as LTS, or a future one.
That might be okay in enterprise world (even though, here I'm regularily
stumbling across much older code in production), but in industrial
world, the product lifetimes are much longer - 20+yrs years
usualstandard.are pretty common.
--mtx
--
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
info@metux.net -- +49-151-27565287