On Tue, Mar 16, 2021 at 11:29 PM Johannes Berg
[off-list ref] wrote:

Hi,

Hi Johannes,

quoted

My interpretation of MMU/NOMMU is like this;

With (emulated) MMU architecture you will have more smooth integration
with other subsystems of kernel tree, because some subsystems/features
are written with "#ifdef CONFIG_MMU".  While NOMMU doesn't, it will
bring a simplified design with better portability.

LKL takes rather to benefit better portability.

I don't think it *matters* so much for portability? I mean, every system
under the sun is going to allow some kind of "mprotect", right? You
don't really want to port LKL to systems that don't have even that?

One use case where this matters are non OS environments such as
bootloaders [1], running on bare-bone hardware or kernel drivers [2,
3]. IMO it would be nice to keep these properties.

[1] https://www.freelists.org/post/linux-kernel-library/UEFI-LKL-port
[2] https://github.com/lkl/lkl-win-fsd
[3] https://www.haiku-os.org/tags/lkl-haiku-fsd/

quoted

quoted

 * Why pthreads and all? You already require jump_buf, so UML's
   switch_threads() ought to be just fine for scheduling? It almost
   seems like you're doing this just so you can serialize against "other
   threads" (application threads), but wouldn't that trivially be
   handled by the application? You could let it hook into switch_to() or
   something, but why should a single "LKL" CPU ever require multiple
   threads? Seems to me that the userspace could be required to
   "lkl_run()" or so (vs. lkl_start()). Heck, you could even exit
   lkl_run() every time you switch tasks in the kernel, and leave
   scheduling the kernel vs. the application entirely up to the
   application? (A trivial application would be simply doing something
   like "while (1) { lkl_run(); pause(); }" mimicking the idle loop of
   UML.

There is a description about this design choice in the LKL paper (*1);

  "implementations based on setjmp - longjmp require usage of a single
  stack space partitioned between all threads. As the Linux kernel
  uses deep stacks (especially in the VFS layer), in an environment
  with small stack sizes (e.g. inside another operating system's
  kernel) this will place a very low limit on the number of possible
  threads."

(from page 2, Section II, 2) Thread Support)

This is a reason of using pthread as a context primitive.

That impliciation (setjmp doesnt do stacks, so must use pthread) really
isn't true, you also have posix contexts or windows fibers. That would
probably be much easier to understands, since real threads imply that
you have actual concurrency, which _shouldn't_ be true in the case of
Linux emulated as being on a single CPU.

Perhaps that just means you chose the wrong abstraction.

In usfstl (something I've been working on) for example, we have an
abstraction called (execution) "contexts", and they can be implemented
using pthreads, fibers, or posix contexts, and you switch between them.

(see https://github.com/linux-test-project/usfstl/blob/main/src/ctx-common.c)

Using real pthreads implies that you have real threading, but then you
need access to real mutexes, etc.

If your abstraction was instead "switch context" then you could still
implement it using pthreads+mutexes, or you could implement it using
fibers on windows, or posix contexts - but you'd have a significantly
reduced API surface, since you'd only expose __switch_to() or similar,
and maybe a new stack allocation etc.

You are right. When I started the implementation for ucontext it was
obvious that it would be much simpler to have abstractions closer to
what Linux has (alloc, free and switch threads). But I never got to
finish that and then things went into a different direction.

Additionally, I do wonder how UML does this now, it *does* use setjmp,
so are you saying it doesn't properly use the kernel stacks?

To clarify a bit the statement in the paper, the context there was
that we should push the thread implementation to the
application/environment we run rather than providing "LKL" threads.
This was particularly important for running LKL in other OSes kernel
drivers. But you are right, we can use the switch abstraction and
implement it with threads and mutexes for those environments where it
helps.

quoted

to design that with UML, what we need to do are;

1) change Makefile to output liblinux.a

or liblinux.so, I guess, dynamic linking should be ok.

quoted

we faced linker script issue, which is related with generating
relocatable object in the middle.

2) make the linker-script clean with 2-stage build
we fix the linker issues of (1)

3) expose syscall as a function call
conflicts names (link-time and compile-time conflicts)

4) header rename, object localization
to fix the issue (3)

This is a common set of modifications to a library of UML.

All of this is just _build_ issues. It doesn't mean you couldn't take
some minimal code + liblinux.a and link it to get a "linux" equivalent
to the current UML?

TBH, I started thinking that it might be _really_ nice to be able to
write an application that's *not quite UML* but has all the properties
of UML built into it, i.e. can run userspace etc.

quoted

Other parts are a choice of design, I believe.
Because a library is more _reusable_ than an executable (by it means), the
choice of LKL is to be portable, which the current UML doesn't pursue it
extensibly (focus on intel platforms).

I don't think this really conflicts.

You could have a liblinux.a/liblinux.so and some code that links it all
together to get "linux" (UML). Having userspace running inside the UML
(liblinux) might only be supported on x86 for now, MMU vs. NOMMU might
be something that's configurable at build time, and if you pick NOMMU
you cannot run userspace either, etc.

But conceptually, why wouldn't it be possible to have a liblinux.so that
*does* build with MMU and userspace support, and UML is a wrapper around
it?

This is an interesting idea. Conceptually I think it is possible.
There are lots of details to be figured out before we do this. I think
that having a NOMMU version could be a good step in the right
direction, especially since I think a liblinux.so has more NOMMU
usecases than MMU usecases - but I haven't given too much thought to
the MMU usecases.