On Wed, 2025-05-07 at 09:47 +0200, Arnout Engelen wrote:

On Tue, May 6, 2025, at 15:24, James Bottomley wrote:

quoted

I'll repeat the key point again: all modern hermetic build systems
come with provenance which is usually a signature.

I'm not sure the 'hermetic build' parallel is so applicable here:
typically a hermetic build will produce an artifact and a signature,
and when you embed that result in a larger aggregate, you only embed
the artifact (not the signature) and sign the aggregate.

That depends whether you want to demonstrate the provenance of the
result to someone consuming your aggregate or not; Some people are OK
with the trust my signature approach, others want tracing to point of
origin.

 With module signatures, the module *and* their signatures are
embedded in the aggregate (e.g. ISO, disk image), which is
where (at least in my case) the friction comes from.

For Linux in particular, most people won't be booting any image unless
the binary is secure boot signed, so this problem doesn't go away if
you strip module signatures.

quoted

Plus, you've got to remember that a signature is a cryptographic
function of the hash over the build minus the signature.  You can't
verify a signature unless you know how to get the build minus the
signature.  So the process is required to be deterministic.

Right: there is no friction validating the module signatures, that is
fine. There is friction validating the aggregate artifact (e.g. ISO,
disk image), though, because of those signatures embedded into it.

I think we understand the problem with signatures (particularly the
ones which add entropy and can thus change every time the same object
is signed).  However, I don't think we can accept that no signatures
can be on the ISO ... we'll have to have at least secure boot
signatures and if there's a way of doing that then there should be a
way of doing other signatures.

As you mentioned earlier, of course this is *possible* to do (for
example by adding the signatures as inputs to the second
'independent' build or by creating a hard-to-validate 'check recipe'
running the build in reverse). Still, checking modules at run time by
hash instead of by signature would be a much simpler option for such
scenario's.

Well, my objection was merely to the description saying verifying
reproducibility with signatures was not possible (it is).

However, the problem with distros adopting an immutable hash list for
module loading would be DKMS, but I think the distributions that go
that route have all solved the reproducibility issues with signatures
anyway, so perhaps that's not an issue.

quoted

quoted

quoted

All current secure build processes (hermetic builds, SLSA and
the like) are requiring output provenance (i.e. signed
artifacts).  If you try to stand like Canute against this tide
saying "no signed builds", you're simply opposing progress for
the sake of it

I don't think anyone is saying 'no signed builds', but we'd enjoy
being able to keep the signatures as detached metadata instead of
having to embed them into the 'actual' artifacts.

We had this debate about 15 years ago when Debian first started
reproducible builds for the kernel.  Their initial approach was
detached module signatures.  This was the original patch set:

https://lore.kernel.org/linux-modules/20160405001611.GJ21187@decadent.org.uk/ (local)

And this is the reason why Debian abandoned it:

https://lists.debian.org/debian-kernel/2016/05/msg00384.html

That is interesting history, thanks for digging that up. Of the 2
problems Ben mentions running into there, '1' does not seem universal
(I think this feature is indeed mainly interesting for systems where
you don't _want_ anyone to be able to load locally-built modules),
and '2' is a problem that detached signatures have but module hashes
don't have.

I think Debian ended up going with 2, but since they also provide DKMS
infrastructure, hash module lists won't work for them anyway.

quoted

The specific problem is why detached signatures are almost always a
problem: after a period of time, particularly if the process for
creating updated artifacts gets repeated often matching the output
to the right signature becomes increasingly error prone.

I haven't experienced that issue with the module hashes yet.

Heh, I'll repeat this question after you've done umpteen builds of the
same kernel for debugging purposes. The problem that will bite me is
that I often just rebuild a single module and reinsert to try to chase
a bug down.  With this scheme I can't simply reinsert, I'd have to
rebuild the hash list and reboot the entire vmlinux.

quoted

Debian was, however, kind enough to attach what they currently do
to get reproducible builds to the kernel documentation:

https://docs.kernel.org/kbuild/reproducible-builds.html

Cool, I was aware of that page but didn't know it was initially
contributed by Debian.

quoted

However, if you want to detach the module signatures for packaging,
so the modules can go in a reproducible section and the signatures
elsewhere, then I think we could accommodate that (the output of
the build is actually unsigned modules, they just get signed on
install).

At least I don't really come to this from the packaging perspective,
but from the "building an independently verifiable ISO/disk image"
perspective. Separating the modules and the signatures into separate
packages doesn't help me there, since they'd still both need to be
present on the image.

So how do you cope with secure boot?  I mean if the object is to
produce an ISO that is demonstrably reproducible but otherwise
unusable, we can certainly script a way to excise all the signatures
including the secure boot one.

Regards,

James