RE: [musl] Re: Regression: git no longer works with musl libc's regex impl

Hi Johannes,

-------- Original Message --------
Subject: RE: [musl] Re: Regression: git no longer works with musl libc's
regex impl
From: Johannes Schindelin <redacted>
Date: Wed, October 05, 2016 3:49 am
To: writeonce@midipix.org
Cc: musl@lists.openwall.com, git@vger.kernel.org, Jeff King
[off-list ref]

Hi writeonce,

On Tue, 4 Oct 2016, writeonce@midipix.org wrote:

quoted

< On Tue, 4 Oct 2016, Rich Felker wrote:
< 
< > On Tue, Oct 04, 2016 at 11:27:22AM -0400, Jeff King wrote:
< > > On Tue, Oct 04, 2016 at 11:08:48AM -0400, Rich Felker wrote:
< > > 
< > > > 1. is nonzero mod page size, it just works; the remainder of the
< > > > last page reads as zero bytes when mmapped.
< > > 
< > > Is that a portable assumption?
< > 
< > Yes.
< 
< No, it is not. You quote POSIX, but the matter of the fact is that we
< use a subset of POSIX in order to be able to keep things running on
< Windows.

As far as I can tell (and as the attached program may help demonstrate),
the above assumption has been valid on all versions of Windows since at
least Windows 2000.

And since W2K is already past its end of life, it would be safe for
practical considerations.

However, I have to add two comments to that:

- it is *not* guaranteed. The behavior is undefined, even if you see
 consistent behavior so far. Future Windows versions might break that
 assumption freely, though.

That is of course the official language, and generally speaking a good
rule of thumb. However... there is enough information to suggest that
when it comes to mapping of file-backed sections, the NT kernel
developers will choose to keep things the way they are. In brief, here's
why:

Given a "gray zone" that spans from EOF to end-of-page, there are in
essence three possible behaviors:

[1] bytes in the gray zone are accessible and are all zero's. This is
the current behavior.
[2] bytes in the gray zone are not accessible; trying to read past EOF
would result in a segfault.
[3] bytes in the gray zone are accessible but might contain random data
or junk.

Assessment:

[1] backward-compatible, POSIX-compliant, single code path for both
WIN32 and LXW.
[2] requires changing memory access granularity from 4096 bytes to a
single byte, and is therefore extremely costly.
[3] introduces a whole new class of security vulnerabilities, and will
thus be a lot of fun to watch:-)

All in all taken, then, I'd argue that relying on the current behavior
is very reasonable. If you, too, find the above assessment valid, and
since you mentioned that you were a Microsoft employee, it would be
great if you could make a good-faith effort to have the current behavior
added to the Driver Documentation and thus guaranteed.

PS. this isn't to say that the regex extension should or should not be
used, only that a decision on the matter should not be based on the
undocumentedness of current behavior.

midipix

- some implementations of the REG_STARTEND feature have the nice property
 that they can read past NUL characters. Granted, not all of them do
 (AFAIU one example is FreeBSD itself, the first platform to sport
 REG_STARTEND), but we at least reap the benefit whenever using a regex
 that *can* read past NUL characters.

quoted

In this context, one thing to remember is that the page-size for the mod
operation is 4096, whereas the POSIX page-size (for the purpose of mmap
and mremap) is 65536.

Indeed. A colleague of mine spotted the segfault when diffing a file that
was *exactly* 4,096 bytes.

quoted

Note also that in the case of file-backed mapped sections, using
kernel32.dll or msvcrt.dll or cygwin/newlib or midipix/musl is of little
significance, specifically since all invoke ZwCreateSection and
ZwMapViewOfSection under the hood.

Right. It's all backed by the very same kernel functions.

Ciao,
Johannes