[PATCH] regexec: work around macOS TRE memory leak on invalid UTF-8

On macOS the system regex engine (TRE) leaks the buffer it allocates for
a match (in tre_tnfa_run_parallel()) whenever regexec() encounters an
invalid multibyte sequence in a UTF-8 locale: it returns REG_ILLSEQ
without freeing that buffer.  Because regexec_buf() is called once per
line, grepping a file that mixes text with binary data (for example a PDF
checked into a repository) leaks a buffer per line.  The leaked buffer is
sized in proportion to the line length and the pattern's automaton, so a
large case-insensitive alternation over long binary lines can leak
gigabytes in a single command; this has been observed to exhaust memory
and trigger a kernel watchdog panic on the machine running "git grep".

A line that contains an invalid multibyte sequence can still contain
real matches: when a match lies on either side of the invalid bytes, TRE
finds it, returns REG_OK, and frees the buffer normally.  So the fix
cannot simply reject or truncate such a line -- that would silently drop
matches the platform engine itself would report.  Nor can it search the
whole line first and fall back to segmentation only on REG_ILLSEQ: by the
time regexec() returns REG_ILLSEQ it has already leaked, so the fallback
cannot prevent it.

Add a Darwin-only regexec_buf() (compat/regexec.c) that never hands the
platform matcher an invalid byte.  It walks the buffer, splits it at each
invalid sequence, and searches each maximal run of valid text between
them, returning the first match.  Each segment is searched with
REG_STARTEND (whose offsets are relative to the buffer, so no translation
is needed) and with REG_NOTBOL / REG_NOTEOL suppressed only when the
segment does not reach the true start or end of the buffer, so "^" and
"$" keep matching exactly where they would on the whole line.  The split
points are chosen with mbrtowc(), so the buffer is cut at exactly the
bytes the platform's own decoder -- and therefore TRE, which decodes the
same way -- rejects.

The split runs only where a byte can be invalid.  In a single-byte
locale (MB_CUR_MAX == 1) nothing is invalid, so the whole buffer is
searched as before and behavior is unchanged.  The leak is not specific
to UTF-8 -- any multibyte locale drives TRE through the same REG_ILLSEQ
path -- so the guard keys on MB_CUR_MAX rather than the codeset name,
which is also simpler.  MB_CUR_MAX reflects the LC_CTYPE the C library
installed, the same locale mbrtowc() decodes against.

regexec_buf() backs every regex caller, so this affects "git grep",
diffcore-pickaxe (-G/-S --pickaxe-regex) and diff word splitting alike.
The fix does not reproduce the platform's exact output on lines with
invalid bytes -- doing so would mean deliberately hiding matches that are
really there.  No match the platform engine reported before is lost, and
every match reported now is a genuine one: each segment is searched as
valid text at its real offset, so nothing spurious is introduced.
Unanchored matches -- which the engine already found, even next to
invalid bytes -- are unchanged.  The only additional matches are
zero-width ones at a true line boundary next to invalid bytes: a "^",
"$", or word boundary that holds there now matches, where the engine used
to hit the invalid byte, return REG_ILLSEQ, and report nothing even
though the match is genuine.

The workaround is compiled in only when the platform's own regex engine
is used, across the Makefile, Meson and CMake builds: it is skipped
whenever the bundled engine (which has no REG_ILLSEQ and does not leak)
is selected by NO_REGEX -- as it is under AddressSanitizer in the
Makefile and Meson builds.  compat/regexec.c guards its body on the same
macro, so a build that does not enable the workaround compiles it to
nothing rather than colliding with the inline regexec_buf().  The
workaround avoids the leaking path rather than depending on it, so it
stays correct if the platform is fixed and can be dropped once a macOS
without the leak is the minimum supported baseline.

A regression test cannot bound a process's memory portably -- setrlimit
RLIMIT_AS is not enforced on macOS -- so t7810 checks the observable
behavior instead: valid text matches on both sides of an invalid byte
and in a run between two invalid sequences, and "^" and "$" do not match
across an invalid byte in mid-line.  A further check, guarded by the
MACOS prerequisite because only macOS runs this code, confirms that "^"
and "$" match at the true start and end of such a line.  (A line that is
entirely invalid bytes still does not match an unanchored empty-width
pattern such as "^", as before.)

Signed-off-by: Chungmin Lee <redacted>
---
This came out of a real incident: "git grep -i" over a repository that
contains PDFs exhausted memory on an otherwise idle Mac mini and took the
machine down with a kernel watchdog panic ("no checkins from watchdogd").
The leak is in the system regex engine, not in git, but git is what
drives it into the leaking path, once per line.

Why this belongs in regexec_buf():

  - regexec_buf() already exists as the place to wrap platform regexec()
    quirks.  It was added in v2.11.0 (backported to v2.10.1) for exactly
    this kind of reason -- "a regexec_buf() helper that takes a <ptr,len>
    pair with REG_STARTEND extension" (RelNotes/2.11.0).

  - git already treats REG_ILLSEQ on invalid UTF-8 as a cross-platform
    reality -- "As FreeBSD is not the only platform whose regexp library
    reports a REG_ILLSEQ error when fed invalid UTF-8..."
    (RelNotes/2.28.0).  This patch keeps returning the matches around the
    invalid bytes, and just stops leaking on the way there.

On the behavior change: the fix does not reproduce the platform engine's
exact output on a line with invalid bytes, because that output is "leak,
then report nothing".  It never drops a match the engine reported before
and never invents a spurious one; the only additional matches are
zero-width assertions at a true line boundary next to the invalid bytes
(the log describes the exact envelope).

Reproducing (macOS, UTF-8 locale):

    perl -e 'print "\377" x 16, "\n" for 1..300000' >binfile
    git init -q r && mv binfile r && git -C r add binfile &&
        git -C r commit -qm x
    P='aa|bb|cc|dd|ee|ff|gg|hh|ii|jj|kk|ll|mm|nn|oo|pp|qq|rr|ss|tt'
    /usr/bin/time -l git -C r grep -i -E "$P" >/dev/null

  Max RSS on this machine (grep is multithreaded, so the stock figure
  varies run to run):
    stock  git 2.50.1 (Apple):  ~1-2 GiB
    patched:                    ~13 MiB
  The gap grows without bound with the number of invalid-byte lines and
  the size of the pattern's automaton; the original incident is believed
  to have reached tens of GiB before the watchdog panic.

I also have a standalone, self-verifying reproducer that measures the
heap directly (malloc_zone_statistics(), no external tools); it reports
~3072 bytes leaked per REG_ILLSEQ call, allocated in
tre_tnfa_run_parallel() (confirmed with MallocStackLogging + leaks(1)).
I can post it, and I have prepared it to file with Apple as well.

Environment:
    macOS 26.5 (build 25F71), Darwin 25.5.0 arm64, Apple M4
    Apple clang 21.0.0; /usr/bin/git 2.50.1 (Apple Git-155)

Tested: t7810 passes (268/268), including the invalid-byte cases added
here.  (t7812 exits clean but its cases are all skipped on this machine
for lack of a suitable GETTEXT_LOCALE; it exercises the system-regex path
on platforms that have one.)  Built with the platform regex engine on
macOS; NO_REGEX builds, and Makefile/Meson ASAN builds, use the bundled
engine and skip the workaround.

 Makefile                            |   4 ++
 compat/regexec.c                    | 108 ++++++++++++++++++++++++++++
 config.mak.uname                    |   1 +
 contrib/buildsystems/CMakeLists.txt |   5 ++
 git-compat-util.h                   |   5 ++
 meson.build                         |   7 ++
 t/t7810-grep.sh                     |  28 ++++++++
 7 files changed, 158 insertions(+)
 create mode 100644 compat/regexec.c

diff --git a/Makefile b/Makefile
index 1cec251f4..b568dde52 100644
--- a/Makefile
+++ b/Makefile

@@ -2264,6 +2264,10 @@ ifdef USE_ENHANCED_BASIC_REGULAR_EXPRESSIONS
 	COMPAT_CFLAGS += -DUSE_ENHANCED_BASIC_REGULAR_EXPRESSIONS
 	COMPAT_OBJS += compat/regcomp_enhanced.o
 endif
+ifdef DARWIN_TRE_REGEXEC_LEAK_WORKAROUND
+	COMPAT_OBJS += compat/regexec.o
+	BASIC_CFLAGS += -DREGEXEC_MAY_LEAK_ON_ILLSEQ
+endif
 endif
 ifdef NATIVE_CRLF
 	BASIC_CFLAGS += -DNATIVE_CRLF

diff --git a/compat/regexec.c b/compat/regexec.c
new file mode 100644
index 000000000..0677162a8
--- /dev/null
+++ b/compat/regexec.c

@@ -0,0 +1,108 @@
+#include "git-compat-util.h"
+
+#ifdef REGEXEC_MAY_LEAK_ON_ILLSEQ
+
+#include <wchar.h>
+
+/*
+ * macOS's libc regex engine (TRE) leaks the buffer it allocates for a
+ * match whenever regexec() encounters an invalid multibyte sequence in
+ * a multibyte locale: it returns REG_ILLSEQ without freeing that buffer.
+ * A single "git grep" over a file with binary data can call regexec()
+ * once per line and leak gigabytes, which has been observed to exhaust
+ * memory and trigger a kernel watchdog panic.
+ *
+ * The leak happens inside regexec() before it returns, so reacting to
+ * REG_ILLSEQ cannot avoid it: the invalid bytes must never reach the
+ * matcher.  Split the buffer at each invalid sequence and search the
+ * surrounding runs of valid text separately.  A match on either side of
+ * the invalid bytes is still found (the same result the matcher gives on
+ * valid input), but the leaking REG_ILLSEQ path is never reached.
+ *
+ * Use mbrtowc() to decide where to split, so that we split at exactly the
+ * bytes the platform's own decoder -- and thus the regex engine, which
+ * decodes the same way -- rejects.  A hand-rolled validator would
+ * have to guess that boundary; being too lenient reintroduces the leak.
+ */
+
+/*
+ * Search buf[start, end) for a match.  REG_STARTEND reports offsets
+ * relative to buf, so a hit needs no translation.  ^ may only match at
+ * the real start of the buffer and $ only at its real end, so suppress
+ * them when this segment does not reach those boundaries.
+ */
+static int regexec_segment(const regex_t *preg, const char *buf,
+			   size_t start, size_t end, size_t size,
+			   size_t nmatch, regmatch_t pmatch[], int eflags)
+{
+	eflags |= REG_STARTEND;
+	if (start > 0)
+		eflags |= REG_NOTBOL;
+	if (end < size)
+		eflags |= REG_NOTEOL;
+	pmatch[0].rm_so = start;
+	pmatch[0].rm_eo = end;
+	return regexec(preg, buf, nmatch, pmatch, eflags);
+}
+
+int regexec_buf(const regex_t *preg, const char *buf, size_t size,
+		size_t nmatch, regmatch_t pmatch[], int eflags)
+{
+	size_t seg_start = 0, i = 0;
+	mbstate_t mbs;
+
+	assert(nmatch > 0 && pmatch);
+
+	/*
+	 * Only a multibyte locale drives TRE through the leaking multibyte
+	 * path.  In a single-byte locale (MB_CUR_MAX == 1) no byte is
+	 * invalid, so search the whole buffer as before.  MB_CUR_MAX
+	 * reflects the current LC_CTYPE, the same locale mbrtowc() below
+	 * decodes against.
+	 */
+	if (MB_CUR_MAX == 1) {
+		pmatch[0].rm_so = 0;
+		pmatch[0].rm_eo = size;
+		return regexec(preg, buf, nmatch, pmatch, eflags | REG_STARTEND);
+	}
+
+	memset(&mbs, 0, sizeof(mbs));
+	while (i < size) {
+		unsigned char c = (unsigned char)buf[i];
+		size_t n;
+
+		if (c < 0x80) {		/* ASCII fast path */
+			i++;
+			continue;
+		}
+
+		n = mbrtowc(NULL, buf + i, size - i, &mbs);
+		if (!n)			/* embedded NUL decodes to one byte */
+			n = 1;
+		if (n != (size_t)-1 && n != (size_t)-2) {
+			i += n;
+			continue;
+		}
+
+		/* buf[i] begins an invalid sequence; search the run before it */
+		if (i > seg_start) {
+			int ret = regexec_segment(preg, buf, seg_start, i, size,
+						  nmatch, pmatch, eflags);
+			if (ret != REG_NOMATCH)
+				return ret;
+		}
+		i++;			/* skip the invalid byte and resync */
+		seg_start = i;
+		memset(&mbs, 0, sizeof(mbs));
+	}
+
+	/*
+	 * Search the final run.  Do this even when it is empty (a line that
+	 * ends in invalid bytes, or an empty buffer) so that "$" and
+	 * empty-matching patterns still match at the true end of the buffer.
+	 */
+	return regexec_segment(preg, buf, seg_start, size, size,
+			       nmatch, pmatch, eflags);
+}
+
+#endif /* REGEXEC_MAY_LEAK_ON_ILLSEQ */

diff --git a/config.mak.uname b/config.mak.uname
index 9ebd24037..2402a2449 100644
--- a/config.mak.uname
+++ b/config.mak.uname

@@ -154,6 +154,7 @@ ifeq ($(uname_S),Darwin)
 	HAVE_DEV_TTY = YesPlease
 	COMPAT_OBJS += compat/precompose_utf8.o
 	BASIC_CFLAGS += -DPRECOMPOSE_UNICODE
+	DARWIN_TRE_REGEXEC_LEAK_WORKAROUND = YesPlease
 	BASIC_CFLAGS += -DPROTECT_HFS_DEFAULT=1
 	HAVE_BSD_SYSCTL = YesPlease
 	FREAD_READS_DIRECTORIES = UnfortunatelyYes

diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
index a57c4b464..5cfb48f78 100644
--- a/contrib/buildsystems/CMakeLists.txt
+++ b/contrib/buildsystems/CMakeLists.txt

@@ -519,6 +519,11 @@ if(NOT HAVE_REGEX)
 	include_directories(${CMAKE_SOURCE_DIR}/compat/regex)
 	list(APPEND compat_SOURCES compat/regex/regex.c )
 	add_compile_definitions(NO_REGEX NO_MBSUPPORT GAWK)
+elseif(APPLE)
+	# macOS's system regex engine (TRE) leaks memory when regexec()
+	# hits invalid UTF-8; work around it via compat/regexec.c.
+	list(APPEND compat_SOURCES compat/regexec.c)
+	add_compile_definitions(REGEXEC_MAY_LEAK_ON_ILLSEQ)
 endif()
 
 

diff --git a/git-compat-util.h b/git-compat-util.h
index 880977640..3861c9353 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h

@@ -992,6 +992,10 @@ static inline int strtol_i(char const *s, int base, int *result)
 #error "Git requires REG_STARTEND support. Compile with NO_REGEX=NeedsStartEnd"
 #endif
 
+#ifdef REGEXEC_MAY_LEAK_ON_ILLSEQ
+int regexec_buf(const regex_t *preg, const char *buf, size_t size,
+		size_t nmatch, regmatch_t pmatch[], int eflags);
+#else
 static inline int regexec_buf(const regex_t *preg, const char *buf, size_t size,
 			      size_t nmatch, regmatch_t pmatch[], int eflags)
 {

@@ -1000,6 +1004,7 @@ static inline int regexec_buf(const regex_t *preg, const char *buf, size_t size,
 	pmatch[0].rm_eo = size;
 	return regexec(preg, buf, nmatch, pmatch, eflags | REG_STARTEND);
 }
+#endif
 
 #ifdef USE_ENHANCED_BASIC_REGULAR_EXPRESSIONS
 int git_regcomp(regex_t *preg, const char *pattern, int cflags);

diff --git a/meson.build b/meson.build
index 3247697f7..2ce37b607 100644
--- a/meson.build
+++ b/meson.build

@@ -1387,6 +1387,13 @@ if not get_option('b_sanitize').contains('address') and get_option('regex').allo
     libgit_c_args += '-DUSE_ENHANCED_BASIC_REGULAR_EXPRESSIONS'
     compat_sources += 'compat/regcomp_enhanced.c'
   endif
+
+  # macOS's system regex engine (TRE) leaks memory when regexec()
+  # hits invalid UTF-8; work around it via compat/regexec.c.
+  if host_machine.system() == 'darwin'
+    libgit_c_args += '-DREGEXEC_MAY_LEAK_ON_ILLSEQ'
+    compat_sources += 'compat/regexec.c'
+  endif
 elif not get_option('regex').enabled()
   libgit_c_args += [
     '-DNO_REGEX',

diff --git a/t/t7810-grep.sh b/t/t7810-grep.sh
index d61c4a4d7..b325a2e11 100755
--- a/t/t7810-grep.sh
+++ b/t/t7810-grep.sh

@@ -89,6 +89,8 @@ test_expect_success setup '
 	function dummy() {}
 	EOF
 	printf "\200\nASCII\n" >invalid-utf8 &&
+	printf "before\346world\n" >invalid-utf8-embedded &&
+	printf "a\346b\347c\n" >invalid-utf8-multi &&
 	if test_have_prereq FUNNYNAMES
 	then
 		echo unusual >"\"unusual\" pathname" &&

@@ -595,6 +597,32 @@ test_expect_success MB_REGEX 'grep two chars in single-char multibyte file' '
 	LC_ALL=en_US.UTF-8 test_expect_code 1 git grep ".." reverse-question-mark
 '
 
+test_expect_success MB_REGEX 'grep matches valid text on both sides of invalid UTF-8' '
+	LC_ALL=en_US.UTF-8 git grep -h before invalid-utf8-embedded >actual &&
+	test_cmp invalid-utf8-embedded actual &&
+	LC_ALL=en_US.UTF-8 git grep -h world invalid-utf8-embedded >actual &&
+	test_cmp invalid-utf8-embedded actual
+'
+
+test_expect_success MB_REGEX 'grep matches a run between two invalid sequences' '
+	LC_ALL=en_US.UTF-8 git grep -h b invalid-utf8-multi >actual &&
+	test_cmp invalid-utf8-multi actual
+'
+
+test_expect_success MB_REGEX 'grep does not anchor ^ or $ inside an invalid-byte line' '
+	test_expect_code 1 env LC_ALL=en_US.UTF-8 \
+		git grep -h "^world" invalid-utf8-embedded &&
+	test_expect_code 1 env LC_ALL=en_US.UTF-8 \
+		git grep -h "before\$" invalid-utf8-embedded
+'
+
+test_expect_success MACOS,MB_REGEX 'grep anchors ^ and $ at true line ends past invalid UTF-8' '
+	LC_ALL=en_US.UTF-8 git grep -h "^before" invalid-utf8-embedded >actual &&
+	test_cmp invalid-utf8-embedded actual &&
+	LC_ALL=en_US.UTF-8 git grep -h "world\$" invalid-utf8-embedded >actual &&
+	test_cmp invalid-utf8-embedded actual
+'
+
 cat >expected <<EOF
 file
 EOF

base-commit: e9019fcafe0040228b8631c30f97ae1adb61bcdc
-- 
2.55.0