Re: Runaway sed memory use in test on older sed+glibc (was "Re: [PATCH v6 1/3] test: add helper functions for git-bundle")

Ævar Arnfjörð Bjarmason wrote:

On Thu, May 27 2021, Jiang Xin wrote:

quoted

Ævar Arnfjörð Bjarmason [off-list ref] 于2021年5月27日周四
上午2:51写道：

quoted

On Mon, Jan 11 2021, Jiang Xin wrote:

quoted

From: Jiang Xin <redacted>

Move git-bundle related functions from t5510 to a library, and this
lib
will be shared with a new testcase t6020 which finds a known
breakage of
"git-bundle".
[...]
+
+# Format the output of git commands to make a user-friendly and
stable
+# text.  We can easily prepare the expect text without having to
worry
+# about future changes of the commit ID and spaces of the output.
+make_user_friendly_and_stable_output () {
+     sed \
+             -e "s/${A%${A#???????}}[0-9a-f]*/<COMMIT-A>/g" \
+             -e "s/${B%${B#???????}}[0-9a-f]*/<COMMIT-B>/g" \
+             -e "s/${C%${C#???????}}[0-9a-f]*/<COMMIT-C>/g" \
+             -e "s/${D%${D#???????}}[0-9a-f]*/<COMMIT-D>/g" \
+             -e "s/${E%${E#???????}}[0-9a-f]*/<COMMIT-E>/g" \
+             -e "s/${F%${F#???????}}[0-9a-f]*/<COMMIT-F>/g" \
+             -e "s/${G%${G#???????}}[0-9a-f]*/<COMMIT-G>/g" \
+             -e "s/${H%${H#???????}}[0-9a-f]*/<COMMIT-H>/g" \
+             -e "s/${I%${I#???????}}[0-9a-f]*/<COMMIT-I>/g" \
+             -e "s/${J%${J#???????}}[0-9a-f]*/<COMMIT-J>/g" \
+             -e "s/${K%${K#???????}}[0-9a-f]*/<COMMIT-K>/g" \
+             -e "s/${L%${L#???????}}[0-9a-f]*/<COMMIT-L>/g" \
+             -e "s/${M%${M#???????}}[0-9a-f]*/<COMMIT-M>/g" \
+             -e "s/${N%${N#???????}}[0-9a-f]*/<COMMIT-N>/g" \
+             -e "s/${O%${O#???????}}[0-9a-f]*/<COMMIT-O>/g" \
+             -e "s/${P%${P#???????}}[0-9a-f]*/<COMMIT-P>/g" \
+             -e "s/${TAG1%${TAG1#???????}}[0-9a-f]*/<TAG-1>/g" \
+             -e "s/${TAG2%${TAG2#???????}}[0-9a-f]*/<TAG-2>/g" \
+             -e "s/${TAG3%${TAG3#???????}}[0-9a-f]*/<TAG-3>/g" \
+             -e "s/ *\$//"
+}

On one of the gcc farm boxes, a i386 box (gcc45) this fails because
sed
gets killed after >500MB of memory use (I was just eyeballing it in
htop) on the "reate bundle from special rev: main^!" test. This with
GNU
sed 4.2.2.

I suspect this regex pattern creates some runaway behavior in sed
that's
since been fixed (or maybe it's the glibc regex engine?). The glibc is
2.19-18+deb8u10:

    + git bundle list-heads special-rev.bdl
    + make_user_friendly_and_stable_output
    + sed -e s/[0-9a-f]*/<COMMIT-A>/g -e s/[0-9a-f]*/<COMMIT-B>/g -e
s/[0-9a-f]*/<COMMIT-C>/g -e s/[0-9a-f]*/<COMMIT-D>/g -e
s/[0-9a-f]*/<COMMIT-E>/g -e s/[0-9a-f]*/<COMMIT-F>/g -e
s/[0-9a-f]*/<COMMIT-G>/g -e s/[0-9a-f]*/<COMMIT-H>/g -e
s/[0-9a-f]*/<COMMIT-I>/g -e s/[0-9a-f]*/<COMMIT-J>/g -e
s/[0-9a-f]*/<COMMIT-K>/g -e s/[0-9a-f]*/<COMMIT-L>/g -e
s/[0-9a-f]*/<COMMIT-M>/g -e s/[0-9a-f]*/<COMMIT-N>/g -e
s/[0-9a-f]*/<COMMIT-O>/g -e s/[0-9a-f]*/<COMMIT-P>/g -e
s/[0-9a-f]*/<TAG-1>/g -e s/[0-9a-f]*/<TAG-2>/g -e
s/[0-9a-f]*/<TAG-3>/g -e s/ *$//
    sed: couldn't re-allocate memory

I wrote a program on macOS to check memory footprint for sed and perl.
See:

    https://github.com/jiangxin/compare-sed-perl

Interesting use of Go for as a /usr/bin/time -v replacement :)

Here's a Ruby version:
https://dpaste.com/FYT2QKHJE

I'm not sure if will be useful in this particular case, but Ruby code
always ends up simpler ;)

-- 
Felipe Contreras