[PATCH 1/2] mailsplit: Remove any '>' characters used to escape From_ lines in mbox.

Subsystems: the rest

DORMANTno replies

2 messages, 1 author, 2016-06-15 · open the first message on its own page

[PATCH 1/2] mailsplit: Remove any '>' characters used to escape From_ lines in mbox.

From: Carl Worth <hidden>
Date: 2016-06-15 22:48:56

In order to encode an email message in an mbox, a client must notice any
lines in the email body that look like so-called From_ lines, (that is
lines begin with "From "), and add a preceding '>' character.
From Jonathan de Boyne Pollard[*] we learn of two long-standing (since 1995
at least) conventions used for this escaping. The original "mboxo" format
does only the escaping described above, which leads to unavoidable
corruption of some messages. The newer "mboxrd" format also adds a '>' to
any line originally beginning with one or more '>' characters followed by
"From ". This ensures that the original email can be extracted without
corruption.

Git wasn't formerly un-escaping these lines in any case, so invocations of
"git am" would lead to errant '>' characters in the commit message. Here,
we now fix git-mailsplit to perform the necessary un-escaping. We assume
mboxrd format, since designing for the original mboxo format would
guarantee corruption in at least some cases.

[*] http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/mail-mbox-formats.html

Signed-off-by: Carl Worth <redacted>
---
 builtin/mailsplit.c |   26 +++++++++++++++++++++++++-
 1 files changed, 25 insertions(+), 1 deletions(-)
diff --git a/builtin/mailsplit.c b/builtin/mailsplit.c
index cdfc1b7..a3fb9f7 100644
--- a/builtin/mailsplit.c
+++ b/builtin/mailsplit.c
@@ -46,6 +46,30 @@ static int is_from_line(const char *line, int len)
 static struct strbuf buf = STRBUF_INIT;
 static int keep_cr;
 
+/* Write the line in 'buf' to 'output', but if we are splitting an mbox,
+ * then remove the first '>' from any line that begins with one or more
+ * '>' characters followed by "From ".
+ *
+ * Return 0 if successful, 1 for any write error.
+ */
+static int write_buf_unescaping(FILE *output, int is_mbox)
+{
+	const char *line = buf.buf;
+	size_t len = buf.len;
+
+	if (is_mbox && *line == '>') {
+		const char *s = line;
+		while (*s == '>')
+			s++;
+		if (strncmp (s, "From ", 5) == 0) {
+			line = line + 1;
+			len = len - 1;
+		}
+	}
+
+	return fwrite(line, 1, len, output) != len;
+}
+
 /* Called with the first line (potentially partial)
  * already in buf[] -- normally that should begin with
  * the Unix "From " line.  Write it into the specified
@@ -76,7 +100,7 @@ static int split_one(FILE *mbox, const char *name, int allow_bare)
 			strbuf_addch(&buf, '\n');
 		}
 
-		if (fwrite(buf.buf, 1, buf.len, output) != buf.len)
+		if (write_buf_unescaping(output, !is_bare))
 			die_errno("cannot write output");
 
 		if (strbuf_getwholeline(&buf, mbox, '\n')) {
-- 
1.7.0.4

[PATCH 2/2] Add test from From_-line escaping.

From: Carl Worth <hidden>
Date: 2016-06-15 22:48:56

As implemented in the previous commit. We test that when applying from an
mbox that all escaped From_ lines are properly unescaped. We also test that
when applying from an email message the unescaping does not occur.

Signed-off-by: Carl Worth <redacted>
---
 t/t4152-am-From_.sh |   64 +++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 64 insertions(+), 0 deletions(-)
 create mode 100755 t/t4152-am-From_.sh
diff --git a/t/t4152-am-From_.sh b/t/t4152-am-From_.sh
new file mode 100755
index 0000000..02821ee
--- /dev/null
+++ b/t/t4152-am-From_.sh
@@ -0,0 +1,64 @@
+#!/bin/sh
+
+test_description='git am properly unescaping From_ lines'
+
+. ./test-lib.sh
+
+cat >msg <<EOF
+From_ lines
+
+This is a commit message that contains a From_ line, which is line
+that begins with the characters "From ". Get ready for it, now...
+From this time forward, we'll have no From_-line bugs.
+
+Additionally, we'll also test lines that are escaped versions of From_
+lines. These are lines that begin with one or more '>' characters that
+are then followed by the characters "From ". We want to ensure that
+none of these intentional '>' characters get swallowed. Let's try that
+with three variations, (with 1, 2, and 3 leading '>' characters):
+
+>From now on (with one leading '>')
+>>From there to here (with two leading '>' characters)
+>>>From Here to Eternity (with three leading '>' characters)
+
+EOF
+
+test_expect_success setup '
+	echo hello >file &&
+	git add file &&
+	test_tick &&
+	git commit -m first &&
+	git tag first &&
+	echo world >>file &&
+	git add file &&
+	test_tick &&
+	git commit -s -F msg &&
+	git tag second &&
+	git format-patch --stdout first | sed -e "1{p;d};s/^\(>*From \)/>\1/" > From_ &&
+	{
+		echo "X-Fake-Field: Line One" &&
+		echo "X-Fake-Field: Line Two" &&
+		echo "X-Fake-Field: Line Three" &&
+		git format-patch --stdout first | sed -e "1d"
+	} > From_.eml
+'
+
+test_expect_success 'am unescapes From_ lines from mbox' '
+	git checkout first &&
+	git am From_ &&
+	! test -d .git/rebase-apply &&
+	test -z "$(git diff second)" &&
+	test "$(git rev-parse second)" = "$(git rev-parse HEAD)" &&
+	test "$(git rev-parse second^)" = "$(git rev-parse HEAD^)"
+'
+
+test_expect_success 'am does not unescape From_ lines from email' '
+	git checkout first &&
+	git am From_.eml &&
+	! test -d .git/rebase-apply &&
+	test -z "$(git diff second)" &&
+	test "$(git rev-parse second)" = "$(git rev-parse HEAD)" &&
+	test "$(git rev-parse second^)" = "$(git rev-parse HEAD^)"
+'
+
+test_done
-- 
1.7.0.4
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help