html userdiff is not showing all my changes

Subsystems: the rest

9 messages, 4 authors, 2016-06-15 · open the first message on its own page

html userdiff is not showing all my changes

From: Scott Johnson <hidden>
Date: 2016-06-15 22:50:15

I am attempting to do a word diff of an html source file. Part of the removed 
html is disappearing from the diff when I enable the fancy html word diff.

Here's the output from basic `git diff`:
diff --git a/adv_layout_source.html b/adv_layout_source.html
index 18a81dd..c4ed609 100644
--- a/adv_layout_source.html
+++ b/adv_layout_source.html
@@ -42,8 +42,8 @@
       <ul>
         <li class="ydn-patterns"><em></em><a href="#">ydn-patterns</a></li>
         <li class="ydn-mail"><em></em><a href="#">ydn-mail</a></li>
-        <li class="yws-maps"><em></em><a href="#">yws-maps</a></li>
-        <li class="ydn-delicious"><em></em><a href="#">ydn-delicious</a></li>
+        <li><em></em><a href="#">yws-maps</a></li>
+        <li><em></em><a href="#">ydn-delicious</a></li>
         <li class="yws-flickr"><em></em><a href="#">yws-flickr</a></li>
         <li class="yws-events"><em></em><a href="#">yws-events</a></li>
       </ul>

Here's the default `git diff --word-diff`:
diff --git a/adv_layout_source.html b/adv_layout_source.html
index 18a81dd..c4ed609 100644
--- a/adv_layout_source.html
+++ b/adv_layout_source.html
@@ -42,8 +42,8 @@
      <ul>
        <li class="ydn-patterns"><em></em><a href="#">ydn-patterns</a></li>
        <li class="ydn-mail"><em></em><a href="#">ydn-mail</a></li>
        [-<li class="yws-maps"><em></em><a-]{+<li><em></em><a+} 
href="#">yws-maps</a></li>
        [-<li class="ydn-delicious"><em></em><a-]{+<li><em></em><a+} 
href="#">ydn-delicious</a></li>
        <li class="yws-flickr"><em></em><a href="#">yws-flickr</a></li>
        <li class="yws-events"><em></em><a href="#">yws-events</a></li>
      </ul>

Which is correct, but less than ideal because it highlights much more than the 
actual changes.

So I create a .gitattributes file with one line:
*.html diff=html

And rerun `git diff --word-diff`:
diff --git a/adv_layout_source.html b/adv_layout_source.html
index 18a81dd..c4ed609 100644
--- a/adv_layout_source.html
+++ b/adv_layout_source.html
@@ -42,8 +42,8 @@
      <ul>
        <li class="ydn-patterns"><em></em><a href="#">ydn-patterns</a></li>
        <li class="ydn-mail"><em></em><a href="#">ydn-mail</a></li>
        <li[-class="yws-maps"-]><em></em><a href="#">yws-maps</a></li>
        <li><em></em><a href="#">ydn-delicious</a></li>
        <li class="yws-flickr"><em></em><a href="#">yws-flickr</a></li>
        <li class="yws-events"><em></em><a href="#">yws-events</a></li>
      </ul>
Yikes! What happened to the second line of changes? The removed code is not 
displayed at all.

This is running git 1.7.3.3.

I suspect the problem is in the html patterns in userdiff.c, but I don't 
understand the word-diff-regex well enough to fix it.

Re: html userdiff is not showing all my changes

From: Michael J Gruber <hidden>
Date: 2016-06-15 22:50:15

Scott Johnson venit, vidit, dixit 15.12.2010 04:47:
quoted hunk
I am attempting to do a word diff of an html source file. Part of the removed 
html is disappearing from the diff when I enable the fancy html word diff.

Here's the output from basic `git diff`:
diff --git a/adv_layout_source.html b/adv_layout_source.html
index 18a81dd..c4ed609 100644
--- a/adv_layout_source.html
+++ b/adv_layout_source.html
@@ -42,8 +42,8 @@
       <ul>
         <li class="ydn-patterns"><em></em><a href="#">ydn-patterns</a></li>
         <li class="ydn-mail"><em></em><a href="#">ydn-mail</a></li>
-        <li class="yws-maps"><em></em><a href="#">yws-maps</a></li>
-        <li class="ydn-delicious"><em></em><a href="#">ydn-delicious</a></li>
+        <li><em></em><a href="#">yws-maps</a></li>
+        <li><em></em><a href="#">ydn-delicious</a></li>
         <li class="yws-flickr"><em></em><a href="#">yws-flickr</a></li>
         <li class="yws-events"><em></em><a href="#">yws-events</a></li>
       </ul>

Here's the default `git diff --word-diff`:
diff --git a/adv_layout_source.html b/adv_layout_source.html
index 18a81dd..c4ed609 100644
--- a/adv_layout_source.html
+++ b/adv_layout_source.html
@@ -42,8 +42,8 @@
      <ul>
        <li class="ydn-patterns"><em></em><a href="#">ydn-patterns</a></li>
        <li class="ydn-mail"><em></em><a href="#">ydn-mail</a></li>
        [-<li class="yws-maps"><em></em><a-]{+<li><em></em><a+} 
href="#">yws-maps</a></li>
        [-<li class="ydn-delicious"><em></em><a-]{+<li><em></em><a+} 
href="#">ydn-delicious</a></li>
        <li class="yws-flickr"><em></em><a href="#">yws-flickr</a></li>
        <li class="yws-events"><em></em><a href="#">yws-events</a></li>
      </ul>

Which is correct, but less than ideal because it highlights much more than the 
actual changes.

So I create a .gitattributes file with one line:
*.html diff=html

And rerun `git diff --word-diff`:
diff --git a/adv_layout_source.html b/adv_layout_source.html
index 18a81dd..c4ed609 100644
--- a/adv_layout_source.html
+++ b/adv_layout_source.html
@@ -42,8 +42,8 @@
      <ul>
        <li class="ydn-patterns"><em></em><a href="#">ydn-patterns</a></li>
        <li class="ydn-mail"><em></em><a href="#">ydn-mail</a></li>
        <li[-class="yws-maps"-]><em></em><a href="#">yws-maps</a></li>
        <li><em></em><a href="#">ydn-delicious</a></li>
        <li class="yws-flickr"><em></em><a href="#">yws-flickr</a></li>
        <li class="yws-events"><em></em><a href="#">yws-events</a></li>
      </ul>
Yikes! What happened to the second line of changes? The removed code is not 
displayed at all.

This is running git 1.7.3.3.

I suspect the problem is in the html patterns in userdiff.c, but I don't 
understand the word-diff-regex well enough to fix it.
The wordRegex should really only control what comprises a word, i.e. the
granularity of --word-diff. (Where do we insert additional line-breaks
before running ordinary diff?)

If a wordRegex can make parts of diff disappear than there is problem
deeper in the diff machinery. Can you trim this down to a minimal example?

Michael

Re: html userdiff is not showing all my changes

From: Matthijs Kooijman <hidden>
Date: 2016-06-15 22:50:15

Hi Michael,
If a wordRegex can make parts of diff disappear than there is problem
deeper in the diff machinery.
It can do exactly that. The word regex determines what is a word, but
everything else is counted as "whitespace". The word diff view shows
only differences in words, not in whitespace (which is intentional,
since whitespace changes in things like LaTeX or HTML are not
interesting). Note that it doesn't show whitespace _differences_, but it
does show the whitespace itself (taken from the "new" version of the
file).


So, if the word regex somehow doesn't match the second line at all (or
at least not the differen part), the differences could get ignored.
Can you trim this down to a minimal example?
That would be useful in any case.

Gr.

Matthijs

Re: html userdiff is not showing all my changes

From: Michael J Gruber <hidden>
Date: 2016-06-15 22:50:15

Matthijs Kooijman venit, vidit, dixit 15.12.2010 10:12:
Hi Michael,
quoted
If a wordRegex can make parts of diff disappear than there is problem
deeper in the diff machinery.
It can do exactly that. The word regex determines what is a word, but
everything else is counted as "whitespace". The word diff view shows
only differences in words, not in whitespace (which is intentional,
since whitespace changes in things like LaTeX or HTML are not
interesting). Note that it doesn't show whitespace _differences_, but it
does show the whitespace itself (taken from the "new" version of the
file).
Yep, I just found out myself experimenting with a wordRegex for csv.
Seems like quite a "Gimme rope" feature...

So, it's the regex.
So, if the word regex somehow doesn't match the second line at all (or
at least not the differen part), the differences could get ignored.
quoted
Can you trim this down to a minimal example?
That would be useful in any case.
What strikes me is that both lines are semantically identical, yet one
is treated correctly and the other isn't.

Michael

[PATCH 1/4] diff.c: pass struct diff_words into find_word_boundaries

From: Thomas Rast <hidden>
Date: 2016-06-15 22:50:15

We need the word_regex_check member.  Instead of adding another
argument, just pass in the whole struct for future extensibility.

Signed-off-by: Thomas Rast <redacted>
---
 diff.c |   11 ++++++-----
 1 files changed, 6 insertions(+), 5 deletions(-)
diff --git a/diff.c b/diff.c
index a16ce69..8758a51 100644
--- a/diff.c
+++ b/diff.c
@@ -778,12 +778,13 @@ static void fn_out_diff_words_aux(void *priv, char *line, unsigned long len)
 }
 
 /* This function starts looking at *begin, and returns 0 iff a word was found. */
-static int find_word_boundaries(mmfile_t *buffer, regex_t *word_regex,
+static int find_word_boundaries(mmfile_t *buffer, struct diff_words_data *diff_words,
 		int *begin, int *end)
 {
-	if (word_regex && *begin < buffer->size) {
+	if (diff_words->word_regex && *begin < buffer->size) {
 		regmatch_t match[1];
-		if (!regexec(word_regex, buffer->ptr + *begin, 1, match, 0)) {
+		if (!regexec(diff_words->word_regex, buffer->ptr + *begin,
+			     1, match, 0)) {
 			char *p = memchr(buffer->ptr + *begin + match[0].rm_so,
 					'\n', match[0].rm_eo - match[0].rm_so);
 			*end = p ? p - buffer->ptr : match[0].rm_eo + *begin;
@@ -813,7 +814,7 @@ static int find_word_boundaries(mmfile_t *buffer, regex_t *word_regex,
  * in buffer->orig.
  */
 static void diff_words_fill(struct diff_words_buffer *buffer, mmfile_t *out,
-		regex_t *word_regex)
+		struct diff_words_data *diff_words)
 {
 	int i, j;
 	long alloc = 0;
@@ -827,7 +828,7 @@ static void diff_words_fill(struct diff_words_buffer *buffer, mmfile_t *out,
 	buffer->orig_nr = 1;
 
 	for (i = 0; i < buffer->text.size; i++) {
-		if (find_word_boundaries(&buffer->text, word_regex, &i, &j))
+		if (find_word_boundaries(&buffer->text, diff_words, &i, &j))
 			return;
 
 		/* store original boundaries */
-- 
1.7.3.3.807.g6ee1f

[PATCH 2/4] diff.c: implement a sanity check for word regexes

From: Thomas Rast <hidden>
Date: 2016-06-15 22:50:15

Word regexes are a bit of a dangerous beast, since it is easily
possible to not match a non-space part, which is subsequently ignored
for the purposes of emitting the word diff.  This was clearly stated
in the docs, but users still tripped over it.

Implement a safeguard that verifies two basic sanity assumptions:

* The word regex matches anything that is !isspace().

* The word regex does not match '\n'.  (This case is not very harmful,
  but we used to silently cut off at the '\n' which may go against
  user expectations.)

This is configurable via 'diff.wordRegexCheck', and defaults to
'warn'.

Reported-by: Scott Johnson <redacted>
Signed-off-by: Thomas Rast <redacted>
---
 Documentation/config.txt |    8 ++++
 diff.c                   |   93 +++++++++++++++++++++++++++++++++++++++++++--
 diff.h                   |    1 +
 t/t4034-diff-words.sh    |   65 +++++++++++++++++++++++++++++++-
 4 files changed, 161 insertions(+), 6 deletions(-)
diff --git a/Documentation/config.txt b/Documentation/config.txt
index bf9479e..2e033ea 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -897,6 +897,14 @@ diff.wordRegex::
 	sequences that match the regular expression are "words", all other
 	characters are *ignorable* whitespace.
 
+diff.wordRegexCheck::
+	Perform a simple sanity check on matches of the word regex.
+	Currently this check ensures that the word regex matches all
+	non-space characters, and that the word regex does not match a
+	newline.  The setting controls what to do when the check
+	fails: 'false'/'off'/'ignore' ignore, 'true'/'on'/'warn' emit
+	a warning, and 'error' abort with an error message.
+
 fetch.recurseSubmodules::
 	A boolean value which changes the behavior for fetch and pull, the
 	default is to not recursively fetch populated sumodules unless
diff --git a/diff.c b/diff.c
index 8758a51..becefcf 100644
--- a/diff.c
+++ b/diff.c
@@ -22,11 +22,17 @@
 #define FAST_WORKING_DIRECTORY 1
 #endif
 
+#define REGEX_CHECK_UNSET -1
+#define REGEX_CHECK_OFF 0
+#define REGEX_CHECK_WARN 1
+#define REGEX_CHECK_ERROR 2
+
 static int diff_detect_rename_default;
 static int diff_rename_limit_default = 200;
 static int diff_suppress_blank_empty;
 int diff_use_color_default = -1;
 static const char *diff_word_regex_cfg;
+static int diff_word_regex_check_cfg = REGEX_CHECK_UNSET;
 static const char *external_diff_cmd_cfg;
 int diff_auto_refresh_index = 1;
 static int diff_mnemonic_prefix;
@@ -75,6 +81,19 @@ static int git_config_rename(const char *var, const char *value)
 	return git_config_bool(var,value) ? DIFF_DETECT_RENAME : 0;
 }
 
+static int parse_regex_check_level(int *b, const char *k, const char *v)
+{
+	if (v && !strcasecmp(v, "ignore"))
+		*b = REGEX_CHECK_OFF;
+	else if (v && !strcasecmp(v, "warn"))
+		*b = REGEX_CHECK_WARN;
+	else if (v && !strcasecmp(v, "error"))
+		*b = REGEX_CHECK_ERROR;
+	else
+		*b = git_config_bool(k, v);
+	return 1;
+}
+
 /*
  * These are to give UI layer defaults.
  * The core-level commands such as git-diff-files should
@@ -107,6 +126,8 @@ int git_diff_ui_config(const char *var, const char *value, void *cb)
 		return git_config_string(&external_diff_cmd_cfg, var, value);
 	if (!strcmp(var, "diff.wordregex"))
 		return git_config_string(&diff_word_regex_cfg, var, value);
+	if (!strcmp(var, "diff.wordregexcheck"))
+		return parse_regex_check_level(&diff_word_regex_check_cfg, var, value);
 
 	if (!strcmp(var, "diff.ignoresubmodules"))
 		handle_ignore_submodules_arg(&default_diff_options, value);
@@ -777,6 +798,50 @@ static void fn_out_diff_words_aux(void *priv, char *line, unsigned long len)
 	diff_words->last_minus = minus_first;
 }
 
+
+static void check_word_regex_match(struct diff_words_data *diff_words,
+		char *start, int len, int unmatched)
+{
+	int check = diff_words->opt->word_regex_check;
+	void (*report_fn)(const char *, ...);
+
+	if (check == REGEX_CHECK_OFF)
+		return;
+
+	if (check == REGEX_CHECK_WARN)
+		report_fn = warning;
+	else if (check == REGEX_CHECK_ERROR)
+		report_fn = die;
+	else
+		assert(!"expected REGEX_CHECK_WARN or _ERROR");
+
+	if (unmatched) {
+		int i;
+		char *match_str;
+		for (i = 0; i < len; i++) {
+			if (isspace(start[i]))
+				continue;
+			match_str = xmemdupz(start, len);
+			report_fn("The following snippet contains non-space "
+				  "characters, but was not\nmatched by the "
+				  "word regex:\n'%s'\n"
+				  "They would be ignored for the purposes of "
+				  "the diff, which is\nusually not what you want.",
+				  match_str);
+			free(match_str);
+			break;
+		}
+	} else {
+		if (memchr(start, '\n', len)) {
+			char *match_str = xmemdupz(start, len);
+			report_fn("The following word regex match contains a newline "
+				  "and will be truncated there:\n'%s'",
+				  match_str);
+			free(match_str);
+		}
+	}
+}
+
 /* This function starts looking at *begin, and returns 0 iff a word was found. */
 static int find_word_boundaries(mmfile_t *buffer, struct diff_words_data *diff_words,
 		int *begin, int *end)
@@ -785,8 +850,15 @@ static int find_word_boundaries(mmfile_t *buffer, struct diff_words_data *diff_w
 		regmatch_t match[1];
 		if (!regexec(diff_words->word_regex, buffer->ptr + *begin,
 			     1, match, 0)) {
-			char *p = memchr(buffer->ptr + *begin + match[0].rm_so,
-					'\n', match[0].rm_eo - match[0].rm_so);
+			char *prev_start = buffer->ptr + *begin;
+			char *match_start = prev_start + match[0].rm_so;
+			int match_len = match[0].rm_eo - match[0].rm_so;
+			char *p;
+			check_word_regex_match(diff_words, prev_start,
+					       match_start-prev_start, 1);
+			check_word_regex_match(diff_words, match_start,
+					       match_len, 0);
+			p = memchr(match_start, '\n', match_len);
 			*end = p ? p - buffer->ptr : match[0].rm_eo + *begin;
 			*begin += match[0].rm_so;
 			return *begin >= *end;
@@ -829,7 +901,7 @@ static void diff_words_fill(struct diff_words_buffer *buffer, mmfile_t *out,
 
 	for (i = 0; i < buffer->text.size; i++) {
 		if (find_word_boundaries(&buffer->text, diff_words, &i, &j))
-			return;
+			break;
 
 		/* store original boundaries */
 		ALLOC_GROW(buffer->orig, buffer->orig_nr + 1,
@@ -846,6 +918,11 @@ static void diff_words_fill(struct diff_words_buffer *buffer, mmfile_t *out,
 
 		i = j - 1;
 	}
+
+	/* no more boundaries, check any non-matched chunk remaining */
+	if (i < buffer->text.size)
+		check_word_regex_match(diff_words, buffer->text.ptr + i,
+				       buffer->text.size-i, 1);
 }
 
 /* this executes the word diff on the accumulated buffers */
@@ -882,8 +959,8 @@ static void diff_words_show(struct diff_words_data *diff_words)
 
 	memset(&xpp, 0, sizeof(xpp));
 	memset(&xecfg, 0, sizeof(xecfg));
-	diff_words_fill(&diff_words->minus, &minus, diff_words->word_regex);
-	diff_words_fill(&diff_words->plus, &plus, diff_words->word_regex);
+	diff_words_fill(&diff_words->minus, &minus, diff_words);
+	diff_words_fill(&diff_words->plus, &plus, diff_words);
 	xpp.flags = 0;
 	/* as only the hunk header will be parsed, we need a 0-context */
 	xecfg.ctxlen = 0;
@@ -2021,6 +2098,10 @@ static void builtin_diff(const char *name_a,
 				o->word_regex = userdiff_word_regex(two);
 			if (!o->word_regex)
 				o->word_regex = diff_word_regex_cfg;
+			if (o->word_regex_check == REGEX_CHECK_UNSET)
+				o->word_regex_check = diff_word_regex_check_cfg;
+			if (o->word_regex_check == REGEX_CHECK_UNSET)
+				o->word_regex_check = REGEX_CHECK_WARN;
 			if (o->word_regex) {
 				ecbdata.diff_words->word_regex = (regex_t *)
 					xmalloc(sizeof(regex_t));
@@ -2861,6 +2942,8 @@ void diff_setup(struct diff_options *options)
 		options->a_prefix = "a/";
 		options->b_prefix = "b/";
 	}
+
+	options->word_regex_check = REGEX_CHECK_UNSET;
 }
 
 int diff_setup_done(struct diff_options *options)
diff --git a/diff.h b/diff.h
index 165f368..5f6a7be 100644
--- a/diff.h
+++ b/diff.h
@@ -123,6 +123,7 @@ struct diff_options {
 	int stat_width;
 	int stat_name_width;
 	const char *word_regex;
+	int word_regex_check;
 	enum diff_words_type word_diff;
 
 	/* this is set by diffcore for DIFF_FORMAT_PATCH */
diff --git a/t/t4034-diff-words.sh b/t/t4034-diff-words.sh
index 8096d8a..ebe72ce 100755
--- a/t/t4034-diff-words.sh
+++ b/t/t4034-diff-words.sh
@@ -8,7 +8,8 @@ test_expect_success setup '
 
 	git config diff.color.old red &&
 	git config diff.color.new green &&
-	git config diff.color.func magenta
+	git config diff.color.func magenta &&
+	git config diff.wordRegexCheck off
 
 '
 
@@ -331,4 +332,66 @@ test_expect_success '--word-diff=none' '
 
 '
 
+echo abcd > pre
+echo aXYd > post
+
+test_expect_success 'diff.wordRegexCheck="error" catches nonspaces' '
+
+	git config diff.wordRegexCheck error &&
+	test_must_fail git diff --no-index --word-diff-regex="a|d" pre post 2>out &&
+	grep "fatal.*contains non-space characters" out
+
+'
+
+newline="
+"
+
+test_expect_success 'diff.wordRegexCheck="error" catches newlines' '
+
+	git config diff.wordRegexCheck error &&
+	test_must_fail git diff --no-index --word-diff-regex=".|$newline" pre post 2>out &&
+	grep "fatal.*contains a newline" out
+
+'
+
+test_expect_success 'diff.wordRegexCheck="warn" works' '
+
+	git config diff.wordRegexCheck warn &&
+	test_must_fail git diff --no-index --word-diff-regex="a|d" pre post 2>out &&
+	grep "warning.*contains non-space characters" out
+
+'
+
+test_expect_success 'diff.wordRegexCheck="ignore" works' '
+
+	git config diff.wordRegexCheck ignore &&
+	test_must_fail git diff --no-index --word-diff-regex="a|d" pre post 2>out &&
+	! grep "contains non-space characters" out
+
+'
+
+test_expect_success 'diff.wordRegexCheck="false" is like "ignore"' '
+
+	git config diff.wordRegexCheck false &&
+	test_must_fail git diff --no-index --word-diff-regex="a|d" pre post 2>out &&
+	! grep "contains non-space characters" out
+
+'
+
+test_expect_success 'diff.wordRegexCheck="true" is like "warn"' '
+
+	git config diff.wordRegexCheck true &&
+	test_must_fail git diff --no-index --word-diff-regex="a|d" pre post 2>out &&
+	grep "warning.*contains non-space characters" out
+
+'
+
+test_expect_success 'diff.wordRegexCheck unset is like "warn"' '
+
+	git config --unset diff.wordRegexCheck &&
+	test_must_fail git diff --no-index --word-diff-regex="a|d" pre post 2>out &&
+	grep "warning.*contains non-space characters" out
+
+'
+
 test_done
-- 
1.7.3.3.807.g6ee1f

[PATCH 3/4] userdiff: fix typo in ruby word regex

From: Thomas Rast <hidden>
Date: 2016-06-15 22:50:15

The regex had an unclosed ] that pretty much ruined the safeguard
against not matching a non-space char.

Signed-off-by: Thomas Rast <redacted>
---
 userdiff.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/userdiff.c b/userdiff.c
index f9e05b5..4d6433b 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -81,7 +81,7 @@
 	 "(@|@@|\\$)?[a-zA-Z_][a-zA-Z0-9_]*"
 	 "|[-+0-9.e]+|0[xXbB]?[0-9a-fA-F]+|\\?(\\\\C-)?(\\\\M-)?."
 	 "|//=?|[-+*/<>%&^|=!]=|<<=?|>>=?|===|\\.{1,3}|::|[!=]~"
-	 "|[^[:space:]|[\x80-\xff]+"),
+	 "|[^[:space:]]|[\x80-\xff]+"),
 PATTERNS("bibtex", "(@[a-zA-Z]{1,}[ \t]*\\{{0,1}[ \t]*[^ \t\"@',\\#}{~%]*).*$",
 	 "[={}\"]|[^={}\" \t]+"),
 PATTERNS("tex", "^(\\\\((sub)*section|chapter|part)\\*{0,1}\\{.*)$",
-- 
1.7.3.3.807.g6ee1f

[PATCH 0/4] --word-regex sanity checking and such

From: Thomas Rast <hidden>
Date: 2016-06-15 22:50:15

[Forgot the list and Matthijs on the first sending.  Sorry for the
spam!]

Scott Johnson wrote [trimmed and wraps fixed up]:
Here's the default `git diff --word-diff`:
        [-<li class="yws-maps"><em></em><a-]{+<li><em></em><a+} href="#">yws-maps</a></li>
        [-<li class="ydn-delicious"><em></em><a-]{+<li><em></em><a+} href="#">ydn-delicious</a></li>

Which is correct, but less than ideal because it highlights much more than the 
actual changes.

So I create a .gitattributes file with one line:
*.html diff=html

And rerun `git diff --word-diff`:
        <li[-class="yws-maps"-]><em></em><a href="#">yws-maps</a></li>
        <li><em></em><a href="#">ydn-delicious</a></li>

Yikes! What happened to the second line of changes? The removed code is not 
displayed at all.
Michael J Gruber wrote:
Yep, I just found out myself experimenting with a wordRegex for csv.
Seems like quite a "Gimme rope" feature...

So, it's the regex.
Well. Yes. No. Maybe.

Thanks for bringing this to my attention.  I currently have enough
more serious work to avoid that this actually motivated me to hack up
a sanity check.  It's just far too error prone as it is now.

But I cannot reproduce the problem!  I put Scott's two offending lines
(taken from his "straight" diff) into t4034/html/{pre,post}, and I
think the output is valid.  Also, the word regex for html has long
included the |[^[:space:]] safeguard (actually they all do except for
bibtex, which is even more lenient on what it matches).  So you either
found an example that depends on more context (which would be *really*
bad) or there is another source of bad regexes.  Anyway, the safeguard
should easily catch the latter case.

This did unearth a bug in the ruby regex, though, so it's been worth
the trouble.

Various small issues with this patch series:

* [4/4] I stole the html test from Scott's mail, and some of the rest
  from various Wikibooks sources on "Hello World" in each language,
  usually extended by a bit of code that tests the world-splitting
  power.  I hope this is ok with Scott and the Copyright overlords.
  There are only so many ways to spell "Hello World", and only so many
  languages I know myself.

* [4/4] Many patterns do not split 1+2, probably because they stick +2
  together as a signed integer literal, even though I think they
  should.  I ran out of time to investigate however.

* [3/4] was actually detected with the help of [4/4], but putting it
  after would require heavy special casing.

* [2/4] It's a weird idiosyncrasy of the word-diff code that the exit
  status of git-diff does not depend on whether word-diff found any
  differences, and in fact the shown hunks do not either.  So the
  tests are "test_must_fail" regardless of word regex, because the
  input files differ at a byte level.  Maybe at least hunks without
  word differences should be suppressed?


Thomas Rast (4):
  diff.c: pass struct diff_words into find_word_boundaries
  diff.c: implement a sanity check for word regexes
  userdiff: fix typo in ruby word regex
  t4034: bulk verify builtin word regex sanity

 Documentation/config.txt |    8 ++++
 diff.c                   |  104 +++++++++++++++++++++++++++++++++++++++++----
 diff.h                   |    1 +
 t/t4034-diff-words.sh    |   85 +++++++++++++++++++++++++++++++++++++-
 t/t4034/bibtex/expect    |   15 +++++++
 t/t4034/bibtex/post      |   10 ++++
 t/t4034/bibtex/pre       |    9 ++++
 t/t4034/cpp/expect       |   10 ++++
 t/t4034/cpp/post         |    5 ++
 t/t4034/cpp/pre          |    4 ++
 t/t4034/csharp/expect    |   12 +++++
 t/t4034/csharp/post      |    8 ++++
 t/t4034/csharp/pre       |    8 ++++
 t/t4034/fortran/expect   |   12 +++++
 t/t4034/fortran/post     |    7 +++
 t/t4034/fortran/pre      |    7 +++
 t/t4034/html/expect      |    7 +++
 t/t4034/html/post        |    2 +
 t/t4034/html/pre         |    2 +
 t/t4034/java/expect      |   11 +++++
 t/t4034/java/post        |    6 +++
 t/t4034/java/pre         |    6 +++
 t/t4034/objc/expect      |   11 +++++
 t/t4034/objc/post        |    7 +++
 t/t4034/objc/pre         |    7 +++
 t/t4034/pascal/expect    |   12 +++++
 t/t4034/pascal/post      |    7 +++
 t/t4034/pascal/pre       |    7 +++
 t/t4034/php/expect       |    7 +++
 t/t4034/php/post         |    2 +
 t/t4034/php/pre          |    2 +
 t/t4034/python/expect    |    8 ++++
 t/t4034/python/post      |    3 +
 t/t4034/python/pre       |    3 +
 t/t4034/ruby/expect      |    7 +++
 t/t4034/ruby/post        |    2 +
 t/t4034/ruby/pre         |    2 +
 t/t4034/tex/expect       |    9 ++++
 t/t4034/tex/post         |    4 ++
 t/t4034/tex/pre          |    4 ++
 userdiff.c               |    2 +-
 41 files changed, 433 insertions(+), 12 deletions(-)
 create mode 100644 t/t4034/bibtex/expect
 create mode 100644 t/t4034/bibtex/post
 create mode 100644 t/t4034/bibtex/pre
 create mode 100644 t/t4034/cpp/expect
 create mode 100644 t/t4034/cpp/post
 create mode 100644 t/t4034/cpp/pre
 create mode 100644 t/t4034/csharp/expect
 create mode 100644 t/t4034/csharp/post
 create mode 100644 t/t4034/csharp/pre
 create mode 100644 t/t4034/fortran/expect
 create mode 100644 t/t4034/fortran/post
 create mode 100644 t/t4034/fortran/pre
 create mode 100644 t/t4034/html/expect
 create mode 100644 t/t4034/html/post
 create mode 100644 t/t4034/html/pre
 create mode 100644 t/t4034/java/expect
 create mode 100644 t/t4034/java/post
 create mode 100644 t/t4034/java/pre
 create mode 100644 t/t4034/objc/expect
 create mode 100644 t/t4034/objc/post
 create mode 100644 t/t4034/objc/pre
 create mode 100644 t/t4034/pascal/expect
 create mode 100644 t/t4034/pascal/post
 create mode 100644 t/t4034/pascal/pre
 create mode 100644 t/t4034/php/expect
 create mode 100644 t/t4034/php/post
 create mode 100644 t/t4034/php/pre
 create mode 100644 t/t4034/python/expect
 create mode 100644 t/t4034/python/post
 create mode 100644 t/t4034/python/pre
 create mode 100644 t/t4034/ruby/expect
 create mode 100644 t/t4034/ruby/post
 create mode 100644 t/t4034/ruby/pre
 create mode 100644 t/t4034/tex/expect
 create mode 100644 t/t4034/tex/post
 create mode 100644 t/t4034/tex/pre

-- 
1.7.3.3.807.g6ee1f

[PATCH 4/4] t4034: bulk verify builtin word regex sanity

From: Thomas Rast <hidden>
Date: 2016-06-15 22:50:15

The builtin word regexes should be tested with some simple examples
against simple issues, like failing to match a non-space character.

Do this in bulk.  Many of these patterns are a rather ad-hoc
combination of a few simple lines of code, so they can certainly be
improved.  However, they already unearthed a typo in the ruby pattern
(previous commit).

Signed-off-by: Thomas Rast <redacted>
---
 t/t4034-diff-words.sh  |   20 ++++++++++++++++++++
 t/t4034/bibtex/expect  |   15 +++++++++++++++
 t/t4034/bibtex/post    |   10 ++++++++++
 t/t4034/bibtex/pre     |    9 +++++++++
 t/t4034/cpp/expect     |   10 ++++++++++
 t/t4034/cpp/post       |    5 +++++
 t/t4034/cpp/pre        |    4 ++++
 t/t4034/csharp/expect  |   12 ++++++++++++
 t/t4034/csharp/post    |    8 ++++++++
 t/t4034/csharp/pre     |    8 ++++++++
 t/t4034/fortran/expect |   12 ++++++++++++
 t/t4034/fortran/post   |    7 +++++++
 t/t4034/fortran/pre    |    7 +++++++
 t/t4034/html/expect    |    7 +++++++
 t/t4034/html/post      |    2 ++
 t/t4034/html/pre       |    2 ++
 t/t4034/java/expect    |   11 +++++++++++
 t/t4034/java/post      |    6 ++++++
 t/t4034/java/pre       |    6 ++++++
 t/t4034/objc/expect    |   11 +++++++++++
 t/t4034/objc/post      |    7 +++++++
 t/t4034/objc/pre       |    7 +++++++
 t/t4034/pascal/expect  |   12 ++++++++++++
 t/t4034/pascal/post    |    7 +++++++
 t/t4034/pascal/pre     |    7 +++++++
 t/t4034/php/expect     |    7 +++++++
 t/t4034/php/post       |    2 ++
 t/t4034/php/pre        |    2 ++
 t/t4034/python/expect  |    8 ++++++++
 t/t4034/python/post    |    3 +++
 t/t4034/python/pre     |    3 +++
 t/t4034/ruby/expect    |    7 +++++++
 t/t4034/ruby/post      |    2 ++
 t/t4034/ruby/pre       |    2 ++
 t/t4034/tex/expect     |    9 +++++++++
 t/t4034/tex/post       |    4 ++++
 t/t4034/tex/pre        |    4 ++++
 37 files changed, 265 insertions(+), 0 deletions(-)
 create mode 100644 t/t4034/bibtex/expect
 create mode 100644 t/t4034/bibtex/post
 create mode 100644 t/t4034/bibtex/pre
 create mode 100644 t/t4034/cpp/expect
 create mode 100644 t/t4034/cpp/post
 create mode 100644 t/t4034/cpp/pre
 create mode 100644 t/t4034/csharp/expect
 create mode 100644 t/t4034/csharp/post
 create mode 100644 t/t4034/csharp/pre
 create mode 100644 t/t4034/fortran/expect
 create mode 100644 t/t4034/fortran/post
 create mode 100644 t/t4034/fortran/pre
 create mode 100644 t/t4034/html/expect
 create mode 100644 t/t4034/html/post
 create mode 100644 t/t4034/html/pre
 create mode 100644 t/t4034/java/expect
 create mode 100644 t/t4034/java/post
 create mode 100644 t/t4034/java/pre
 create mode 100644 t/t4034/objc/expect
 create mode 100644 t/t4034/objc/post
 create mode 100644 t/t4034/objc/pre
 create mode 100644 t/t4034/pascal/expect
 create mode 100644 t/t4034/pascal/post
 create mode 100644 t/t4034/pascal/pre
 create mode 100644 t/t4034/php/expect
 create mode 100644 t/t4034/php/post
 create mode 100644 t/t4034/php/pre
 create mode 100644 t/t4034/python/expect
 create mode 100644 t/t4034/python/post
 create mode 100644 t/t4034/python/pre
 create mode 100644 t/t4034/ruby/expect
 create mode 100644 t/t4034/ruby/post
 create mode 100644 t/t4034/ruby/pre
 create mode 100644 t/t4034/tex/expect
 create mode 100644 t/t4034/tex/post
 create mode 100644 t/t4034/tex/pre
diff --git a/t/t4034-diff-words.sh b/t/t4034-diff-words.sh
index ebe72ce..b085948 100755
--- a/t/t4034-diff-words.sh
+++ b/t/t4034-diff-words.sh
@@ -394,4 +394,24 @@ test_expect_success 'diff.wordRegexCheck unset is like "warn"' '
 
 '
 
+test_expect_success 'set diff.wordRegexCheck=error for language tests' '
+
+	git config diff.wordRegexCheck error
+
+'
+
+word_diff_for_language () {
+	cp $TEST_DIRECTORY/t4034/$1/pre $TEST_DIRECTORY/t4034/$1/post \
+		$TEST_DIRECTORY/t4034/$1/expect . &&
+	echo "* diff=$1" > .gitattributes &&
+	word_diff --color-words
+}
+
+for lang_dir in $TEST_DIRECTORY/t4034/*; do
+	lang=${lang_dir#$TEST_DIRECTORY/t4034/}
+	test_expect_success "diff driver '$lang' has sane word regex" "
+		word_diff_for_language $lang
+	"
+done
+
 test_done
diff --git a/t/t4034/bibtex/expect b/t/t4034/bibtex/expect
new file mode 100644
index 0000000..a157774
--- /dev/null
+++ b/t/t4034/bibtex/expect
@@ -0,0 +1,15 @@
+<BOLD>diff --git a/pre b/post<RESET>
+<BOLD>index 95cd55b..ddcba9b 100644<RESET>
+<BOLD>--- a/pre<RESET>
+<BOLD>+++ b/post<RESET>
+<CYAN>@@ -1,9 +1,10 @@<RESET>
+@article{aldous1987uie,<RESET>
+  title={{Ultimate instability of exponential back-off protocol for acknowledgment-based transmission control of random access communication channels}},<RESET>
+  author={Aldous, <RED>D.<RESET><GREEN>David<RESET>},
+  journal={Information Theory, IEEE Transactions on},<RESET>
+  volume={<RED>33<RESET><GREEN>Bogus.<RESET>},
+  number={<RED>2<RESET><GREEN>4<RESET>},
+  pages={219--223},<RESET>
+  year=<GREEN>1987,<RESET>
+<GREEN>  note={This is in fact a rather funny read since ethernet works well in practice. The<RESET> {<RED>1987<RESET><GREEN>\em pre} reference is the right one, however.<RESET>}<RED>,<RESET>
+}<RESET>
diff --git a/t/t4034/bibtex/post b/t/t4034/bibtex/post
new file mode 100644
index 0000000..ddcba9b
--- /dev/null
+++ b/t/t4034/bibtex/post
@@ -0,0 +1,10 @@
+@article{aldous1987uie,
+  title={{Ultimate instability of exponential back-off protocol for acknowledgment-based transmission control of random access communication channels}},
+  author={Aldous, David},
+  journal={Information Theory, IEEE Transactions on},
+  volume={Bogus.},
+  number={4},
+  pages={219--223},
+  year=1987,
+  note={This is in fact a rather funny read since ethernet works well in practice. The {\em pre} reference is the right one, however.}
+}
diff --git a/t/t4034/bibtex/pre b/t/t4034/bibtex/pre
new file mode 100644
index 0000000..95cd55b
--- /dev/null
+++ b/t/t4034/bibtex/pre
@@ -0,0 +1,9 @@
+@article{aldous1987uie,
+  title={{Ultimate instability of exponential back-off protocol for acknowledgment-based transmission control of random access communication channels}},
+  author={Aldous, D.},
+  journal={Information Theory, IEEE Transactions on},
+  volume={33},
+  number={2},
+  pages={219--223},
+  year={1987},
+}
diff --git a/t/t4034/cpp/expect b/t/t4034/cpp/expect
new file mode 100644
index 0000000..e529842
--- /dev/null
+++ b/t/t4034/cpp/expect
@@ -0,0 +1,10 @@
+<BOLD>diff --git a/pre b/post<RESET>
+<BOLD>index 5517c3c..17aa265 100644<RESET>
+<BOLD>--- a/pre<RESET>
+<BOLD>+++ b/post<RESET>
+<CYAN>@@ -1,4 +1,5 @@<RESET>
+class Foo : public Thing{
+	Foo() : x(0<RED>&&1<RESET><GREEN>&42<RESET>) {
+		<GREEN>bar(x);<RESET>
+	}
+}<GREEN>;<RESET>
diff --git a/t/t4034/cpp/post b/t/t4034/cpp/post
new file mode 100644
index 0000000..17aa265
--- /dev/null
+++ b/t/t4034/cpp/post
@@ -0,0 +1,5 @@
+class Foo : public Thing{
+	Foo() : x(0&42) {
+		bar(x);
+	}
+};
diff --git a/t/t4034/cpp/pre b/t/t4034/cpp/pre
new file mode 100644
index 0000000..5517c3c
--- /dev/null
+++ b/t/t4034/cpp/pre
@@ -0,0 +1,4 @@
+class Foo:public Thing{
+	Foo():x(0&&1){}
+}
+
diff --git a/t/t4034/csharp/expect b/t/t4034/csharp/expect
new file mode 100644
index 0000000..c8b6b8f
--- /dev/null
+++ b/t/t4034/csharp/expect
@@ -0,0 +1,12 @@
+<BOLD>diff --git a/pre b/post<RESET>
+<BOLD>index 8ff9319..9869fa9 100644<RESET>
+<BOLD>--- a/pre<RESET>
+<BOLD>+++ b/post<RESET>
+<CYAN>@@ -2,7 +2,7 @@<RESET> <RESET><MAGENTA>class Program<RESET>
+{<RESET>
+	static void Main()<RESET>
+	{<RESET>
+		Console.WriteLine("Hello, <GREEN>New<RESET> World!");
+		int i = <RED>5<RESET><GREEN>5+0<RESET>;
+	}<RESET>
+}<RESET>
diff --git a/t/t4034/csharp/post b/t/t4034/csharp/post
new file mode 100644
index 0000000..9869fa9
--- /dev/null
+++ b/t/t4034/csharp/post
@@ -0,0 +1,8 @@
+class Program
+{
+	static void Main()
+	{
+		Console.WriteLine("Hello, New World!");
+		int i = 5+0;
+	}
+}
diff --git a/t/t4034/csharp/pre b/t/t4034/csharp/pre
new file mode 100644
index 0000000..8ff9319
--- /dev/null
+++ b/t/t4034/csharp/pre
@@ -0,0 +1,8 @@
+class Program
+{
+	static void Main()
+	{
+		Console.WriteLine("Hello, World!");
+		int i=5;
+	}
+}
diff --git a/t/t4034/fortran/expect b/t/t4034/fortran/expect
new file mode 100644
index 0000000..5a25663
--- /dev/null
+++ b/t/t4034/fortran/expect
@@ -0,0 +1,12 @@
+<BOLD>diff --git a/pre b/post<RESET>
+<BOLD>index 08b4e5a..fb7cb51 100644<RESET>
+<BOLD>--- a/pre<RESET>
+<BOLD>+++ b/post<RESET>
+<CYAN>@@ -1,7 +1,7 @@<RESET>
+program hello<RESET>
+   print *, "Hello World<RED>!<RESET><GREEN>?<RESET>"
+
+   DO10I = 1,10<RESET>
+   <RED>DO10I<RESET><GREEN>DO 10 I<RESET> = 1,10
+   <RED>DO10I<RESET><GREEN>DO 1 0 I<RESET> = 1,10
+end program hello<RESET>
diff --git a/t/t4034/fortran/post b/t/t4034/fortran/post
new file mode 100644
index 0000000..fb7cb51
--- /dev/null
+++ b/t/t4034/fortran/post
@@ -0,0 +1,7 @@
+program hello
+   print *, "Hello World?"
+
+   DO10I = 1,10
+   DO 10 I = 1,10
+   DO 1 0 I = 1,10
+end program hello
diff --git a/t/t4034/fortran/pre b/t/t4034/fortran/pre
new file mode 100644
index 0000000..08b4e5a
--- /dev/null
+++ b/t/t4034/fortran/pre
@@ -0,0 +1,7 @@
+program hello
+   print *, "Hello World!"
+
+   DO10I = 1,10
+   DO10I = 1,10
+   DO10I = 1,10
+end program hello
diff --git a/t/t4034/html/expect b/t/t4034/html/expect
new file mode 100644
index 0000000..78d28d4
--- /dev/null
+++ b/t/t4034/html/expect
@@ -0,0 +1,7 @@
+<BOLD>diff --git a/pre b/post<RESET>
+<BOLD>index 8bf936a..125bdd5 100644<RESET>
+<BOLD>--- a/pre<RESET>
+<BOLD>+++ b/post<RESET>
+<CYAN>@@ -1,2 +1,2 @@<RESET>
+<li<RED>class="yws-maps"<RESET>><em></em><a href="#">yws-maps</a></li>
+<li<RED>class="ydn-delicious"<RESET>><em></em><a href="#">ydn-delicious</a></li>
diff --git a/t/t4034/html/post b/t/t4034/html/post
new file mode 100644
index 0000000..125bdd5
--- /dev/null
+++ b/t/t4034/html/post
@@ -0,0 +1,2 @@
+<li><em></em><a href="#">yws-maps</a></li>
+<li><em></em><a href="#">ydn-delicious</a></li>
diff --git a/t/t4034/html/pre b/t/t4034/html/pre
new file mode 100644
index 0000000..8bf936a
--- /dev/null
+++ b/t/t4034/html/pre
@@ -0,0 +1,2 @@
+<li class="yws-maps"><em></em><a href="#">yws-maps</a></li>
+<li class="ydn-delicious"><em></em><a href="#">ydn-delicious</a></li>
diff --git a/t/t4034/java/expect b/t/t4034/java/expect
new file mode 100644
index 0000000..9d99523
--- /dev/null
+++ b/t/t4034/java/expect
@@ -0,0 +1,11 @@
+<BOLD>diff --git a/pre b/post<RESET>
+<BOLD>index ae11cd3..fd61213 100644<RESET>
+<BOLD>--- a/pre<RESET>
+<BOLD>+++ b/post<RESET>
+<CYAN>@@ -1,6 +1,6 @@<RESET>
+public class HelloWorld {<RESET>
+   public static void main(String[] args) {<RESET>
+       System.out.println("Hello <RED>,<RESET><GREEN>--<RESET> world!");
+       int i = <RED>1+2<RESET><GREEN>1 + 2<RESET>;
+   }<RESET>
+}<RESET>
diff --git a/t/t4034/java/post b/t/t4034/java/post
new file mode 100644
index 0000000..fd61213
--- /dev/null
+++ b/t/t4034/java/post
@@ -0,0 +1,6 @@
+public class HelloWorld {
+   public static void main(String[] args) {
+       System.out.println("Hello -- world!");
+       int i = 1 + 2;
+   }
+}
diff --git a/t/t4034/java/pre b/t/t4034/java/pre
new file mode 100644
index 0000000..ae11cd3
--- /dev/null
+++ b/t/t4034/java/pre
@@ -0,0 +1,6 @@
+public class HelloWorld {
+   public static void main(String[] args) {
+       System.out.println("Hello, world!");
+       int i = 1+2;
+   }
+}
diff --git a/t/t4034/objc/expect b/t/t4034/objc/expect
new file mode 100644
index 0000000..a29fec5
--- /dev/null
+++ b/t/t4034/objc/expect
@@ -0,0 +1,11 @@
+<BOLD>diff --git a/pre b/post<RESET>
+<BOLD>index 8eb298d..e728a08 100644<RESET>
+<BOLD>--- a/pre<RESET>
+<BOLD>+++ b/post<RESET>
+<CYAN>@@ -2,6 +2,6 @@<RESET>
+ <RESET>
+int main (void)<RESET>
+{<RESET>
+	int i = <RED>1+2<RESET><GREEN>1 + 2<RESET>;
+	printf ("Hello<GREEN>, new<RESET> world!\n");
+}<RESET>
diff --git a/t/t4034/objc/post b/t/t4034/objc/post
new file mode 100644
index 0000000..e728a08
--- /dev/null
+++ b/t/t4034/objc/post
@@ -0,0 +1,7 @@
+#import <stdio.h>
+ 
+int main (void)
+{
+	int i = 1 + 2;
+	printf ("Hello, new world!\n");
+}
diff --git a/t/t4034/objc/pre b/t/t4034/objc/pre
new file mode 100644
index 0000000..8eb298d
--- /dev/null
+++ b/t/t4034/objc/pre
@@ -0,0 +1,7 @@
+#import <stdio.h>
+ 
+int main (void)
+{
+	int i = 1+2;
+	printf ("Hello world!\n");
+}
diff --git a/t/t4034/pascal/expect b/t/t4034/pascal/expect
new file mode 100644
index 0000000..10953cf
--- /dev/null
+++ b/t/t4034/pascal/expect
@@ -0,0 +1,12 @@
+<BOLD>diff --git a/pre b/post<RESET>
+<BOLD>index 7c5fbef..bdd5df9 100644<RESET>
+<BOLD>--- a/pre<RESET>
+<BOLD>+++ b/post<RESET>
+<CYAN>@@ -1,7 +1,7 @@<RESET>
+program HelloWorld;<RESET>
+var<RESET>
+   i : integer;
+begin<RESET>
+   i = i <RED>+1<RESET><GREEN>+ 1<RESET>;
+   writeln('Hello<GREEN>, new<RESET> world!');
+end.<RESET>
diff --git a/t/t4034/pascal/post b/t/t4034/pascal/post
new file mode 100644
index 0000000..bdd5df9
--- /dev/null
+++ b/t/t4034/pascal/post
@@ -0,0 +1,7 @@
+program HelloWorld;
+var
+   i : integer;
+begin
+   i = i + 1;
+   writeln('Hello, new world!');
+end.
diff --git a/t/t4034/pascal/pre b/t/t4034/pascal/pre
new file mode 100644
index 0000000..7c5fbef
--- /dev/null
+++ b/t/t4034/pascal/pre
@@ -0,0 +1,7 @@
+program HelloWorld;
+var
+   i: integer;
+begin
+   i = i+1;
+   writeln('Hello world!');
+end.
diff --git a/t/t4034/php/expect b/t/t4034/php/expect
new file mode 100644
index 0000000..171cfd5
--- /dev/null
+++ b/t/t4034/php/expect
@@ -0,0 +1,7 @@
+<BOLD>diff --git a/pre b/post<RESET>
+<BOLD>index 646a13d..1b1fe22 100644<RESET>
+<BOLD>--- a/pre<RESET>
+<BOLD>+++ b/post<RESET>
+<CYAN>@@ -1,2 +1,2 @@<RESET>
+$i =<RED>$i+1<RESET><GREEN>CONSTANT + 1<RESET>;
+echo "Hello, <GREEN>New<RESET> World!\n";
diff --git a/t/t4034/php/post b/t/t4034/php/post
new file mode 100644
index 0000000..1b1fe22
--- /dev/null
+++ b/t/t4034/php/post
@@ -0,0 +1,2 @@
+$i =CONSTANT + 1;
+echo "Hello, New World!\n";
diff --git a/t/t4034/php/pre b/t/t4034/php/pre
new file mode 100644
index 0000000..646a13d
--- /dev/null
+++ b/t/t4034/php/pre
@@ -0,0 +1,2 @@
+$i=$i+1;
+echo "Hello, World!\n";
diff --git a/t/t4034/python/expect b/t/t4034/python/expect
new file mode 100644
index 0000000..bf6a30b
--- /dev/null
+++ b/t/t4034/python/expect
@@ -0,0 +1,8 @@
+<BOLD>diff --git a/pre b/post<RESET>
+<BOLD>index 2261a37..1076af0 100644<RESET>
+<BOLD>--- a/pre<RESET>
+<BOLD>+++ b/post<RESET>
+<CYAN>@@ -1,3 +1,3 @@<RESET>
+<RED>foo.bar()<RESET><GREEN>foo . bar (stuff)<RESET>
+i = <RED>i+1<RESET><GREEN>i + 1<RESET>
+print "Hello, <GREEN>New<RESET> World!\n"
diff --git a/t/t4034/python/post b/t/t4034/python/post
new file mode 100644
index 0000000..1076af0
--- /dev/null
+++ b/t/t4034/python/post
@@ -0,0 +1,3 @@
+foo . bar (stuff)
+i = i + 1
+print "Hello, New World!\n"
diff --git a/t/t4034/python/pre b/t/t4034/python/pre
new file mode 100644
index 0000000..2261a37
--- /dev/null
+++ b/t/t4034/python/pre
@@ -0,0 +1,3 @@
+foo.bar()
+i = i+1
+print "Hello, World!\n"
diff --git a/t/t4034/ruby/expect b/t/t4034/ruby/expect
new file mode 100644
index 0000000..72ff72b
--- /dev/null
+++ b/t/t4034/ruby/expect
@@ -0,0 +1,7 @@
+<BOLD>diff --git a/pre b/post<RESET>
+<BOLD>index 1961e79..d954376 100644<RESET>
+<BOLD>--- a/pre<RESET>
+<BOLD>+++ b/post<RESET>
+<CYAN>@@ -1,2 +1,2 @@<RESET>
+10.downto(<RED>1<RESET><GREEN>2<RESET>) { |x| puts x }
+puts 'Hello <GREEN>new<RESET> world'
diff --git a/t/t4034/ruby/post b/t/t4034/ruby/post
new file mode 100644
index 0000000..d954376
--- /dev/null
+++ b/t/t4034/ruby/post
@@ -0,0 +1,2 @@
+10.downto(2) { |x| puts x }
+puts 'Hello new world'
diff --git a/t/t4034/ruby/pre b/t/t4034/ruby/pre
new file mode 100644
index 0000000..1961e79
--- /dev/null
+++ b/t/t4034/ruby/pre
@@ -0,0 +1,2 @@
+10.downto(1) {|x| puts x}
+puts 'Hello world'
diff --git a/t/t4034/tex/expect b/t/t4034/tex/expect
new file mode 100644
index 0000000..604969b
--- /dev/null
+++ b/t/t4034/tex/expect
@@ -0,0 +1,9 @@
+<BOLD>diff --git a/pre b/post<RESET>
+<BOLD>index 2b2dfcb..65cab61 100644<RESET>
+<BOLD>--- a/pre<RESET>
+<BOLD>+++ b/post<RESET>
+<CYAN>@@ -1,4 +1,4 @@<RESET>
+\section{Something <GREEN>new<RESET>}
+<RED>\emph<RESET><GREEN>\textbf<RESET>{Macro style}
+{<RED>\em<RESET><GREEN>\bfseries<RESET> State toggle style}
+\\[<RED>1em<RESET><GREEN>1cm<RESET>]
diff --git a/t/t4034/tex/post b/t/t4034/tex/post
new file mode 100644
index 0000000..65cab61
--- /dev/null
+++ b/t/t4034/tex/post
@@ -0,0 +1,4 @@
+\section{Something new}
+\textbf{Macro style}
+{\bfseries State toggle style}
+\\[1cm]
diff --git a/t/t4034/tex/pre b/t/t4034/tex/pre
new file mode 100644
index 0000000..2b2dfcb
--- /dev/null
+++ b/t/t4034/tex/pre
@@ -0,0 +1,4 @@
+\section{Something}
+\emph{Macro style}
+{\em State toggle style}
+\\[1em]
-- 
1.7.3.3.807.g6ee1f
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help