[PATCH 1/3] Add --blob-filter option to filter-branch.

From: Jeff King <redacted>

On Tue, Apr 22, 2008 at 12:51:14PM -0400, Avery Pennarun wrote:

Do you think git would benefit from having a generalized version of
this script?  Basically, the user provides a "munge" script on the
command line, and there's a git-filter-branch mode for auto-munging
(with a cache) every file in every checkin.  Even if it's *only* ever
used for CRLF, I can imagine this being useful to a lot of people.

It was easy enough to work up the patch below, which allows

  git filter-branch --blob-filter 'tr a-z A-Z'

However, it's _still_ horribly slow. Shell script is nice and flexible,
but running a tight loop like this is just painful. I suspect
filter-branch in something like perl would be a lot faster and just as
flexible (you could even do it in C, but you'd probably have to invent a
little domain-specific scripting language).

It is still much better performance than a tree filter, though:

  $ cd git && time git filter-branch --tree-filter '
      find . -type f | while read f; do
        tr a-z A-Z <"$f" >tmp
        mv tmp "$f"
      done
    ' HEAD~10..HEAD

  real    4m38.626s
  user    1m32.726s
  sys     2m51.163s

  $ cd git && git filter-branch --blob-filter 'tr a-z A-Z' HEAD~10..HEAD
  real    1m40.809s
  user    0m36.822s
  sys     1m14.273s

Lots of system time in both. I'm sure we spend a fair bit of time
hitting our very large map and blob-cache directories, which would be
much more nicely implemented as associative arrays in memory (if we were
using a more featureful language).

Anyway, here is the patch. I don't know if it is even worth applying,
since it is still painfully slow.

Acked-by: Johannes Schindelin <redacted>
---
 git-filter-branch.sh |   30 ++++++++++++++++++++++++++++++
 1 files changed, 30 insertions(+), 0 deletions(-)

diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index ea59015..980c431 100755
--- a/git-filter-branch.sh
+++ b/git-filter-branch.sh

@@ -54,6 +54,23 @@ EOF
 
 eval "$functions"
 
+munge_blobs() {
+	while read mode sha1 stage path
+	do
+		if ! test -r "$workdir/../blob-cache/$sha1"
+		then
+			new=`git cat-file blob $sha1 |
+			     eval "$filter_blob" |
+			     git hash-object -w --stdin`
+			printf $new >$workdir/../blob-cache/$sha1
+		fi
+		printf "%s %s\t%s\n" \
+			"$mode" \
+			$(cat "$workdir/../blob-cache/$sha1") \
+			"$path"
+	done
+}
+
 # When piped a commit, output a script to set the ident of either
 # "author" or "committer
 

@@ -105,6 +122,7 @@ tempdir=.git-rewrite
 filter_env=
 filter_tree=
 filter_index=
+filter_blob=
 filter_parent=
 filter_msg=cat
 filter_commit='git commit-tree "$@"'

@@ -150,6 +168,9 @@ do
 	--index-filter)
 		filter_index="$OPTARG"
 		;;
+	--blob-filter)
+		filter_blob="$OPTARG"
+		;;
 	--parent-filter)
 		filter_parent="$OPTARG"
 		;;

@@ -227,6 +248,9 @@ ret=0
 # map old->new commit ids for rewriting parents
 mkdir ../map || die "Could not create map/ directory"
 
+# cache rewritten blobs for blob filter
+mkdir ../blob-cache || die "Could not create blob-cache/ directory"
+
 case "$filter_subdir" in
 "")
 	git rev-list --reverse --topo-order --default HEAD \

@@ -295,6 +319,12 @@ while read commit parents; do
 	eval "$filter_index" < /dev/null ||
 		die "index filter failed: $filter_index"
 
+	if test -n "$filter_blob"; then
+		git ls-files --stage |
+		munge_blobs |
+		git update-index --index-info
+	fi
+
 	parentstr=
 	for parent in $parents; do
 		for reparent in $(map "$parent"); do

-- 
1.5.4.3

Signed-off-by: Avery Pennarun <redacted>
---
 Documentation/git-filter-branch.txt |   17 +++++++++++++++++
 1 files changed, 17 insertions(+), 0 deletions(-)

diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt
index 2a78549..367f119 100644
--- a/Documentation/git-filter-branch.txt
+++ b/Documentation/git-filter-branch.txt

@@ -12,6 +12,7 @@ SYNOPSIS
 	[--index-filter <command>] [--parent-filter <command>]
 	[--msg-filter <command>] [--commit-filter <command>]
 	[--tag-name-filter <command>] [--subdirectory-filter <directory>]
+	[--blob-filter <command]
 	[--original <namespace>] [-d <directory>] [-f | --force]
 	[<rev-list options>...]
 

@@ -143,6 +144,15 @@ definition impossible to preserve signatures at any rate.)
 	The result will contain that directory (and only that) as its
 	project root.
 
+--blob-filter <command>::
+	This is the filter for modifying the contents of each file (blob)
+	in the tree.  The contents of a file are provided on stdin, and
+	the new file contents should be provided on stdout.  For efficiency,
+	the before/after results of a given blob are only calculated once
+	and then cached, so your filter must always return the same output
+	blob for any given input blob.  You might use this filter for
+	converting CRLF to LF in all your files, for example.
+
 --original <namespace>::
 	Use this option to set the namespace where the original commits
 	will be stored. The default value is 'refs/original'.

@@ -185,6 +195,13 @@ git filter-branch --index-filter 'git update-index --remove filename' HEAD
 
 Now, you will get the rewritten history saved in HEAD.
 
+To convert CRLF to LF in all your files using the "fromdos" program (be
+careful: this will attempt to modify binary files too!):
+
+----------------------------------------------
+git filter-branch --blob-filter 'fromdos' HEAD
+----------------------------------------------
+
 To set a commit (which typically is at the tip of another
 history) to be the parent of the current initial commit, in
 order to paste the other history behind the current history:

-- 
1.5.4.3

The main loop of munge_blobs() had to fork-exec "cat" every time through the
loop, even when a blob was already cached.  Let's use the sh builtin 'read'
instead for a huge speedup.

cd git
time git filter-branch --blob-filter 'tr a-z A-Z' HEAD~10..HEAD

(original --blob-filter)
real    3m58.569s
user    0m22.900s
sys     3m32.030s

(with 'cat' calls removed)
real	1m11.931s
user	0m8.520s
sys	1m2.900s

(with 'cat' calls removed and blob cache already filled)
real	0m19.660s
user	0m3.930s
sys	0m15.720s

Signed-off-by: Avery Pennarun <redacted>
---
 git-filter-branch.sh |   18 +++++++++++-------
 1 files changed, 11 insertions(+), 7 deletions(-)

diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index 980c431..37ac99d 100755
--- a/git-filter-branch.sh
+++ b/git-filter-branch.sh

@@ -57,16 +57,18 @@ eval "$functions"
 munge_blobs() {
 	while read mode sha1 stage path
 	do
-		if ! test -r "$workdir/../blob-cache/$sha1"
+		if ! test -r "$cachedir/$sha1"
 		then
-			new=`git cat-file blob $sha1 |
-			     eval "$filter_blob" |
-			     git hash-object -w --stdin`
-			printf $new >$workdir/../blob-cache/$sha1
+			new=$(git cat-file blob $sha1 |
+			      eval "$filter_blob" |
+			      git hash-object -w --stdin)
+			printf $new >$cachedir/$sha1
+		else
+			read new <"$cachedir/$sha1"
 		fi
 		printf "%s %s\t%s\n" \
 			"$mode" \
-			$(cat "$workdir/../blob-cache/$sha1") \
+			"$new" \
 			"$path"
 	done
 }

@@ -108,6 +110,7 @@ USAGE="[--env-filter <command>] [--tree-filter <command>] \
 [--index-filter <command>] [--parent-filter <command>] \
 [--msg-filter <command>] [--commit-filter <command>] \
 [--tag-name-filter <command>] [--subdirectory-filter <directory>] \
+[--blob-filter <command>] \
 [--original <namespace>] [-d <directory>] [-f | --force] \
 [<rev-list options>...]"
 

@@ -249,7 +252,8 @@ ret=0
 mkdir ../map || die "Could not create map/ directory"
 
 # cache rewritten blobs for blob filter
-mkdir ../blob-cache || die "Could not create blob-cache/ directory"
+cachedir="$workdir/../blob-cache"
+mkdir "$cachedir" || die "Could not create blob-cache/ directory"
 
 case "$filter_subdir" in
 "")

-- 
1.5.4.3

Hi,

On Wed, 23 Apr 2008, Avery Pennarun wrote:

From: Jeff King <redacted>

On Tue, Apr 22, 2008 at 12:51:14PM -0400, Avery Pennarun wrote:

quoted

Do you think git would benefit from having a generalized version of
this script?  Basically, the user provides a "munge" script on the
command line, and there's a git-filter-branch mode for auto-munging
(with a cache) every file in every checkin.  Even if it's *only* ever
used for CRLF, I can imagine this being useful to a lot of people.

It was easy enough to work up the patch below, which allows

  git filter-branch --blob-filter 'tr a-z A-Z'

However, it's _still_ horribly slow. Shell script is nice and flexible,
but running a tight loop like this is just painful. I suspect
filter-branch in something like perl would be a lot faster and just as
flexible (you could even do it in C, but you'd probably have to invent a
little domain-specific scripting language).

It is still much better performance than a tree filter, though:

  $ cd git && time git filter-branch --tree-filter '
      find . -type f | while read f; do
        tr a-z A-Z <"$f" >tmp
        mv tmp "$f"
      done
    ' HEAD~10..HEAD

  real    4m38.626s
  user    1m32.726s
  sys     2m51.163s

  $ cd git && git filter-branch --blob-filter 'tr a-z A-Z' HEAD~10..HEAD
  real    1m40.809s
  user    0m36.822s
  sys     1m14.273s

Lots of system time in both. I'm sure we spend a fair bit of time
hitting our very large map and blob-cache directories, which would be
much more nicely implemented as associative arrays in memory (if we were
using a more featureful language).

Anyway, here is the patch. I don't know if it is even worth applying,
since it is still painfully slow.

Not all of this belongs in the commit messaage.

Acked-by: Johannes Schindelin <redacted>

This does.

A good general rule is: if you think it would be funny/strange to read 
this message in the output of "git log", it should be changed.

Ciao,
Dscho

On 4/23/08, Johannes Schindelin [off-list ref] wrote:

Not all of this belongs in the commit messaage.

 > Acked-by: Johannes Schindelin [off-list ref]

This does.

 A good general rule is: if you think it would be funny/strange to read
 this message in the output of "git log", it should be changed.

I felt uncomfortable modifying the message attached to Jeff's patch
(which actually does pretty clearly explain what's going on), since
it's not mine.  If he wants to send me a better commit message,
that'll be fine too.

Have fun,

Avery

Hi,

On Wed, 23 Apr 2008, Avery Pennarun wrote:

On 4/23/08, Johannes Schindelin [off-list ref] wrote:

quoted

Not all of this belongs in the commit messaage.

 > Acked-by: Johannes Schindelin [off-list ref]

This does.

 A good general rule is: if you think it would be funny/strange to 
 read this message in the output of "git log", it should be changed.

I felt uncomfortable modifying the message attached to Jeff's patch
(which actually does pretty clearly explain what's going on), since
it's not mine.  If he wants to send me a better commit message,
that'll be fine too.

Well, I think that you would do a good job rephrasing this as a proper 
commit message :-)

Ciao,
Dscho

From: Jeff King <redacted>

This patch allows

  git filter-branch --blob-filter 'tr a-z A-Z'

However, it's _still_ horribly slow. Shell script is nice and flexible,
but running a tight loop like this is just painful. I suspect
filter-branch in something like perl would be a lot faster and just as
flexible.

It is still much better performance than a tree filter, though:

  $ cd git && time git filter-branch --tree-filter '
      find . -type f | while read f; do
        tr a-z A-Z <"$f" >tmp
        mv tmp "$f"
      done
    ' HEAD~10..HEAD

  real    4m38.626s
  user    1m32.726s
  sys     2m51.163s

  $ cd git && git filter-branch --blob-filter 'tr a-z A-Z' HEAD~10..HEAD
  real    1m40.809s
  user    0m36.822s
  sys     1m14.273s

Acked-by: Johannes Schindelin <redacted>
---
 git-filter-branch.sh |   30 ++++++++++++++++++++++++++++++
 1 files changed, 30 insertions(+), 0 deletions(-)

diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index ea59015..980c431 100755
--- a/git-filter-branch.sh
+++ b/git-filter-branch.sh

@@ -54,6 +54,23 @@ EOF
 
 eval "$functions"
 
+munge_blobs() {
+	while read mode sha1 stage path
+	do
+		if ! test -r "$workdir/../blob-cache/$sha1"
+		then
+			new=`git cat-file blob $sha1 |
+			     eval "$filter_blob" |
+			     git hash-object -w --stdin`
+			printf $new >$workdir/../blob-cache/$sha1
+		fi
+		printf "%s %s\t%s\n" \
+			"$mode" \
+			$(cat "$workdir/../blob-cache/$sha1") \
+			"$path"
+	done
+}
+
 # When piped a commit, output a script to set the ident of either
 # "author" or "committer
 

@@ -105,6 +122,7 @@ tempdir=.git-rewrite
 filter_env=
 filter_tree=
 filter_index=
+filter_blob=
 filter_parent=
 filter_msg=cat
 filter_commit='git commit-tree "$@"'

@@ -150,6 +168,9 @@ do
 	--index-filter)
 		filter_index="$OPTARG"
 		;;
+	--blob-filter)
+		filter_blob="$OPTARG"
+		;;
 	--parent-filter)
 		filter_parent="$OPTARG"
 		;;

@@ -227,6 +248,9 @@ ret=0
 # map old->new commit ids for rewriting parents
 mkdir ../map || die "Could not create map/ directory"
 
+# cache rewritten blobs for blob filter
+mkdir ../blob-cache || die "Could not create blob-cache/ directory"
+
 case "$filter_subdir" in
 "")
 	git rev-list --reverse --topo-order --default HEAD \

@@ -295,6 +319,12 @@ while read commit parents; do
 	eval "$filter_index" < /dev/null ||
 		die "index filter failed: $filter_index"
 
+	if test -n "$filter_blob"; then
+		git ls-files --stage |
+		munge_blobs |
+		git update-index --index-info
+	fi
+
 	parentstr=
 	for parent in $parents; do
 		for reparent in $(map "$parent"); do

-- 
1.5.4.3

Hi,

On Wed, 23 Apr 2008, Avery Pennarun wrote:

From: Jeff King <redacted>

This patch allows

  git filter-branch --blob-filter 'tr a-z A-Z'

However, it's _still_ horribly slow. Shell script is nice and flexible,
but running a tight loop like this is just painful. I suspect
filter-branch in something like perl would be a lot faster and just as
flexible.

It is still much better performance than a tree filter, though:

  $ cd git && time git filter-branch --tree-filter '
      find . -type f | while read f; do
        tr a-z A-Z <"$f" >tmp
        mv tmp "$f"
      done
    ' HEAD~10..HEAD

  real    4m38.626s
  user    1m32.726s
  sys     2m51.163s

  $ cd git && git filter-branch --blob-filter 'tr a-z A-Z' HEAD~10..HEAD
  real    1m40.809s
  user    0m36.822s
  sys     1m14.273s

Acked-by: Johannes Schindelin <redacted>
---

Thanks, I really appreciate it.

Ciao,
Dscho

On Wed, Apr 23, 2008 at 04:18:10PM -0400, Avery Pennarun wrote:

From: Jeff King <redacted>

This patch allows

  git filter-branch --blob-filter 'tr a-z A-Z'

The commit message munging you did is fine.

However, I think Johannes Sixt's question about providing the pathname
needs to be resolved. As it is now, the blob-filter is impossible to use
in a mixed binary/text repository, short of the undocumented $path magic
that you described. And I am a little uncomfortable just adding the
$path as he suggested because of the subtle bug it introduces.

-Peff

On 4/23/08, Jeff King [off-list ref] wrote:

On Wed, Apr 23, 2008 at 04:18:10PM -0400, Avery Pennarun wrote:

 > From: Jeff King [off-list ref]
 >
 > This patch allows
 >
 >   git filter-branch --blob-filter 'tr a-z A-Z'

The commit message munging you did is fine.

 However, I think Johannes Sixt's question about providing the pathname
 needs to be resolved. As it is now, the blob-filter is impossible to use
 in a mixed binary/text repository, short of the undocumented $path magic
 that you described. And I am a little uncomfortable just adding the
 $path as he suggested because of the subtle bug it introduces.

It is indeed a very subtle bug; so subtle, in fact, that I never
expect to experience it myself :)

I think it would be fine to index into the cache using $path$sha1,
which would seem to resolve this issue.  The catch is that $path isn't
a very good cachefile name.  I'd suggest doing an md5sum or something
on it, but that would result in an extra fork for every file, which
brings us back to our original level of slowness (or worse).

Hmm, I gues using a cachefile like $sha1/$path would work; it requires
a "mkdir -p", but only when *filling* the cache.

Avery

On Wed, Apr 23, 2008 at 06:07:03PM -0400, Avery Pennarun wrote:

I think it would be fine to index into the cache using $path$sha1,
which would seem to resolve this issue.  The catch is that $path isn't
a very good cachefile name.  I'd suggest doing an md5sum or something
on it, but that would result in an extra fork for every file, which
brings us back to our original level of slowness (or worse).

Hmm, I gues using a cachefile like $sha1/$path would work; it requires
a "mkdir -p", but only when *filling* the cache.

Keep in mind that $path can have slashes. So you actually need to:

   mkdir -p `dirname $sha1/$path`
   echo $new >$sha1/$path

-Peff