git filter-branch --subdirectory-filter, still a mistery

Hi,

I've been puzzling most of today to do something that must be simple.
I've got a big repo which contains a project with several nicely related
subprojects in directories. Only now, we want to share some of these
subprojects with another project. I.e. they must start to live there own
life. Of course, I would like to keep the history. So, I did (git --version:
1.5.6.GIT):

	% git clone /home/git/pl.git
	% cd pl
	% git filter-branch --subdirectory-filter packages/chr HEAD

This indeed creates a nice directory holding only the contents of
packages/chr.  But, starting qgit I see that all commits, also those
that had absolutely nothing to do with this dir are still there.  Also,
all tags are still there with exactly the same SHA1 as the original.
I'd expect the tags to be rewritten such that their SHA1 refers to the
state of this single directory and its contents!?  Of course, these
tags give me access to everything, so the repository doesn't shrink
much too.

I must be missing something important ...  I found similar complaints,
but few decent answers and the few answer I did find appeared outdated.
The one at http://use.perl.org/~rjbs/journal/34411 comes closest, although
the reset --hard is no longer needed and the copying and gc-ing doesn't
help much anymore.

Should I write a tree-filter that removes all but the directory I want 
to keep?  I.e. something like this?  Feels like and overkill and I fear
I'll have a lot of empty commits left.

	'mv packages/chr .. && rm -r * && mv ../chr/* . && rmdir ../chr'

I'll be grateful for a clue!

	Cheers --- Jan

On Wednesday 06 August 2008 15:39:50 you wrote:

Hi,

I've been puzzling most of today to do something that must be simple.
I've got a big repo which contains a project with several nicely related
subprojects in directories. Only now, we want to share some of these
subprojects with another project. I.e. they must start to live there own
life. Of course, I would like to keep the history. So, I did (git
--version: 1.5.6.GIT):

      % git clone /home/git/pl.git
      % cd pl
      % git filter-branch --subdirectory-filter packages/chr HEAD

This indeed creates a nice directory holding only the contents of
packages/chr.  But, starting qgit I see that all commits, also those
that had absolutely nothing to do with this dir are still there.  Also,
all tags are still there with exactly the same SHA1 as the original.
I'd expect the tags to be rewritten such that their SHA1 refers to the
state of this single directory and its contents!?  Of course, these
tags give me access to everything, so the repository doesn't shrink
much too.

I must be missing something important ...  I found similar complaints,
but few decent answers and the few answer I did find appeared outdated.
The one at http://use.perl.org/~rjbs/journal/34411 comes closest, although
the reset --hard is no longer needed and the copying and gc-ing doesn't
help much anymore.

Should I write a tree-filter that removes all but the directory I want
to keep?  I.e. something like this?  Feels like and overkill and I fear
I'll have a lot of empty commits left.

      'mv packages/chr .. && rm -r * && mv ../chr/* . && rmdir ../chr'

I'll be grateful for a clue!

Weirdness goes on.  I tried this:

    git filter-branch --tree-filter '/home/jan/nobackup/tmp2/keep 
packages/chr'

where `keep' is a shell-script:

----------------------------------------------------------------
tmp=/home/jan/nobackup/tmp2
dir="$1"

if [ -d "$dir" ]; then
  b=`basename $dir`
  mv "$dir" $tmp/$b
  rm -rf *
  mv $tmp/$b/* .
  mv $tmp/$b/.??* .
  rmdir $tmp/$b
else
  rm -rf *
fi
----------------------------------------------------------------

This kind of works. I.e. I end up (after 3 hours) with a tree that only
contains files from packages/chr. Using qgit it no longer shows the
other files in the `tree' view. Only, it has *all* commits of the
original project, most of which of course do not change this directory,
but now at least their diff is empty. I'd assume there is a command to
remove these (which?)

Space wise this isn't ok.  The original project GIT is 140M, after this
action and a git gc, it is 63M: *much* too big.

Whats more weird: all tags still have the same sha1. I copied using git
clone --no-hardlinks pl chr, deleted all refs/tags from packed-refs and
gave a "git gc --prune", to end up with 1.1 GIGABYTE repository!?

I'm starting to feel a bit stupid that I can't get this done ...

        Clues?  --- Jan

Jan Wielemaker wrote:
[...]

	% git filter-branch --subdirectory-filter packages/chr HEAD

This indeed creates a nice directory holding only the contents of
packages/chr.  But, starting qgit I see that all commits, also those
that had absolutely nothing to do with this dir are still there.

The trick is to rewrite all refs, not just HEAD.  I usually proceed as
follows:

  cp -a repo repo.old  # just to keep a backup
  cd repo
  git filter-branch --subdirectory-filter somedir -- --all

The --all tells it to rewrite as many refs as possible.  Note that the
-- is required.  Also note that refs/original/* will still point to
the old commits, so they won't "just vanish".  You may want to clone
the repository or delete them manually once you are sure the
filter-branch did the right thing.

- Thomas

Hi Thomas,

On Thursday 07 August 2008 09:50:03 am Thomas Rast wrote:

Jan Wielemaker wrote:
[...]

quoted

	% git filter-branch --subdirectory-filter packages/chr HEAD

This indeed creates a nice directory holding only the contents of
packages/chr.  But, starting qgit I see that all commits, also those
that had absolutely nothing to do with this dir are still there.

The trick is to rewrite all refs, not just HEAD.  I usually proceed as
follows:

  cp -a repo repo.old  # just to keep a backup
  cd repo
  git filter-branch --subdirectory-filter somedir -- --all

The --all tells it to rewrite as many refs as possible.  Note that the
-- is required.  Also note that refs/original/* will still point to
the old commits, so they won't "just vanish".  You may want to clone
the repository or delete them manually once you are sure the
filter-branch did the right thing.

Thanks. That is moving in the right direction! There are some, possibly
related, problems left (using 1.5.6.GIT).  According to git fsck,
my repo is clean.  I got:

  Ref 'refs/tags/V5.6.50' was rewritten
  error: Ref refs/tags/V5.6.50 is at 8678b32f71178019c06aefa40e2d3fb9a2e8ef25 
but
	expected 2e8aef64e2fed088720a19ac2ffa2481e5bc7806
  fatal: Cannot lock the ref 'refs/tags/V5.6.50'.
  Could not rewrite refs/tags/V5.6.50

Now, if I look in .git/packed-refs, I see this (i.e. a second line with
a ^) for all refs that cause problems:

274ec8ac671542206ba3567ff5d72b3e54c5603c refs/tags/V5.6.59
^28920c3c0a184698d9cd15a65cd643367200bbf5
faf203f9d9e350d84b6b38b7746e710b6232fc97 refs/tags/V5.6.58
^1edb1adedcc47ec15c3242234cc6b7ede94bbfba
48488c871227beabcb3ba167b737d6e33ced65bc refs/tags/V5.6.57
^766587b09e3d2f09c87b03ad0d7faf3529c9dcff

After a bit of puzzling I discovered the the SHA1 after the ^ refers to
the actual commit and I changed all these to `lightweight' tags by
putting the SHA1 behind ^ before the tag itself.  I wrote a little
sh/awk script to automate this (attached).

Now it runs to the end.  Unfortunagtely the history is completely
screwed up :-(:

	* There are a lot of commits that are not related to the dir
	* Commits start long before the directory came into existence,
	Looks like it just shows the whole project at this place.

I think the problem is related to the fact that the directory I want to
filter didn't exists at the start of the project. Looking at
git-rev-list, I found --remove-empty, so I added that after the --all,
but that doesn't appear to help.  I must admit I don't really know what
I'm doing (though I still think the result I want it well defined and
its hard to imagine I'm the only person who wants this).

If someone wants to help: clone git://gollem.science.uva.nl/home/git/pl.git
and try to filter the dir packages/chr.  You can browse the git at
http://gollem.science.uva.nl/git/pl.git

	Thanks --- Jan

Document the '--' option that can be used to pass rev-list options
(not just arguments), and give an example usage of '-- --all'.

Signed-off-by: Thomas Rast <redacted>
---

[This went out to Jan and Junio already, but I forgot to CC the list.
Sorry.]

Somehow I'm imagining this is a FAQ.  Either way, I remember figuring
out this exact example by accident when I first needed it.

 Documentation/git-filter-branch.txt |   13 ++++++++++++-
 1 files changed, 12 insertions(+), 1 deletions(-)

diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt
index a518ba6..1f0fcec 100644
--- a/Documentation/git-filter-branch.txt
+++ b/Documentation/git-filter-branch.txt

@@ -13,7 +13,7 @@ SYNOPSIS
 	[--msg-filter <command>] [--commit-filter <command>]
 	[--tag-name-filter <command>] [--subdirectory-filter <directory>]
 	[--original <namespace>] [-d <directory>] [-f | --force]
-	[<rev-list options>...]
+	[--] [<rev-list options>...]
 
 DESCRIPTION
 -----------

@@ -196,6 +196,17 @@ git filter-branch --index-filter 'git rm --cached filename' HEAD
 
 Now, you will get the rewritten history saved in HEAD.
 
+To rewrite the repository to look as if 'foodir/' had been its project
+root, and discard all other history:
+
+-------------------------------------------------------
+git filter-branch --subdirectory-filter foodir -- --all
+-------------------------------------------------------
+
+Thus you can, e.g., turn a library subdirectory into a repository of
+its own.  Note the '--' that separates 'filter-branch' options from
+revision options, and the '--all' to rewrite all branches and tags.
+
 To set a commit (which typically is at the tip of another
 history) to be the parent of the current initial commit, in
 order to paste the other history behind the current history:

-- 
1.6.0.rc1.106.g98a7

Document the '--' option that can be used to pass rev-list options
(not just arguments), and give an example usage of '-- --all'.  Remove
reference to "the new branch name"; filter-branch takes arbitrary
arguments to rev-list since dfd05e3.

Signed-off-by: Thomas Rast <redacted>
---

At second glance, it turned out the documentation was actually older
than the code.  So rewrite the documentation of <rev-list options>.

 Documentation/git-filter-branch.txt |   21 ++++++++++++++++-----
 1 files changed, 16 insertions(+), 5 deletions(-)

diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt
index a518ba6..31d3cae 100644
--- a/Documentation/git-filter-branch.txt
+++ b/Documentation/git-filter-branch.txt

@@ -13,7 +13,7 @@ SYNOPSIS
 	[--msg-filter <command>] [--commit-filter <command>]
 	[--tag-name-filter <command>] [--subdirectory-filter <directory>]
 	[--original <namespace>] [-d <directory>] [-f | --force]
-	[<rev-list options>...]
+	[--] [<rev-list options>...]
 
 DESCRIPTION
 -----------

@@ -168,10 +168,10 @@ to other tags will be rewritten to point to the underlying commit.
 	'refs/original/', unless forced.
 
 <rev-list options>...::
-	When options are given after the new branch name, they will
-	be passed to 'git-rev-list'.  Only commits in the resulting
-	output will be filtered, although the filtered commits can still
-	reference parents which are outside of that set.
+	Arguments for 'git-rev-list'.  All positive refs included by
+	these options are rewritten.  You may also specify options
+	such as '--all', but you must use '--' to separate them from
+	the 'git-filter-branch' options.
 
 
 Examples

@@ -196,6 +196,17 @@ git filter-branch --index-filter 'git rm --cached filename' HEAD
 
 Now, you will get the rewritten history saved in HEAD.
 
+To rewrite the repository to look as if 'foodir/' had been its project
+root, and discard all other history:
+
+-------------------------------------------------------
+git filter-branch --subdirectory-filter foodir -- --all
+-------------------------------------------------------
+
+Thus you can, e.g., turn a library subdirectory into a repository of
+its own.  Note the '--' that separates 'filter-branch' options from
+revision options, and the '--all' to rewrite all branches and tags.
+
 To set a commit (which typically is at the tip of another
 history) to be the parent of the current initial commit, in
 order to paste the other history behind the current history:

-- 
1.6.0.rc1.106.g98a7

Jan Wielemaker wrote:

  Ref 'refs/tags/V5.6.50' was rewritten
  error: Ref refs/tags/V5.6.50 is at 8678b32f71178019c06aefa40e2d3fb9a2e8ef25 
but
	expected 2e8aef64e2fed088720a19ac2ffa2481e5bc7806
  fatal: Cannot lock the ref 'refs/tags/V5.6.50'.
  Could not rewrite refs/tags/V5.6.50

[...]

Now, if I look in .git/packed-refs [...] and I changed all these to
`lightweight' tags

This appears to be a bug.  I've whipped up a patch that will follow
and should fix the bug.  It has nothing to do with packed-refs; the
current filter-branch chokes on annotated tags during
--subdirectory-filter, even though there is support for tag rewriting.

However, to enable tag rewriting, you need to say --tag-name-filter
cat.

Now it runs to the end.  Unfortunagtely the history is completely
screwed up :-(:

	* There are a lot of commits that are not related to the dir
	* Commits start long before the directory came into existence,
	Looks like it just shows the whole project at this place.

For some reason the ancestor detection does not work right.  I'm also
following up with an RFH patch that significantly improves the success
rate (in terms of branches and tags successfully mapped to a rewritten
commit) in the case of your repository.  I doubt more staring at the
code would yield any more ideas at this hour, so ideas would be
appreciated.

The rest is just the other commits/tags showing a lot of the history.
I don't know of any built-in way to prune the branches and tags that
aren't part of the new master, but

  git branch -a --no-merged master

can tell you which branches aren't ancestors of master.

- Thomas

-- 
Thomas Rast
trast@student.ethz.ch

Previously, git-filter-branch failed if it attempted to update an
annotated tag.  Now we ignore this condition if --tag-name-filter is
given, so that we can later rewrite the tag.  If no such option was
provided, we warn the user that he might want to run with
--tag-name-filter cat to achieve the intended effect.

Signed-off-by: Thomas Rast <redacted>
---
 git-filter-branch.sh |   14 +++++++++++---
 1 files changed, 11 insertions(+), 3 deletions(-)

diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index 182822a..a324cf0 100755
--- a/git-filter-branch.sh
+++ b/git-filter-branch.sh

@@ -361,9 +361,17 @@ do
 	;;
 	$_x40)
 		echo "Ref '$ref' was rewritten"
-		git update-ref -m "filter-branch: rewrite" \
-				"$ref" $rewritten $sha1 ||
-			die "Could not rewrite $ref"
+		if ! git update-ref -m "filter-branch: rewrite" \
+					"$ref" $rewritten $sha1 2>/dev/null; then
+			if test $(git cat-file -t "$ref") = tag; then
+				if test -z "$filter_tag_name"; then
+					warn "WARNING: You said to rewrite tagged commits, but not the corresponding tag."
+					warn "WARNING: Perhaps use '--tag-name-filter cat' to rewrite the tag."
+				fi
+			else
+				die "Could not rewrite $ref"
+			fi
+		fi
 	;;
 	*)
 		# NEEDSWORK: possibly add -Werror, making this an error

-- 
1.6.0.rc2.19.g3c9ba

THIS WILL VERY LIKELY NOT WORK IN ALL CASES.

Use git rev-list -1 -- <subdir> to discover a random ancestor, instead
of more correct boundary detection.  Oddly enough, this _increases_
success rate with Jan's repository and --all.  May break randomly with
more complicated args.
---

Maybe someone understands what's going on and can fix the underlying
bug...

 git-filter-branch.sh |   12 +++---------
 1 files changed, 3 insertions(+), 9 deletions(-)

diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index 182822a..52b2bdf 100755
--- a/git-filter-branch.sh
+++ b/git-filter-branch.sh

@@ -325,15 +325,9 @@ while read ref
 do
 	sha1=$(git rev-parse "$ref"^0)
 	test -f "$workdir"/../map/$sha1 && continue
-	# Assign the boundarie(s) in the set of rewritten commits
-	# as the replacement commit(s).
-	# (This would look a bit nicer if --not --stdin worked.)
-	for p in $( (cd "$workdir"/../map; ls | sed "s/^/^/") |
-		git rev-list $ref --boundary --stdin |
-		sed -n "s/^-//p")
-	do
-		map $p >> "$workdir"/../map/$sha1
-	done
+	# Assign the first commit not pruned as the replacement.
+	candidate=$(git rev-list $ref -1 -- "$filter_subdir")
+	test "$candidate" && map "$candidate" > "$workdir"/../map/$sha1
 done < "$tempdir"/heads
 
 # Finally update the refs

-- 
1.6.0.rc2.19.g3c9ba

Hi Thomas,

Thanks for looking into this!

On Friday 08 August 2008 01:48:05 Thomas Rast wrote:

Jan Wielemaker wrote:

quoted

  Ref 'refs/tags/V5.6.50' was rewritten
  error: Ref refs/tags/V5.6.50 is at
8678b32f71178019c06aefa40e2d3fb9a2e8ef25 but
	expected 2e8aef64e2fed088720a19ac2ffa2481e5bc7806
  fatal: Cannot lock the ref 'refs/tags/V5.6.50'.
  Could not rewrite refs/tags/V5.6.50

[...]

quoted

Now, if I look in .git/packed-refs [...] and I changed all these to
`lightweight' tags

This appears to be a bug.  I've whipped up a patch that will follow
and should fix the bug.  It has nothing to do with packed-refs; the
current filter-branch chokes on annotated tags during
--subdirectory-filter, even though there is support for tag rewriting.

However, to enable tag rewriting, you need to say --tag-name-filter
cat.

Great.  I knew a more fundamental approach was asked for, but I bet my
simple-minded work-around gives the same result, no?

quoted

Now it runs to the end.  Unfortunagtely the history is completely
screwed up :-(:

	* There are a lot of commits that are not related to the dir
	* Commits start long before the directory came into existence,
	Looks like it just shows the whole project at this place.

For some reason the ancestor detection does not work right.  I'm also
following up with an RFH patch that significantly improves the success
rate (in terms of branches and tags successfully mapped to a rewritten
commit) in the case of your repository.  I doubt more staring at the
code would yield any more ideas at this hour, so ideas would be
appreciated.

Thanks. As I'm using the GIT version anyway, I'll apply these patches
and see what happens. The trouble is related to tags and possibly to
branches. I get completely correct result if I delete all branches and
tags before filtering.  That at least helps for this particular subproject
(though some of the tags are useful).

I didn't further investigate branches (I think the packages/chr
directory is not involved in any branch; if you are interested, the boot
directory should show traces of the V57X branch).

I did see that (all/some?) tags that involve changes to the packages/chr
directory nicely end up in its history, but others do not appear on the
filtered master branch and give access to the complete project. See for
example V5.6.59 (the latest release tag). Try (in the filtered branch)

	git diff V5.6.59..

That should only show some small changes, but it diffs the entire project
against the subdir ...

The rest is just the other commits/tags showing a lot of the history.
I don't know of any built-in way to prune the branches and tags that
aren't part of the new master, but

  git branch -a --no-merged master

can tell you which branches aren't ancestors of master.

Thanks for the tip.

	Cheers --- Jan

Hi Thomas,

On Friday 08 August 2008 01:48:05 Thomas Rast wrote:

This appears to be a bug.  I've whipped up a patch that will follow
and should fix the bug.  It has nothing to do with packed-refs; the
current filter-branch chokes on annotated tags during
--subdirectory-filter, even though there is support for tag rewriting.

However, to enable tag rewriting, you need to say --tag-name-filter
cat.

That works!

quoted

Now it runs to the end.  Unfortunagtely the history is completely
screwed up :-(:

	* There are a lot of commits that are not related to the dir
	* Commits start long before the directory came into existence,
	Looks like it just shows the whole project at this place.

For some reason the ancestor detection does not work right.  I'm also
following up with an RFH patch that significantly improves the success
rate (in terms of branches and tags successfully mapped to a rewritten
commit) in the case of your repository.  I doubt more staring at the
code would yield any more ideas at this hour, so ideas would be
appreciated.

The rest is just the other commits/tags showing a lot of the history.
I don't know of any built-in way to prune the branches and tags that
aren't part of the new master, but

  git branch -a --no-merged master

can tell you which branches aren't ancestors of master.

I retried with your two patches. That looks a *lot* better. After using
the above and deleting the reported branches there are still some
branches left, but at least switching to them doesn't bring the complete
project back.

Now there are a few weird tags left, some of these may well be the
result of weird things in the repository. The repository was on CVS
until about a year ago and was converted (using SVN as intermediate).

The big problem is anything that relates to the days before the filtered
directory was part of the project. There are lots of tags there and
switching to them brings back the old project. I'd guess the correct
behaviour is that either all these tags refer to an empty tree or (which
I would prefer) all such tags are deleted.

Is this a bug?  Is there a trick here?  git clone --depth doesn't
seem appropriate.

	Cheers --- Jan

Hi,

On Fri, 8 Aug 2008, Thomas Rast wrote:

Is it not just a question of adding '-- "$filter_subdir"' to the rev-list 
call you removed?

Ciao,
Dscho

Johannes Schindelin wrote:

On Fri, 8 Aug 2008, Thomas Rast wrote:

quoted

diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index 182822a..52b2bdf 100755
--- a/git-filter-branch.sh
+++ b/git-filter-branch.sh

@@ -325,15 +325,9 @@ while read ref
 do
 	sha1=$(git rev-parse "$ref"^0)
 	test -f "$workdir"/../map/$sha1 && continue
-	# Assign the boundarie(s) in the set of rewritten commits
-	# as the replacement commit(s).
-	# (This would look a bit nicer if --not --stdin worked.)
-	for p in $( (cd "$workdir"/../map; ls | sed "s/^/^/") |
-		git rev-list $ref --boundary --stdin |
-		sed -n "s/^-//p")
-	do
-		map $p >> "$workdir"/../map/$sha1
-	done
+	# Assign the first commit not pruned as the replacement.
+	candidate=$(git rev-list $ref -1 -- "$filter_subdir")

I think I see the actual problem.  I made a small testing repository
with history that looks like this:

*   a6f2213... (refs/heads/master) Merge branch 'side'
|\
| * 311f888... (refs/heads/side) outside
| * 472893d... inside dir
* | 9bd52bc... (refs/heads/stale) outside
* | d1b451a... inside dir
|/
* 1c48eea... initial

It is available at

  git://persephone.dnsalias.net/git/filtertest.git

if you want to try.  All commits labelled 'inside dir' do something in
dir/; the others don't.  (You can disregard the 'other' branch for
now; I wanted to test the behaviour on completely disconnected history
too, since that's the case with Jan's repo.)

Let's depict this as the following for now, where capitals stand for
"interesting" commits under the subdirectory filter:

   i -- A -- b(stale) -- M(master)
    \                   /
     \- C -- d(side) --/

When saying

  $ git filter-branch --subdirectory-filter dir -- --all'

I would expect the history to look like:

   A(stale) -- M(master)
              /
   C(side) --/

I think treating it this way makes a lot of sense; you get the last
state that your subdirectory had on the corresponding branch or tag.
(Similarly, a leaf branch that does not affect 'dir' should be backed
up until it hits an ancestor that survives the filter.)

Now the problem with the above ancestor detection is the following.
Consider that at this point, the 'map' directory contains the
(unfiltered) SHA1 for every commit that was rewritten during the
filtering process, i.e.

  $ g rev-list --all -- dir | git name-rev --stdin
  093c591b3d751ce778b4a6e5c2a0906b097b5868 (other~1)
  a6f22134f8ab8bcc762949df53f674e3410f7fc3 (master)
  d1b451a4b0657ea894fd772fc609f7863b7dfd15 (stale~1)
  472893d579383f56f006ff42c563dcbb730bc5b8 (side~1)

So 'map' has the values for M, A, and C.  Now if you expand the call

  (cd "$workdir"/../map; ls | sed "s/^/^/") |
          git rev-list $ref --boundary --stdin

you'll find that during ref=refs/heads/side, it is equivalent to

  $ git rev-list side --boundary ^master ^side~1 ^stale~1 ^other~1
  [no output!]

Oops, it seems that wasn't what we wanted.  The '^master', which
reaches 'side' already, precludes all output.

So now that I've finally understood what is going on, I think a more
careful use of rev-list -1 is actually a correct and easy way to
figure out an ancestor.  Patch follows.

- Thomas

-- 
Thomas Rast
trast@student.ethz.ch

The previous code failed on any refs that are (pre-rewrite) ancestors
of commits marked for rewriting.  This means that in a situation

   A -- B(topic) -- C(master)

where B is dropped by --subdirectory-filter pruning, the 'topic' is
not moved up to A as intended, but left unrewritten.

Fix this by using a more stupid approach: we let 'rev-list -1' figure
out a nearby ancestor, which handles the pruning automatically.

Signed-off-by: Thomas Rast <redacted>
---

I guarded it with a $filter_subdir check to not cause any unintended
harm.  It might be useful in some border cases of rev-list arguments
given to filter-branch too, but I can't figure out a safe way to
handle that.

Either way, this fixes the problem.

- Thomas

 git-filter-branch.sh |   27 +++++++++++----------------
 1 files changed, 11 insertions(+), 16 deletions(-)

diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index a324cf0..7924aa1 100755
--- a/git-filter-branch.sh
+++ b/git-filter-branch.sh

@@ -317,24 +317,19 @@ done <../revs
 
 # In case of a subdirectory filter, it is possible that a specified head
 # is not in the set of rewritten commits, because it was pruned by the
-# revision walker.  Fix it by mapping these heads to the next rewritten
-# ancestor(s), i.e. the boundaries in the set of rewritten commits.
+# revision walker.  Fix it by mapping these heads to a (random!) nearby
+# ancestor that survived the pruning.
 
-# NEEDSWORK: we should sort the unmapped refs topologically first
-while read ref
-do
-	sha1=$(git rev-parse "$ref"^0)
-	test -f "$workdir"/../map/$sha1 && continue
-	# Assign the boundarie(s) in the set of rewritten commits
-	# as the replacement commit(s).
-	# (This would look a bit nicer if --not --stdin worked.)
-	for p in $( (cd "$workdir"/../map; ls | sed "s/^/^/") |
-		git rev-list $ref --boundary --stdin |
-		sed -n "s/^-//p")
+if test "$filter_subdir"
+then
+	while read ref
 	do
-		map $p >> "$workdir"/../map/$sha1
-	done
-done < "$tempdir"/heads
+		sha1=$(git rev-parse "$ref"^0)
+		test -f "$workdir"/../map/$sha1 && continue
+		ancestor=$(git rev-list -1 $ref -- "$filter_subdir")
+		test "$ancestor" && echo $(map $ancestor) >> "$workdir"/../map/$sha1
+	done < "$tempdir"/heads
+fi
 
 # Finally update the refs
 

-- 
1.6.0.rc2.22.g7d28.dirty

Hi,

On Fri, 8 Aug 2008, Thomas Rast wrote:

I think a more careful use of rev-list -1 is actually a correct and easy 
way to figure out an ancestor.

I have not looked at your patch closely, or at your explanation, but I am 
really certain that every attempt to replace the --boundary with a -1 must 
fail.

Let me show you why I think that.  Just look at this history:

A - B - C
  /
D

Where all commits except B touch the inside directory.  Two options:

- you make C a merge (that's what I tried with --boundary), or

- you record B, and C as a commit that does not introduce changes, which 
  is obviously wrong, or

- you record B as a merge, with identical content as A and D, which is 
  pretty tricky (which is why I avoided it).

Anyway, I am really swamped in work, and will not have time to review big 
changes or explanations.  Besides, filter-branch is no fun.  
rewrite-commits would have been, but Sven chickened out.

Ciao,
Dscho

Johannes Schindelin wrote:

On Fri, 8 Aug 2008, Thomas Rast wrote:

quoted

I think a more careful use of rev-list -1 is actually a correct and easy 
way to figure out an ancestor.

I have not looked at your patch closely, or at your explanation, but I am 
really certain that every attempt to replace the --boundary with a -1 must 
fail.

Let me show you why I think that.  Just look at this history:

A - B - C
  /
D

Where all commits except B touch the inside directory.  Two options:

'rev-list' "solves" this problem for us.  At the point where we are
rewriting the branch pointers, commits have already been rewritten to
whatever 'git rev-list --parents -- $subdir' told us to make them.  I
think there are only two cases for its output:

(a) Both A and D bring the same subdirectory contents.  'rev-list
    --parents -- $subdir' drops one side of the merge during pruning.
    It does not look past the merge to see whether the contents were
    arrived at via different changesets.  Thus the history becomes

      A' -- C'

      D'

    and even that only if D was reachable by a different ref,
    otherwise D' is simply dropped.

(b) A and D bring different $subdir contents.  Then the merge is
    interesting and remains.  History is now

      A' -- B' -- C'
           /
      D' -/

Neither of those cases is a problem for the -1 strategy.  A branch
'topic' pointing to B will be rewritten to (a) A' and (b) B'.

IOW, either the merge remains and there is no problem, or the side
branches vanish too and there is no problem.  rev-list never "forward
simplifies" merges; it merely tries to prune away commits on the
incoming side of the merge until all its parents are interesting.

Either that, or I missed something obvious.  I think I'll have to come
up with a better commit message...

- Thomas

-- 
Thomas Rast
trast@student.ethz.ch

The previous ancestor discovery code failed on any refs that are
(pre-rewrite) ancestors of commits marked for rewriting.  This means
that in a situation

   A -- B(topic) -- C(master)

where B is dropped by --subdirectory-filter pruning, the 'topic' was
not moved up to A as intended, but left unrewritten because we asked
about 'git rev-list ^master topic', which does not return anything.

Instead, we use the straightforward

   git rev-list -1 $ref -- $filter_subdir

to find the right ancestor.  To justify this, note that the nearest
ancestor is unique: We use the output of

  git rev-list --parents -- $filter_subdir

to rewrite commits in the first pass, before any ref rewriting.  If B
is a non-merge commit, the only candidate is its parent.  If it is a
merge, there are two cases:

- All sides of the merge bring the same subdirectory contents.  Then
  rev-list already pruned away the merge in favour for just one of its
  parents, so there is only one candidate.

- Some merge sides, or the merge outcome, differ.  Then the merge is
  not pruned and can be rewritten directly.

So it is always safe to use rev-list -1.

Signed-off-by: Thomas Rast <redacted>
---

Only comments and commit message changed since v1, to update the
justification.

 git-filter-branch.sh |   27 +++++++++++----------------
 1 files changed, 11 insertions(+), 16 deletions(-)

diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index a324cf0..a140337 100755
--- a/git-filter-branch.sh
+++ b/git-filter-branch.sh

@@ -317,24 +317,19 @@ done <../revs
 
 # In case of a subdirectory filter, it is possible that a specified head
 # is not in the set of rewritten commits, because it was pruned by the
-# revision walker.  Fix it by mapping these heads to the next rewritten
-# ancestor(s), i.e. the boundaries in the set of rewritten commits.
+# revision walker.  Fix it by mapping these heads to the unique nearest
+# ancestor that survived the pruning.
 
-# NEEDSWORK: we should sort the unmapped refs topologically first
-while read ref
-do
-	sha1=$(git rev-parse "$ref"^0)
-	test -f "$workdir"/../map/$sha1 && continue
-	# Assign the boundarie(s) in the set of rewritten commits
-	# as the replacement commit(s).
-	# (This would look a bit nicer if --not --stdin worked.)
-	for p in $( (cd "$workdir"/../map; ls | sed "s/^/^/") |
-		git rev-list $ref --boundary --stdin |
-		sed -n "s/^-//p")
+if test "$filter_subdir"
+then
+	while read ref
 	do
-		map $p >> "$workdir"/../map/$sha1
-	done
-done < "$tempdir"/heads
+		sha1=$(git rev-parse "$ref"^0)
+		test -f "$workdir"/../map/$sha1 && continue
+		ancestor=$(git rev-list -1 $ref -- "$filter_subdir")
+		test "$ancestor" && echo $(map $ancestor) >> "$workdir"/../map/$sha1
+	done < "$tempdir"/heads
+fi
 
 # Finally update the refs
 

-- 
1.6.0.rc2.23.ge69de8

With --delete-unchanged, we nuke refs whose targets did not change
during rewriting.  It is intended to be used along with
--subdirectory-filter to clean out old refs from before the first
commit to the filtered subdirectory.  (They would otherwise keep the
old history alive.)

Obviously this is a rather dangerous mode of operation.

Note the "sort -u" is required: Without it, --all includes
'origin/master' twice (from 'origin/master' and via 'origin/HEAD'),
and the second pass concludes it is unchanged and nukes the ref.

Signed-off-by: Thomas Rast <redacted>
---

This applies on top of "filter-branch: be more helpful when an
annotated tag changes".

I'm not really sure if this should go in, but it might have solved
Jan's problem.

 git-filter-branch.sh |   33 +++++++++++++++++++++++----------
 1 files changed, 23 insertions(+), 10 deletions(-)

diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index a140337..539b2e6 100755
--- a/git-filter-branch.sh
+++ b/git-filter-branch.sh

@@ -114,6 +114,7 @@ filter_tag_name=
 filter_subdir=
 orig_namespace=refs/original/
 force=
+delete_unchanged=
 while :
 do
 	case "$1" in

@@ -126,6 +127,11 @@ do
 		force=t
 		continue
 		;;
+	--delete-unchanged-refs)
+		shift
+		delete_unchanged=t
+		continue
+		;;
 	-*)
 		;;
 	*)

@@ -215,6 +221,7 @@ export GIT_DIR GIT_WORK_TREE
 
 # The refs should be updated if their heads were rewritten
 git rev-parse --no-flags --revs-only --symbolic-full-name --default HEAD "$@" |
+sort -u |
 sed -e '/^^/d' >"$tempdir"/heads
 
 test -s "$tempdir"/heads ||

@@ -344,7 +351,7 @@ do
 	sha1=$(git rev-parse "$ref"^0)
 	rewritten=$(map $sha1)
 
-	test $sha1 = "$rewritten" &&
+	test $sha1 = "$rewritten" -a -z "$delete_unchanged" &&
 		warn "WARNING: Ref '$ref' is unchanged" &&
 		continue
 

@@ -355,16 +362,22 @@ do
 			die "Could not delete $ref"
 	;;
 	$_x40)
-		echo "Ref '$ref' was rewritten"
-		if ! git update-ref -m "filter-branch: rewrite" \
-					"$ref" $rewritten $sha1 2>/dev/null; then
-			if test $(git cat-file -t "$ref") = tag; then
-				if test -z "$filter_tag_name"; then
-					warn "WARNING: You said to rewrite tagged commits, but not the corresponding tag."
-					warn "WARNING: Perhaps use '--tag-name-filter cat' to rewrite the tag."
+		if test "$delete_unchanged" -a $sha1 = "$rewritten"; then
+			echo "Ref '$ref' was deleted because it is unchanged"
+			git update-ref -m "filter-branch: delete" -d "$ref" $sha1 ||
+				die "Could not delete $ref"
+		else
+			echo "Ref '$ref' was rewritten"
+			if ! git update-ref -m "filter-branch: rewrite" \
+			   "$ref" $rewritten $sha1 2>/dev/null; then
+				if test $(git cat-file -t "$ref") = tag; then
+					if test -z "$filter_tag_name"; then
+						warn "WARNING: You said to rewrite tagged commits, but not the corresponding tag."
+						warn "WARNING: Perhaps use '--tag-name-filter cat' to rewrite the tag."
+					fi
+				else
+					die "Could not rewrite $ref"
 				fi
-			else
-				die "Could not rewrite $ref"
 			fi
 		fi
 	;;

-- 
1.6.0.rc2.24.gf1dd.dirty

Hi,

On Fri, 8 Aug 2008, Thomas Rast wrote:

Johannes Schindelin wrote:

quoted

On Fri, 8 Aug 2008, Thomas Rast wrote:

quoted

I think a more careful use of rev-list -1 is actually a correct and 
easy way to figure out an ancestor.

I have not looked at your patch closely, or at your explanation, but I 
am really certain that every attempt to replace the --boundary with a 
-1 must fail.

Let me show you why I think that.  Just look at this history:

A - B - C
  /
D

quoted

quoted

Where all commits except B touch the inside directory.  Two 
options:

'rev-list' "solves" this problem for us.  At the point where we are 
rewriting the branch pointers, commits have already been rewritten to 
whatever 'git rev-list --parents -- $subdir' told us to make them.  I 
think there are only two cases for its output:

(a) Both A and D bring the same subdirectory contents.  'rev-list
    --parents -- $subdir' drops one side of the merge during pruning. It 
    does not look past the merge to see whether the contents were 
    arrived at via different changesets.  Thus the history becomes

      A' -- C'

      D'

    and even that only if D was reachable by a different ref,
    otherwise D' is simply dropped.

And this is what I call wrong.  Simply dropping one side of the equation 
is not what I call "sane".

If you drop information, you are disagreeing with "content is king".

But hey, if other people agree with you, and this kind of thinking ends 
up in Git proper, I can still resort to other DVCSes.

Ciao,
Dscho

Hi,

On Fri, 8 Aug 2008, Thomas Rast wrote:

With --delete-unchanged, we nuke refs whose targets did not change
during rewriting.

Frankly, I do not see any value in this.  Not even with your explanation.

Ciao,
Dscho

Johannes Schindelin wrote:

But hey, if other people agree with you, and this kind of thinking ends 
up in Git proper, I can still resort to other DVCSes.

BTW, the following is fairly ironic.  (It was later rewritten in
813b473 to the current one-shot 'rev-list --parents' form.)

commit 685ef546b62d063c72b401cd38b83a879301aac4
Author: Johannes Schindelin [off-list ref]
Date:   Fri Jun 8 01:30:35 2007 +0100

    Teach filter-branch about subdirectory filtering
    
    With git-filter-branch --subdirectory-filter <subdirectory> you can
    get at the history, as seen by a certain subdirectory. The history
    of the rewritten branch will only contain commits that touched that
    subdirectory, and the subdirectory will be rewritten to be the new
    project root.
    
    Signed-off-by: Johannes Schindelin [off-list ref]
    Signed-off-by: Junio C Hamano [off-list ref]

diff --git a/git-filter-branch.sh b/git-filter-branch.sh

[...snip...]

@@ -224,7 +228,13 @@ set_ident () {
 
 # list all parent's object names for a given commit
 get_parents () {
-	git-rev-list -1 --parents "$1" | sed "s/^[0-9a-f]*//"
+	case "$filter_subdir" in
+	"")
+		git-rev-list -1 --parents "$1"
+		;;
+	*)
+		git-rev-list -1 --parents "$1" -- "$filter_subdir"
+	esac | sed "s/^[0-9a-f]*//"
 }
 
 tempdir=.git-rewrite

[...snip...]

-- 
Thomas Rast
trast@student.ethz.ch

Hi Thomas,

On Friday 08 August 2008 10:10:24 pm Thomas Rast wrote:

With --delete-unchanged, we nuke refs whose targets did not change
during rewriting.  It is intended to be used along with
--subdirectory-filter to clean out old refs from before the first
commit to the filtered subdirectory.  (They would otherwise keep the
old history alive.)

Obviously this is a rather dangerous mode of operation.

Note the "sort -u" is required: Without it, --all includes
'origin/master' twice (from 'origin/master' and via 'origin/HEAD'),
and the second pass concludes it is unchanged and nukes the ref.

Signed-off-by: Thomas Rast <redacted>
---

This applies on top of "filter-branch: be more helpful when an
annotated tag changes".

I'm not really sure if this should go in, but it might have solved
Jan's problem.

I may hope it isn't just `my problem' :-)  I tested with this patch,
and I can confirm the following produces precisily what I want:

git clone /home/git/pl.git/
cd pl
git remote rm origin
git filter-branch --subdirectory-filter packages/chr --tag-name-filter 
cat --delete-unchanged-refs -- --all
rm -r .git/refs/original
cd ..
git clone file://pl chr

chr is now a nice clean 2 MB repository, starting in 2004, the epoch of
this directory rather than 1992 (the overall project epoch). 

B.t.w. Pretending a remote clone was the only way to get a nice 2MB
repo.  The initial is 140MB.  After the filtering it is 62 MB.  Funny:
after a git gc it grows to 1.1 Gb!?

Anyway, thanks a lot and I hope this makes it into the next git release!

	Cheers --- Jan

Thomas Rast [off-list ref] writes:

Johannes Schindelin wrote:

quoted

But hey, if other people agree with you, and this kind of thinking ends 
up in Git proper, I can still resort to other DVCSes.

BTW, the following is fairly ironic.  (It was later rewritten in
813b473 to the current one-shot 'rev-list --parents' form.)

Hmm, Dscho, perhaps we should take Thomas's patch as a "revert to 685ef54
to fix breakage introduced by 813b473", and demonstrate the breakage with
one of the new tests in his series?

I think it is Ok to use the "view --parents for all branches, instead of
looping with -1" approach when there is no path limiter, and that might be
faster, but if it complicates the logic too much, it probably is not worth
it.  I also _suspect_ that if you use --simplify-merges, the optimization
made by 813b473 would still be usable even with path limiter.

By the way, I am not sure if using --simplify-merges unconditionally is
necessarily a good thing to do.

The user who filters the branches may be interested in a full history
(where using --simplify-merges is the right thing to do), or may be
interested in getting one simplest possible explanation of the end result,
similar to what you get from rev-list without the option.

Junio C Hamano wrote:

Hmm, Dscho, perhaps we should take Thomas's patch as a "revert to 685ef54
to fix breakage introduced by 813b473", and demonstrate the breakage with
one of the new tests in his series?

Now you've lost me.

If you're saying 813b473 is at fault: it is not.  The code I'm trying
to fix came about in dfd05e38.

To see that the change in 813b473 is ok, you can simply run the
following in git.git:

  diff -u <(git rev-list --reverse --parents --topo-order HEAD -- gitk) \
    <(git rev-list --reverse --topo-order HEAD -- gitk | while read commit
      do echo $(git rev-list -1 --parents $commit -- gitk); done)

The one thing that breaks down is (04c6e9e:git-filter-branch.sh:331)

        for p in $( (cd "$workdir"/../map; ls | sed "s/^/^/") |
                git rev-list $ref --boundary --stdin |
                sed -n "s/^-//p")

I also _suspect_ that if you use --simplify-merges, the optimization
made by 813b473 would still be usable even with path limiter.

It is always usable, if we are careful enough to use the same limiting
arguments in all rev-lists involved.

By the way, I am not sure if using --simplify-merges unconditionally is
necessarily a good thing to do.

I think filter-branch would need a generic mechanism to pass arguments
that affect commit selection.  Passing '-- -- file' or '-- ^commit' to
filter-branch --subdirectory-filter will probably break a few things,
so it either needs to recognize those arguments itself or have a
mechanism to specify them, if we want to support it.  This also goes
for the simplification mode.

- Thomas

-- 
Thomas Rast
trast@student.ethz.ch

On Fri, Aug 8, 2008 at 11:10 PM, Thomas Rast [off-list ref] wrote:

With --delete-unchanged, we nuke refs whose targets did not change
during rewriting.  It is intended to be used along with
--subdirectory-filter to clean out old refs from before the first
commit to the filtered subdirectory.  (They would otherwise keep the
old history alive.)

Obviously this is a rather dangerous mode of operation.

Note the "sort -u" is required: Without it, --all includes
'origin/master' twice (from 'origin/master' and via 'origin/HEAD'),
and the second pass concludes it is unchanged and nukes the ref.

This is really useful, why isn't it merged?

Personally I use filter-branch to, duh, filter a branch, so I don't
want the commit objects that are not filtered, nor the refs to them.

-- 
Felipe Contreras