[PATCH v7 p2 0/2] fast-export fixes

Hi,

Here's version 7 part 2; I've dropped all the unnecessary patches, nobody seems
to care about the current brokedness, and I did them only to show that these
are correct, and that remote helpers without marks just don't work. These are
the ones I care about.

Below is a summary of what happens when you apply both patches, and how the new
behavior is obviously correct. All the refs point to the same object.

== before ==

  % git fast-export --export--marks=marks master
  # exported stuff
  % git fast-export --{import,export}-marks=marks test
  # nothing
  % git fast-export --{import,export}-marks=marks master ^uninteresting
  reset refs/heads/uninteresting
  from :6

  % git fast-export --{import,export}-marks=marks ^uninteresting master ^foo test
  reset refs/heads/test
  from :6

  reset refs/heads/foo
  from :6

  reset refs/heads/master
  from :6

  % git fast-export --{import,export}-marks=marks uninteresting..master

== after ==

  % git fast-export --export--marks=marks master
  # exported stuff
  % git fast-export --{import,export}-marks=marks test
  reset refs/heads/test
  from :6

  % git fast-export --{import,export}-marks=marks master ^uninteresting
  reset refs/heads/master
  from :6

  % git fast-export --{import,export}-marks=marks ^uninteresting master ^foo test
  reset refs/heads/test
  from :6

  reset refs/heads/master
  from :6

  % git fast-export --{import,export}-marks=marks uninteresting..master
  reset refs/heads/master
  from :6

Changes since v6:

  * Drop all the extra patches
  * Reorder patches so tests never fail

Felipe Contreras (2):
  fast-export: don't handle uninteresting refs
  fast-export: make sure updated refs get updated

 builtin/fast-export.c     | 21 ++++++++++++++-------
 t/t5801-remote-helpers.sh | 28 ++++++++++++++++------------
 t/t9350-fast-export.sh    | 45 +++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 75 insertions(+), 19 deletions(-)

-- 
1.8.0.1

They have been marked as UNINTERESTING for a reason, lets respect that.

Currently the first ref is handled properly, but not the rest:

  % git fast-export master ^uninteresting ^foo ^bar
  reset refs/heads/bar
  from :0

  reset refs/heads/foo
  from :0

  reset refs/heads/uninteresting
  from :0

  % git fast-export ^uninteresting ^foo ^bar master
  reset refs/heads/master
  from :0

  reset refs/heads/bar
  from :0

  reset refs/heads/foo
  from :0

Clearly this is wrong; the negative refs should be ignored.

After this patch:

  % git fast-export ^uninteresting ^foo ^bar master
  # nothing
  % git fast-export master ^uninteresting ^foo ^bar
  # nothing

And even more, it would only happen if the ref is pointing to exactly
the same commit, but not otherwise:

 % git fast-export ^next next
 reset refs/heads/next
 from :0

 % git fast-export ^next next^{commit}
 # nothing
 % git fast-export ^next next~0
 # nothing
 % git fast-export ^next next~1
 # nothing
 % git fast-export ^next next~2
 # nothing

The reason this happens is that before traversing the commits,
fast-export checks if any of the refs point to the same object, and any
duplicated ref gets added to a list in order to issue 'reset' commands
after the traversing. Unfortunately, it's not even checking if the
commit is flagged as UNINTERESTING. The fix of course, is to do
precisely that.

However, in order to do it properly we need to get the UNINTERESTING flag
from the command line ref, not from the commit object. Fortunately we
can simply use revs.pending, which contains all the information we need
for get_tags_and_duplicates(), plus the ref flag. This way the rest of
the positive refs will remain untouched; it's only the negative ones
that change in behavior.

Signed-off-by: Felipe Contreras <redacted>
---
 builtin/fast-export.c     | 11 +++++++----
 t/t5801-remote-helpers.sh |  8 ++++++++
 t/t9350-fast-export.sh    | 30 ++++++++++++++++++++++++++++++
 3 files changed, 45 insertions(+), 4 deletions(-)

diff --git a/builtin/fast-export.c b/builtin/fast-export.c
index 191936c..2547e6c 100644
--- a/builtin/fast-export.c
+++ b/builtin/fast-export.c

@@ -474,18 +474,21 @@ static void handle_tag(const char *name, struct tag *tag)
 	       (int)message_size, (int)message_size, message ? message : "");
 }
 
-static void get_tags_and_duplicates(struct object_array *pending,
+static void get_tags_and_duplicates(struct rev_cmdline_info *info,
 				    struct string_list *extra_refs)
 {
 	struct tag *tag;
 	int i;
 
-	for (i = 0; i < pending->nr; i++) {
-		struct object_array_entry *e = pending->objects + i;
+	for (i = 0; i < info->nr; i++) {
+		struct rev_cmdline_entry *e = info->rev + i;
 		unsigned char sha1[20];
 		struct commit *commit;
 		char *full_name;
 
+		if (e->flags & UNINTERESTING)
+			continue;
+
 		if (dwim_ref(e->name, strlen(e->name), sha1, &full_name) != 1)
 			continue;
 

@@ -681,7 +684,7 @@ int cmd_fast_export(int argc, const char **argv, const char *prefix)
 	if (import_filename && revs.prune_data.nr)
 		full_tree = 1;
 
-	get_tags_and_duplicates(&revs.pending, &extra_refs);
+	get_tags_and_duplicates(&revs.cmdline, &extra_refs);
 
 	if (prepare_revision_walk(&revs))
 		die("revision walk setup failed");

diff --git a/t/t5801-remote-helpers.sh b/t/t5801-remote-helpers.sh
index 12ae256..ece8fd5 100755
--- a/t/t5801-remote-helpers.sh
+++ b/t/t5801-remote-helpers.sh

@@ -162,4 +162,12 @@ test_expect_failure 'pushing without marks' '
 	compare_refs local2 HEAD server HEAD
 '
 
+test_expect_success 'push all with existing object' '
+	(cd local &&
+	git branch dup2 master &&
+	git push origin --all
+	) &&
+	compare_refs local dup2 server dup2
+'
+
 test_done

diff --git a/t/t9350-fast-export.sh b/t/t9350-fast-export.sh
index 1f59862..c8e41c1 100755
--- a/t/t9350-fast-export.sh
+++ b/t/t9350-fast-export.sh

@@ -454,4 +454,34 @@ test_expect_success 'test bidirectionality' '
 	git fast-import --export-marks=marks-cur --import-marks=marks-cur
 '
 
+cat > expected << EOF
+blob
+mark :13
+data 5
+bump
+
+commit refs/heads/master
+mark :14
+author A U Thor <author@example.com> 1112912773 -0700
+committer C O Mitter <committer@example.com> 1112912773 -0700
+data 5
+bump
+from :12
+M 100644 :13 file
+
+EOF
+
+test_expect_success 'avoid uninteresting refs' '
+	> tmp-marks &&
+	git fast-export --import-marks=tmp-marks \
+		--export-marks=tmp-marks master > /dev/null &&
+	git tag v1.0 &&
+	git branch uninteresting &&
+	echo bump > file &&
+	git commit -a -m bump &&
+	git fast-export --import-marks=tmp-marks \
+		--export-marks=tmp-marks ^uninteresting ^v1.0 master > actual &&
+	test_cmp expected actual
+'
+
 test_done

-- 
1.8.0.1

When an object has already been exported (and thus is in the marks) it's
flagged as SHOWN, so it will not be exported again, even if in a later
time it's exported through a different ref.

We don't need the object to be exported again, but we want the ref
updated, which doesn't happen.

Since we can't know if a ref was exported or not, let's just assume that
if the commit was marked (flags & SHOWN), the user still wants the ref
updated.

IOW: If it's specified in the command line, it will get updated,
regardless of whether or not the object was marked.

So:

 % git branch test master
 % git fast-export $mark_flags master
 % git fast-export $mark_flags test

Would export 'test' properly.

Additionally, this fixes issues with remote helpers; now they can push
refs whose objects have already been exported, and a few other issues as
well. Update the tests accordingly.

Signed-off-by: Felipe Contreras <redacted>
---
 builtin/fast-export.c     | 10 +++++++---
 t/t5801-remote-helpers.sh | 20 ++++++++------------
 t/t9350-fast-export.sh    | 15 +++++++++++++++
 3 files changed, 30 insertions(+), 15 deletions(-)

diff --git a/builtin/fast-export.c b/builtin/fast-export.c
index 2547e6c..77dffd1 100644
--- a/builtin/fast-export.c
+++ b/builtin/fast-export.c

@@ -526,10 +526,14 @@ static void get_tags_and_duplicates(struct rev_cmdline_info *info,
 				typename(e->item->type));
 			continue;
 		}
-		if (commit->util)
-			/* more than one name for the same object */
+
+		/*
+		 * This ref will not be updated through a commit, lets make
+		 * sure it gets properly updated eventually.
+		 */
+		if (commit->util || commit->object.flags & SHOWN)
 			string_list_append(extra_refs, full_name)->util = commit;
-		else
+		if (!commit->util)
 			commit->util = full_name;
 	}
 }

diff --git a/t/t5801-remote-helpers.sh b/t/t5801-remote-helpers.sh
index ece8fd5..f387027 100755
--- a/t/t5801-remote-helpers.sh
+++ b/t/t5801-remote-helpers.sh

@@ -63,18 +63,6 @@ test_expect_success 'fetch new branch' '
 	compare_refs server HEAD local FETCH_HEAD
 '
 
-#
-# This is only needed because of a bug not detected by this script. It will be
-# fixed shortly, but for now lets not cause regressions.
-#
-test_expect_success 'bump commit in server' '
-	(cd server &&
-	git checkout master &&
-	echo content >>file &&
-	git commit -a -m four) &&
-	compare_refs server HEAD server HEAD
-'
-
 test_expect_success 'fetch multiple branches' '
 	(cd local &&
 	 git fetch

@@ -170,4 +158,12 @@ test_expect_success 'push all with existing object' '
 	compare_refs local dup2 server dup2
 '
 
+test_expect_success 'push ref with existing object' '
+	(cd local &&
+	git branch dup master &&
+	git push origin dup
+	) &&
+	compare_refs local dup server dup
+'
+
 test_done

diff --git a/t/t9350-fast-export.sh b/t/t9350-fast-export.sh
index c8e41c1..9320b4f 100755
--- a/t/t9350-fast-export.sh
+++ b/t/t9350-fast-export.sh

@@ -484,4 +484,19 @@ test_expect_success 'avoid uninteresting refs' '
 	test_cmp expected actual
 '
 
+cat > expected << EOF
+reset refs/heads/master
+from :14
+
+EOF
+
+test_expect_success 'refs are updated even if no commits need to be exported' '
+	> tmp-marks &&
+	git fast-export --import-marks=tmp-marks \
+		--export-marks=tmp-marks master > /dev/null &&
+	git fast-export --import-marks=tmp-marks \
+		--export-marks=tmp-marks master > actual &&
+	test_cmp expected actual
+'
+
 test_done

-- 
1.8.0.1

On 28.11.2012, at 23:23, Felipe Contreras wrote:

They have been marked as UNINTERESTING for a reason, lets respect that.

Currently the first ref is handled properly, but not the rest:

 % git fast-export master ^uninteresting ^foo ^bar

All these refs are assumed to point to the same object, right? I think it would be better if the commit message stated that explicitly. To make up for the lost space, you could then get rid of one of the four refs, I think three are sufficient to drive the message home ;-).


<snip>

The reason this happens is that before traversing the commits,
fast-export checks if any of the refs point to the same object, and any
duplicated ref gets added to a list in order to issue 'reset' commands
after the traversing. Unfortunately, it's not even checking if the
commit is flagged as UNINTERESTING. The fix of course, is to do
precisely that.

Hm... So this might be me being a stupid n00b (I am not yet that familiar with the internal rep of things in git and all...)... but I found the "precisely that" par very confusing, because right afterwards, you say:

However, in order to do it properly we need to get the UNINTERESTING flag
from the command line ref, not from the commit object.

So this sounds like you are saying "we do *precisely* that, except we don't, because it is more complicated, so we actually don't do this *precisely*, just manner of speaking..."

Some details here are beyond my knowledge, I am afraid, so I have to resort to guess: In particular it is not clear to me why the "however" part pops up: Reading it makes it sound as if the commit object also carries an UNINTERESTING flag, but we can't use it because of some reason (perhaps it doesn't have the semantics we need?), so we have to look at revs.pending instead. Right? Wrong? Or is it because the commit objects actually do *not* carry the UNINTERESTING bits, hence we need to look at revs.pending. Or is it due to yet another reason?

I would find it helpful if that could be clarified. E.g. like so:

 "The fix is to add such a check. However, we cannot just use the UNINTERESTING flag of the commit object, because INSERT-REASON."

or

 "The fix is to add such a check. However, the commit object does not contain the UNINTERESTING flag directly."

or something.

Anyway, other than these nitpicky questions, this whole thing looks very logical to me, description and code alike. I also played around with tons of "fast-export" invocations, with and without this patch, and it seems to do what the description says. Finally, I went to the various long threads discussion prior versions of this patch, in particular those starting at
  http://thread.gmane.org/gmane.comp.version-control.git/208725
and
  http://thread.gmane.org/gmane.comp.version-control.git/209355/focus=209370

These contained some concerns. Sadly, several of those discussions ultimately degenerated into not-so-pleasant exchanges :-(, and my impression is that as a result some people are not so inclined to comment on these patches anymore at all. Which is a pity :-(. But overall, it seems this patch makes nothing worse, but fixes some things; and it is simple enough that it shouldn't make future improvements harder.

So *I* at least am quite happy with this, it helps me! My impression is that Felipe's latest patch addresses most concerns people raised by means of an improved description. I couldn't find any in those threads that I feel still applies -- but of course those people should speak for themselves, I am simply afraid they don't want to be part of this anymore :-(.


Still, for what little it might be worth, I think this patch is good and a real improvement. I hope it can be merged soon.


Cheers,
Max

On Thu, Nov 29, 2012 at 2:16 AM, Max Horn [off-list ref] wrote:

On 28.11.2012, at 23:23, Felipe Contreras wrote:

quoted

They have been marked as UNINTERESTING for a reason, lets respect that.

Currently the first ref is handled properly, but not the rest:

 % git fast-export master ^uninteresting ^foo ^bar

All these refs are assumed to point to the same object, right? I think it would be better if the commit message stated that explicitly. To make up for the lost space, you could then get rid of one of the four refs, I think three are sufficient to drive the message home ;-).

Yeah, they point to the same object.

<snip>

quoted

The reason this happens is that before traversing the commits,
fast-export checks if any of the refs point to the same object, and any
duplicated ref gets added to a list in order to issue 'reset' commands
after the traversing. Unfortunately, it's not even checking if the
commit is flagged as UNINTERESTING. The fix of course, is to do
precisely that.

Hm... So this might be me being a stupid n00b (I am not yet that familiar with the internal rep of things in git and all...)... but I found the "precisely that" par very confusing, because right afterwards, you say:

Yeah, the next part was added afterwards.

quoted

However, in order to do it properly we need to get the UNINTERESTING flag
from the command line ref, not from the commit object.

So this sounds like you are saying "we do *precisely* that, except we don't, because it is more complicated, so we actually don't do this *precisely*, just manner of speaking..."

Well, we do check fro the UNINTERESTING flag, but on the ref, not on the commit.

Some details here are beyond my knowledge, I am afraid, so I have to resort to guess: In particular it is not clear to me why the "however" part pops up: Reading it makes it sound as if the commit object also carries an UNINTERESTING flag, but we can't use it because of some reason (perhaps it doesn't have the semantics we need?), so we have to look at revs.pending instead. Right? Wrong? Or is it because the commit objects actually do *not* carry the UNINTERESTING bits, hence we need to look at revs.pending. Or is it due to yet another reason?

It's actually revs.cmdline, I typed the wrong one.

If you have two refs pointing to the same object, and you do 'one
^two', the object (e.g. 8c7a786) will get the UNINTERESTING flag, but
that doesn't tell us anything about the ref being a positive or a
negative one, and revs.pending only has the object flags. On the other
hand revs.cmdline does have the flags for the refs.

Does that explain it?

Anyway, other than these nitpicky questions, this whole thing looks very logical to me, description and code alike. I also played around with tons of "fast-export" invocations, with and without this patch, and it seems to do what the description says. Finally, I went to the various long threads discussion prior versions of this patch, in particular those starting at
  http://thread.gmane.org/gmane.comp.version-control.git/208725
and
  http://thread.gmane.org/gmane.comp.version-control.git/209355/focus=209370

These contained some concerns. Sadly, several of those discussions ultimately degenerated into not-so-pleasant exchanges :-(, and my impression is that as a result some people are not so inclined to comment on these patches anymore at all. Which is a pity :-(. But overall, it seems this patch makes nothing worse, but fixes some things; and it is simple enough that it shouldn't make future improvements harder.

So *I* at least am quite happy with this, it helps me! My impression is that Felipe's latest patch addresses most concerns people raised by means of an improved description. I couldn't find any in those threads that I feel still applies -- but of course those people should speak for themselves, I am simply afraid they don't want to be part of this anymore :-(.

Indeed. For all the concerns given I made a response to how that
either is not true, or doesn't really matter, and in the case of the
latter, I asked for examples where it would matter, only to receive
nothing. For whatever reason involved people are not responding, not a
single valid concern has been raised and remained.

So I think it's good.

Cheers.

-- 
Felipe Contreras