List all commits of a specified file in oldest to newest order

Hi,

I want to list all commits (including renames) of a specified file in 
oldest to newest order. But when I run "git log --follow --reverse -- 
<path/to/the/file>" command, I'm getting a single commit, which points 
to the "rename of the file". This behavior is weird to me because I 
expected to get list of all commit in oldest to newest order, whereas 
"git log --follow -- <path/to/the/file>" command works as expected.

### Steps to reproduce

$ git init test
$ cd test
$ echo "Hello from foo." > a.txt
$ git add a.txt
$ git commit -m 'Initial commit'
$ git mv a.txt b.txt
$ git commit -m 'Rename a.txt to b.txt'
$ git log --follow -- b.txt
commit 55e3e6857755fe815449e787a90fe82feb174817
Author: Redacted <Redacted>
Date:   Fri Nov 5 06:56:58 2021 +0530

     Rename a.txt to b.txt

commit f40baf5e777b2618e02805d16c950687d98356fe
Author: Redacted <Redacted>
Date:   Fri Nov 5 06:53:35 2021 +0530

     Initial commit
$ git log --follow --reverse -- b.txt
commit 55e3e6857755fe815449e787a90fe82feb174817
Author: Redacted <Redacted>
Date:   Fri Nov 5 06:56:58 2021 +0530

     Rename a.txt to b.txt


### Expected output

As far as I understand, using "--reverse" option along with "git log" 
command reverse the chronological order of the commits. So expected 
output of "git log --follow --reverse" command should be in oldest to 
newest commits order that's reverse the chronological order of "git log 
--follow" command's output.

$ git log --follow --reverse -- b.txt
commit f40baf5e777b2618e02805d16c950687d98356fe
Author: Redacted <Redacted>
Date:   Fri Nov 5 06:53:35 2021 +0530

     Initial commit

commit 55e3e6857755fe815449e787a90fe82feb174817
Author: Redacted <Redacted>
Date:   Fri Nov 5 06:56:58 2021 +0530

     Rename a.txt to b.txt


### Actual output

But when I use "--reverse" along with "--follow" option, I'm getting a 
single commit which points to the rename of the file.

$ git log --follow --reverse -- b.txt
commit 55e3e6857755fe815449e787a90fe82feb174817
Author: Redacted <Redacted>
Date:   Fri Nov 5 06:56:58 2021 +0530

     Rename a.txt to b.txt


### Additional

$ git --version
git version 2.33.0

First I thought "--follow --reverse" showing to me the newest commit. 
But later, I found out that, it always shows a single commit which 
points to the rename of the file.

$ echo "Hello from bar." >> b.txt
$ git add b.txt
$ git commit -m 'Update b.txt'
$ git log --follow --reverse -- b.txt
commit 55e3e6857755fe815449e787a90fe82feb174817
Author: Redacted <Redacted>
Date:   Fri Nov 5 06:56:58 2021 +0530

     Rename a.txt to b.txt



Cheers,
Vipul

On Fri, Nov 05, 2021 at 04:10:31AM +0000, Vipul Kumar wrote:

I want to list all commits (including renames) of a specified file in oldest
to newest order. But when I run "git log --follow --reverse --
<path/to/the/file>" command, I'm getting a single commit, which points to
the "rename of the file". This behavior is weird to me because I expected to
get list of all commit in oldest to newest order, whereas "git log --follow
-- <path/to/the/file>" command works as expected.

This has to do with the hacky way --follow is implemented. We don't
compute the full set of commits to show ahead of time, but rather as we
traverse, we change which pathspec we'll match when we hit a commit that
has a rename. Whereas --reverse is applied before we even start the main
traversal, and instead collect all possible commits and then walk them
in reverse order.

So the very first commit we'll look at for --follow is the one that
created the file from the rename, at which point we'll start looking
only for the old name. But of course all of the other commits post-date
the rename, so they don't use that old name.

So yes, it's a bug in the sense that the behavior is nonsense and it
doesn't work as one might expect. But it's also the hacky "--follow"
working as designed, and is just a limitation of its approach. It would
need to be rewritten completely to work correctly.  Arguably we should
at least disallow the combination of --reverse and --follow for now, as
it will never help (OTOH, if there is nothing to follow it should behave
OK, and I suspect some people and scripts may add --follow "just in
case").

As a workaround, you can get what you want by two separate traversals:
one to collect the commits via --follow, and then another to actually
show them (but without doing any further walking). Like:

  git log --follow --format=%H -- $your_file |
  git log --stdin --no-walk --reverse [--oneline, -p, etc]

-Peff

On Fri, Nov 05 2021, Vipul Kumar wrote:

I want to list all commits (including renames) of a specified file in
oldest to newest order. But when I run "git log --follow --reverse -- 
<path/to/the/file>" command, I'm getting a single commit, which points
to the "rename of the file". This behavior is weird to me because I 
expected to get list of all commit in oldest to newest order, whereas
"git log --follow -- <path/to/the/file>" command works as expected.

This is a known caveat of how the history traversal works, but from a
quick glance I couldn't see any explicit documentation for it, aside
from this terse mention in git-log(1):

     "[-M can be used] for following files across renames while
     traversing history, see --follow.".

The key thing being the "traversal", i.e. as we walk history we'll
encounter a tree entry where b.txt was deleted, and see that it was
moved from a.txt.

However, if we walk history from the beginning we have no idea of the
relationship of a->b.txt, since we didn't encounter that commit yet,
that's only something we see while walking the history.

Perhaps we should have some option like --buffer-then-reverse, which
wouldn't change the walking order, but would only change the output, but
we don't.

This caveat doesn't only apply to reverse, try to apply a move of b.txt
on top of your history:

    b.txt -> c.txt

And now do:

    git log [--follow] -- b.txt

What should we output there? If we're arguing that we should first
traverse the history to "look forward" that'll also apply to a
non-reverse walk, since we're asking to follow b.txt.

But we haven't encountered the b->c.txt relationship yet (well, we run
into the rename commit, but once you add a c->d.txt on top...). So maybe
instead of --buffer-then-reverse we'd need a hypothetical --two-pass,
which would also impact options other than --reverse whose behavior
relies on traversal order.

Take all the above as an explanation for how it works now, not some
defense of this being user-friendly. I've also often been annoyed at
this behavior.

For small sets you can feed your into an alias like:

    git show $(git -P log --pretty=format:"%h" --pretty=tformat:%H --follow -- b.txt | tac)

As a poor man's --buffer-then-reverse.

On 11/5/21 8:13 AM, Ævar Arnfjörð Bjarmason wrote:

The key thing being the "traversal", i.e. as we walk history we'll
encounter a tree entry where b.txt was deleted, and see that it was
moved from a.txt.

Thanks, I didn't know "reverse" would change the traversal order. When I 
looked for "--reverse" option in git-log(1), this what I found:

   --reverse
       Output the commits chosen to be shown (see Commit Limiting 
section above) in reverse order. Cannot be combined with --walk-reflogs.

 From this, I inferred that "--follow" would choose the commits and 
"--reverse" reverses those commits order. Can we improve the wording 
here? Especially, about "reverse" changes the traversing order.

However, if we walk history from the beginning we have no idea of the
relationship of a->b.txt, since we didn't encounter that commit yet,
that's only something we see while walking the history.

Not showing commits before rename is expected, but I didn't understand 
why combination "--follow" and "--reverse" option showing me only one 
commit? And it always points to the rename of the file. Shouldn't it 
also list other commits which changes that file? For example, just after 
"rename of a.txt to b.txt", do some changes in b.txt file and then run 
"git log --follow --reverse -- b.txt" command.

$ for i in {1..2}; do echo "$i" >> b.txt; git add b.txt; git commit -m 
"Update$i b.txt"; done
$ git log --follow --reverse -- b.txt
commit 55e3e6857755fe815449e787a90fe82feb174817
Author: Redacted <Redacted>
Date:   Fri Nov 5 06:56:58 2021 +0530

     Rename a.txt to b.txt

Here I expect output to be:

$ git log --follow --reverse -- b.txt
commit 55e3e6857755fe815449e787a90fe82feb174817
Author: Reacted <Redacted>
Date:   Fri Nov 5 06:56:58 2021 +0530

     Rename a.txt to b.txt

commit 57aac6d1af2d869557991e714932847f37035d19
Author: Redacted <Redacted>
Date:   Sun Nov 7 20:30:32 2021 +0530

     Update1 b.txt

commit ea76a8e8af903dc1522626aa058b8058afbe11f4
Author: Redacted <Redacted>
Date:   Sun Nov 7 20:30:32 2021 +0530

     Update2 b.txt

I know, here using "--follow" along with "--reverse" doesn't make any 
sense. Instead we should use "git log --reverse -- b.txt". But I'm just 
curious, is this also an expected caveats of using "--follow" along with 
"--reverse"?

This caveat doesn't only apply to reverse, try to apply a move of b.txt
on top of your history:

     b.txt -> c.txt

And now do:

     git log [--follow] -- b.txt

What should we output there? If we're arguing that we should first
traverse the history to "look forward" that'll also apply to a
non-reverse walk, since we're asking to follow b.txt.

But we haven't encountered the b->c.txt relationship yet (well, we run
into the rename commit, but once you add a c->d.txt on top...). So maybe
instead of --buffer-then-reverse we'd need a hypothetical --two-pass,
which would also impact options other than --reverse whose behavior
relies on traversal order.

"--two-pass" option sounds like a good idea, but not sure how useful it 
would be for others. In my case, I could read the log's command output 
in bottom to top fashion, and now I also know two other approaches to 
get what I wanted. And I usually don't track deleted file.


-v

On 11/5/21 8:17 AM, Jeff King wrote:

So yes, it's a bug in the sense that the behavior is nonsense and it
doesn't work as one might expect. But it's also the hacky "--follow"
working as designed, and is just a limitation of its approach. It would
need to be rewritten completely to work correctly.  Arguably we should
at least disallow the combination of --reverse and --follow for now, as
it will never help (OTOH, if there is nothing to follow it should behave
OK, and I suspect some people and scripts may add --follow "just in
case").

I think, it's a good idea to disallow the combination of "--reverse" and 
"--follow", just like we did for "--reverse" and "--walk-reflogs" 
combination. Because this combination never going to help anybody:

   * If there is nothing to follow, people shouldn't use "--follow" 
option. Using "--follow" along with "--reverse" might arise 
misconception to users that they are also following renames also, which 
actually they don't.

   * But if there are things to follow, using combination of "--follow" 
and "--reverse" only shows a single commit which points to the "rename 
of the file". I quite didn't understand why is this happening? Once we 
encounter rename commit (from "a.txt to b.txt") while traversing in 
reverse order, our pathspec is now "b.txt". So we should also get list 
of commits which changes b.txt file.

As a workaround, you can get what you want by two separate traversals:

   git log --follow --format=%H -- $your_file |
   git log --stdin --no-walk --reverse [--oneline, -p, etc]

Thank you!


-v

On Tue, Nov 09 2021, Vipul Kumar wrote:

On 11/5/21 8:13 AM, Ævar Arnfjörð Bjarmason wrote:

quoted

The key thing being the "traversal", i.e. as we walk history we'll
encounter a tree entry where b.txt was deleted, and see that it was
moved from a.txt.

Thanks, I didn't know "reverse" would change the traversal order. When
I looked for "--reverse" option in git-log(1), this what I found:

  --reverse
      Output the commits chosen to be shown (see Commit Limiting
      section above) in reverse order. Cannot be combined with
     --walk-reflogs.

From this, I inferred that "--follow" would choose the commits and
"--reverse" reverses those commits order. Can we improve the wording 
here? Especially, about "reverse" changes the traversing order.

Yes, definitely. Suggestions most welcome :)

quoted

However, if we walk history from the beginning we have no idea of the
relationship of a->b.txt, since we didn't encounter that commit yet,
that's only something we see while walking the history.

Not showing commits before rename is expected, but I didn't understand
why combination "--follow" and "--reverse" option showing me only one 
commit? And it always points to the rename of the file. Shouldn't it
also list other commits which changes that file? For example, just
after "rename of a.txt to b.txt", do some changes in b.txt file and
then run "git log --follow --reverse -- b.txt" command.

$ for i in {1..2}; do echo "$i" >> b.txt; git add b.txt; git commit -m
"Update$i b.txt"; done
$ git log --follow --reverse -- b.txt
commit 55e3e6857755fe815449e787a90fe82feb174817
Author: Redacted <Redacted>
Date:   Fri Nov 5 06:56:58 2021 +0530

    Rename a.txt to b.txt

Here I expect output to be:

$ git log --follow --reverse -- b.txt
commit 55e3e6857755fe815449e787a90fe82feb174817
Author: Reacted <Redacted>
Date:   Fri Nov 5 06:56:58 2021 +0530

    Rename a.txt to b.txt

commit 57aac6d1af2d869557991e714932847f37035d19
Author: Redacted <Redacted>
Date:   Sun Nov 7 20:30:32 2021 +0530

    Update1 b.txt

commit ea76a8e8af903dc1522626aa058b8058afbe11f4
Author: Redacted <Redacted>
Date:   Sun Nov 7 20:30:32 2021 +0530

    Update2 b.txt

I know, here using "--follow" along with "--reverse" doesn't make any
sense. Instead we should use "git log --reverse -- b.txt". But I'm
just curious, is this also an expected caveats of using "--follow"
along with "--reverse"?

I just assumed this would work, but looking a bit closer it doesn't
really, a better test-case:
    
    (   
        rm -rf /tmp/gt &&
        git init /tmp/gt &&
        cd /tmp/gt &&
        echo hi >a.txt &&
        git add a.txt &&
        git commit -m"initial" &&
        for i in {b..z}
        do
            git mv * $i.txt &&
            git commit -m"Moved to $i.txt"
        done
    )

And if you want to dig the ad-hoc fprintf debugging below is probably a
good start.

I.e. I think we should do this in principle, but I think we trip over
ourselves in the commit parent discovery, i.e. we should probably fake
up the "parent" as the next commit for the purposes of the traversal &
filter options.
    

quoted

This caveat doesn't only apply to reverse, try to apply a move of b.txt
on top of your history:
     b.txt -> c.txt
And now do:
     git log [--follow] -- b.txt
What should we output there? If we're arguing that we should first
traverse the history to "look forward" that'll also apply to a
non-reverse walk, since we're asking to follow b.txt.
But we haven't encountered the b->c.txt relationship yet (well, we
run
into the rename commit, but once you add a c->d.txt on top...). So maybe
instead of --buffer-then-reverse we'd need a hypothetical --two-pass,
which would also impact options other than --reverse whose behavior
relies on traversal order.

"--two-pass" option sounds like a good idea, but not sure how useful
it would be for others. In my case, I could read the log's command
output in bottom to top fashion, and now I also know two other
approaches to get what I wanted. And I usually don't track deleted
file.

I think it would be useful to be able to do "I know this file was
renamed to i.txt at some point, what happened after that?" If we start
at the tip we'll always traverse all the way to the root.

diff --git a/log-tree.c b/log-tree.c
index 644893fd8cf..b3894c995bd 100644
--- a/log-tree.c
+++ b/log-tree.c

@@ -960,6 +960,8 @@ static int log_tree_diff(struct rev_info *opt, struct commit *commit, struct log
 
 		/* Set up the log info for the next parent, if any.. */
 		parents = parents->next;
+		fprintf(stderr, "log_tree_diff() parent: %s (parents?: %d) (showed_log?: %d)\n",
+			oid_to_hex(&parent->object.oid), !!parents, showed_log);
 		if (!parents || opt->first_parent_merges)
 			break;
 		log->parent = parents->item;

diff --git a/tree-diff.c b/tree-diff.c
index 437c98a70e4..eeb7af0a516 100644
--- a/tree-diff.c
+++ b/tree-diff.c

@@ -588,6 +588,10 @@ static void try_to_follow_renames(const struct object_id *old_oid,
 	struct diff_filepair *choice;
 	int i;
 
+	fprintf(stderr, "try_to_follow_renames(): %s -> %s\n",
+		old_oid ? oid_to_hex(old_oid) : "NULL",
+		new_oid ? oid_to_hex(new_oid) : "NULL");
+
 	/*
 	 * follow-rename code is very specific, we need exactly one
 	 * path. Magic that matches more than one path is not

@@ -633,6 +637,7 @@ static void try_to_follow_renames(const struct object_id *old_oid,
 		 * diff_queued_diff, we will also use that as the path in
 		 * the future!
 		 */
+		fprintf(stderr, "%c: for p->one->path = %s...", p->status, p->one->path);
 		if ((p->status == 'R' || p->status == 'C') &&
 		    !strcmp(p->two->path, opt->pathspec.items[0].match)) {
 			const char *path[2];

@@ -657,9 +662,11 @@ static void try_to_follow_renames(const struct object_id *old_oid,
 			 * ourselves; signal diffcore_std() not to muck with
 			 * rename information.
 			 */
+			fprintf(stderr, "found\n");
 			opt->found_follow = 1;
 			break;
 		}
+		fprintf(stderr, "NOT found \n");
 	}
 
 	/*

@@ -704,6 +711,10 @@ void diff_tree_oid(const struct object_id *old_oid,
 {
 	struct strbuf base;
 
+	fprintf(stderr, "diff_tree_oid() %s -> %s\n",
+		old_oid ? oid_to_hex(old_oid) : "NULL",
+		new_oid ? oid_to_hex(new_oid) : "NULL");
+
 	strbuf_init(&base, PATH_MAX);
 	strbuf_addstr(&base, base_str);
 

Hi, sorry for delay reply. I was busy with some other works and couldn't 
find the time to look into it.

On 11/9/21 9:42 AM, Ævar Arnfjörð Bjarmason wrote:

quoted

Thanks, I didn't know "reverse" would change the traversal order. When
I looked for "--reverse" option in git-log(1), this what I found:

   --reverse
       Output the commits chosen to be shown (see Commit Limiting
       section above) in reverse order. Cannot be combined with
      --walk-reflogs.

 From this, I inferred that "--follow" would choose the commits and
"--reverse" reverses those commits order. Can we improve the wording
here? Especially, about "reverse" changes the traversing order.

Yes, definitely. Suggestions most welcome :)

   --reverse
       Traverse the commits chosen to be shown (see Commit Limiting 
section above) in reverse order. Cannot be combined with --walk-reflogs.

If we don't want to disallow combination of "--follow" and "--reverse", 
just like "--walk-reflogs" as combination doesn't make any sense[1]. We 
could do this:

   --reverse
       Traverse the commits chosen to be shown (see Commit Limiting 
section above) in reverse order. Cannot be combined with --walk-reflogs. 
Combination of --follow and --reverse won't produce an expected output, 
if we also want to follow renames. If there is nothing to follow it 
should behave as expected.

[1]: 
https://public-inbox.org/git/6c6ef97c-9de5-176f-f328-c4dffd96d495@onenetbeyond.org/

Does it sound good to you?

-v