A small background why I wanted this: I need to build a new version of
some software hosted in git, add a lot of shared/static stuff and
automatically test it. I want to get the most recent *tag* and create a
folder with the tagname in it, go into that folder and clone the repo,
check out the tag, add the rest of the stuff, build and test
As the default outpout of «git ls-remote --tags» or «git ls-remote» is
topologically sorted by tagname, the most recent tag is likely to be in
the middle.
Linux 4.12.14-lp150.12.16-default [openSUSE Leap 15.0]
$ git --version
git version 2.19.0
$ git ls-remote --tags github.com:Tux/App-ccdiff
5e2513ab6dd4a24c8f3a3ace0a3faba6a291d818 refs/tags/0.04
2f7ea0f1e751dc20c1ddb15f6d61c6fa62d5d6f1 refs/tags/0.05
a3802907be5b10383c7438f1d1c660fe13a05d3f refs/tags/0.06
3e4bfa7cde75fba221650b9d3aa5555b706803df refs/tags/0.07
05829d1ac5b49bbdd2167bc363b94f8a12e752b3 refs/tags/0.08
9c6e5861ea9c6e50c501663d43c5a9f6d31b54bc refs/tags/0.09
e815b059f6326da936c3a92272ba67e273b1dc3e refs/tags/0.10
e6b40e331c945449bb8e71023de4920ca5574adc refs/tags/0.20
be55e6336b1db5ffad23a6a0a663763e2f5da779 refs/tags/0.21
e283d563f02bb8d2131e8b95852072ac204b28b4 refs/tags/0.22
0d3d1830f542121bfef1d984f21343c6d9c774f8 refs/tags/0.23
d7bf195a92095a4f0b810584810450e4001b1a2c refs/tags/0.24
5c517cf3f79cb18173714e63bc5b80a3e3f888f1 refs/tags/0.25
Whether or not supported, it should not dump core
$ git ls-remote --tags --sort=authordate github.com:Tux/App-ccdiff
Segmentation fault (core dumped)
(gdb) where
#0 0x00007ffff74784a6 in __strlen_sse2 () from /lib64/libc.so.6
#1 0x000000000057a956 in for_each_replace_ref ()
#2 0x0000000000596cec in do_lookup_replace_object ()
#3 0x00000000005c14eb in oid_object_info_extended ()
#4 0x000000000058b984 in get_object ()
#5 0x000000000058ddde in populate_value ()
#6 0x000000000058e36b in compare_refs ()
#7 0x000000000061447a in msort_with_tmp.part ()
#8 0x0000000000614505 in msort_with_tmp.part ()
#9 0x0000000000614518 in msort_with_tmp.part ()
#10 0x0000000000614518 in msort_with_tmp.part ()
#11 0x000000000061459e in git_qsort_s ()
#12 0x000000000058ed40 in ref_array_sort ()
#13 0x000000000044ef66 in cmd_ls_remote ()
#14 0x000000000040784f in handle_builtin ()
#15 0x0000000000407bb0 in cmd_main ()
#16 0x0000000000406b04 in main ()
Linux 3.10.0-862.6.3.el7.x86_64 [CentOS Linux 7.5.1804 (Core)]
$ git --version
git version 2.18.0
$ git ls-remote --tags https://github.com/Tux/App-ccdiff
5e2513ab6dd4a24c8f3a3ace0a3faba6a291d818 refs/tags/0.04
2f7ea0f1e751dc20c1ddb15f6d61c6fa62d5d6f1 refs/tags/0.05
a3802907be5b10383c7438f1d1c660fe13a05d3f refs/tags/0.06
3e4bfa7cde75fba221650b9d3aa5555b706803df refs/tags/0.07
05829d1ac5b49bbdd2167bc363b94f8a12e752b3 refs/tags/0.08
9c6e5861ea9c6e50c501663d43c5a9f6d31b54bc refs/tags/0.09
e815b059f6326da936c3a92272ba67e273b1dc3e refs/tags/0.10
e6b40e331c945449bb8e71023de4920ca5574adc refs/tags/0.20
be55e6336b1db5ffad23a6a0a663763e2f5da779 refs/tags/0.21
e283d563f02bb8d2131e8b95852072ac204b28b4 refs/tags/0.22
0d3d1830f542121bfef1d984f21343c6d9c774f8 refs/tags/0.23
d7bf195a92095a4f0b810584810450e4001b1a2c refs/tags/0.24
5c517cf3f79cb18173714e63bc5b80a3e3f888f1 refs/tags/0.25
$ git ls-remote --tags --sort=authordate https://github.com/Tux/App-ccdiff
Segmentation fault
(gdb) where
#0 0x00007ffff751a67f in __strlen_sse42 () from /lib64/libc.so.6
#1 0x0000000000561c06 in for_each_replace_ref ()
#2 0x000000000057c3fa in do_lookup_replace_object ()
#3 0x00000000005a6aa8 in read_object_file_extended ()
#4 0x00000000005731e5 in get_object ()
#5 0x00000000005749df in populate_value ()
#6 0x0000000000574e9d in compare_refs ()
#7 0x00000000005efe57 in msort_with_tmp.part.0 ()
#8 0x00000000005efe31 in msort_with_tmp.part.0 ()
#9 0x00000000005efe0e in msort_with_tmp.part.0 ()
#10 0x00000000005efe0e in msort_with_tmp.part.0 ()
#11 0x00000000005eff5c in git_qsort_s ()
#12 0x00000000005757e0 in ref_array_sort ()
#13 0x000000000044c6b6 in cmd_ls_remote ()
#14 0x000000000040730e in handle_builtin ()
#15 0x000000000040760e in cmd_main ()
#16 0x0000000000406554 in main ()
--
H.Merijn Brand http://tux.nl Perl Monger http://amsterdam.pm.org/
using perl5.00307 .. 5.29 porting perl5 on HP-UX, AIX, and openSUSE
http://mirrors.develooper.com/hpux/http://www.test-smoke.org/http://qa.perl.orghttp://www.goldmark.org/jeff/stupid-disclaimers/
A small background why I wanted this: I need to build a new version of
some software hosted in git, add a lot of shared/static stuff and
automatically test it. I want to get the most recent *tag* and create a
folder with the tagname in it, go into that folder and clone the repo,
check out the tag, add the rest of the stuff, build and test
As the default outpout of «git ls-remote --tags» or «git ls-remote» is
topologically sorted by tagname, the most recent tag is likely to be in
the middle.
Linux 4.12.14-lp150.12.16-default [openSUSE Leap 15.0]
$ git --version
git version 2.19.0
$ git ls-remote --tags github.com:Tux/App-ccdiff
5e2513ab6dd4a24c8f3a3ace0a3faba6a291d818 refs/tags/0.04
2f7ea0f1e751dc20c1ddb15f6d61c6fa62d5d6f1 refs/tags/0.05
a3802907be5b10383c7438f1d1c660fe13a05d3f refs/tags/0.06
3e4bfa7cde75fba221650b9d3aa5555b706803df refs/tags/0.07
05829d1ac5b49bbdd2167bc363b94f8a12e752b3 refs/tags/0.08
9c6e5861ea9c6e50c501663d43c5a9f6d31b54bc refs/tags/0.09
e815b059f6326da936c3a92272ba67e273b1dc3e refs/tags/0.10
e6b40e331c945449bb8e71023de4920ca5574adc refs/tags/0.20
be55e6336b1db5ffad23a6a0a663763e2f5da779 refs/tags/0.21
e283d563f02bb8d2131e8b95852072ac204b28b4 refs/tags/0.22
0d3d1830f542121bfef1d984f21343c6d9c774f8 refs/tags/0.23
d7bf195a92095a4f0b810584810450e4001b1a2c refs/tags/0.24
5c517cf3f79cb18173714e63bc5b80a3e3f888f1 refs/tags/0.25
Whether or not supported, it should not dump core
$ git ls-remote --tags --sort=authordate github.com:Tux/App-ccdiff
Segmentation fault (core dumped)
(gdb) where
#0 0x00007ffff74784a6 in __strlen_sse2 () from /lib64/libc.so.6
#1 0x000000000057a956 in for_each_replace_ref ()
#2 0x0000000000596cec in do_lookup_replace_object ()
#3 0x00000000005c14eb in oid_object_info_extended ()
#4 0x000000000058b984 in get_object ()
#5 0x000000000058ddde in populate_value ()
#6 0x000000000058e36b in compare_refs ()
#7 0x000000000061447a in msort_with_tmp.part ()
#8 0x0000000000614505 in msort_with_tmp.part ()
#9 0x0000000000614518 in msort_with_tmp.part ()
#10 0x0000000000614518 in msort_with_tmp.part ()
#11 0x000000000061459e in git_qsort_s ()
#12 0x000000000058ed40 in ref_array_sort ()
#13 0x000000000044ef66 in cmd_ls_remote ()
#14 0x000000000040784f in handle_builtin ()
#15 0x0000000000407bb0 in cmd_main ()
#16 0x0000000000406b04 in main ()
Linux 3.10.0-862.6.3.el7.x86_64 [CentOS Linux 7.5.1804 (Core)]
$ git --version
git version 2.18.0
$ git ls-remote --tags https://github.com/Tux/App-ccdiff
5e2513ab6dd4a24c8f3a3ace0a3faba6a291d818 refs/tags/0.04
2f7ea0f1e751dc20c1ddb15f6d61c6fa62d5d6f1 refs/tags/0.05
a3802907be5b10383c7438f1d1c660fe13a05d3f refs/tags/0.06
3e4bfa7cde75fba221650b9d3aa5555b706803df refs/tags/0.07
05829d1ac5b49bbdd2167bc363b94f8a12e752b3 refs/tags/0.08
9c6e5861ea9c6e50c501663d43c5a9f6d31b54bc refs/tags/0.09
e815b059f6326da936c3a92272ba67e273b1dc3e refs/tags/0.10
e6b40e331c945449bb8e71023de4920ca5574adc refs/tags/0.20
be55e6336b1db5ffad23a6a0a663763e2f5da779 refs/tags/0.21
e283d563f02bb8d2131e8b95852072ac204b28b4 refs/tags/0.22
0d3d1830f542121bfef1d984f21343c6d9c774f8 refs/tags/0.23
d7bf195a92095a4f0b810584810450e4001b1a2c refs/tags/0.24
5c517cf3f79cb18173714e63bc5b80a3e3f888f1 refs/tags/0.25
$ git ls-remote --tags --sort=authordate https://github.com/Tux/App-ccdiff
Segmentation fault
(gdb) where
#0 0x00007ffff751a67f in __strlen_sse42 () from /lib64/libc.so.6
#1 0x0000000000561c06 in for_each_replace_ref ()
#2 0x000000000057c3fa in do_lookup_replace_object ()
#3 0x00000000005a6aa8 in read_object_file_extended ()
#4 0x00000000005731e5 in get_object ()
#5 0x00000000005749df in populate_value ()
#6 0x0000000000574e9d in compare_refs ()
#7 0x00000000005efe57 in msort_with_tmp.part.0 ()
#8 0x00000000005efe31 in msort_with_tmp.part.0 ()
#9 0x00000000005efe0e in msort_with_tmp.part.0 ()
#10 0x00000000005efe0e in msort_with_tmp.part.0 ()
#11 0x00000000005eff5c in git_qsort_s ()
#12 0x00000000005757e0 in ref_array_sort ()
#13 0x000000000044c6b6 in cmd_ls_remote ()
#14 0x000000000040730e in handle_builtin ()
#15 0x000000000040760e in cmd_main ()
#16 0x0000000000406554 in main ()
I can't reproduce this, I just get for both ssh and https:
$ ~/g/git/git --exec-path=$PWD ls-remote --tags --sort=authordate https://github.com/Tux/App-ccdiff
fatal: missing object 2f7ea0f1e751dc20c1ddb15f6d61c6fa62d5d6f1 for refs/tags/0.05
$ ~/g/git/git --exec-path=$PWD version
git version 2.18.0
Same thing on latest 'master' (v2.19.0-221-g150f307afc).
But maybe that's just a symptom of the same bug, when I clone the repo I
get a working 0.05 tag, and it passes fsck, and I do get the
2f7ea0f1e751dc20c1ddb15f6d61c6fa62d5d6f1 object (which is the 0.05 tag
object).
From: SZEDER Gábor <hidden> Date: 2018-09-22 14:12:19
The command 'git ls-remote --sort=authordate <remote>' segfaults when
run outside of a repository, ever since the introduction of its
'--sort' option in 1fb20dfd8e (ls-remote: create '--sort' option,
2018-04-09).
While in general the 'git ls-remote' command can be run outside of a
repository just fine, its '--sort=<key>' option with certain keys does
require access to the referenced objects. This sorting is implemented
using the generic ref-filter sorting facility, which already handles
missing objects gracefully with the appropriate 'missing object
deadbeef for HEAD' message. However, being generic means that it
checks replace refs while trying to retrieve an object, and while
doing so it accesses the 'git_replace_ref_base' variable, which has
not been initialized and is still a NULL pointer when outside of a
repository, thus causing the segfault.
Make ref-filter more careful and only attempt to retrieve an object
when we are in a repository. Also add a test to ensure that 'git
ls-remote --sort' fails gracefully when executed outside of a
repository.
Reported-by: H.Merijn Brand <redacted>
Signed-off-by: SZEDER Gábor <redacted>
---
I'm not quite sure that this is the best place to add this check...
but hey, it's a Saturday afternoon after all ;)
ref-filter.c | 3 ++-
t/t5512-ls-remote.sh | 6 ++++++
2 files changed, 8 insertions(+), 1 deletion(-)
@@ -302,4 +302,10 @@ test_expect_success 'ls-remote works outside repository' 'nongitgitls-remotedst.git'+test_expect_success'ls-remote --sort fails gracefully outside repository''+# Use a sort key that requires access to the referenced objects.+nongittest_must_failgitls-remote--sort=authordate"$TRASH_DIRECTORY"2>err&&+test_i18ngrep"^fatal: missing object"err+'+ test_done
From: Jeff King <hidden> Date: 2018-09-24 18:17:26
On Sat, Sep 22, 2018 at 04:11:45PM +0200, SZEDER Gábor wrote:
The command 'git ls-remote --sort=authordate <remote>' segfaults when
run outside of a repository, ever since the introduction of its
'--sort' option in 1fb20dfd8e (ls-remote: create '--sort' option,
2018-04-09).
While in general the 'git ls-remote' command can be run outside of a
repository just fine, its '--sort=<key>' option with certain keys does
require access to the referenced objects. This sorting is implemented
using the generic ref-filter sorting facility, which already handles
missing objects gracefully with the appropriate 'missing object
deadbeef for HEAD' message. However, being generic means that it
checks replace refs while trying to retrieve an object, and while
doing so it accesses the 'git_replace_ref_base' variable, which has
not been initialized and is still a NULL pointer when outside of a
repository, thus causing the segfault.
Make ref-filter more careful and only attempt to retrieve an object
when we are in a repository. Also add a test to ensure that 'git
ls-remote --sort' fails gracefully when executed outside of a
repository.
This all makes sense, and I think your fix is going in the right
direction.
But...
I'm not quite sure that this is the best place to add this check...
but hey, it's a Saturday afternoon after all ;)
I also wonder about this. For refs, we already catch these cases at a
low-level and BUG(). That's better than a segfault, and I suspect we
should be doing the same here in oid_object_info_extended(). But that
just shifts the segfault to a BUG().
For the refs code, we've generally tried to catch things at a high-level
and report a more human-friendly error explaining the situation. So
doing the same thing here would mean adding code to ls-remote. But I
think the plumbing gets pretty tricky, since it has no way to ask
ref-filter "hey, are we doing to need to look at objects?".
That's a thing that I think ref-filter _should_ support (it knows it
after having parsed the format string). But it probably ought to come
along with other refactoring, and shouldn't hold up this fix.
So this probably _is_ a reasonable place to check it. However...
@@ -1473,7 +1473,8 @@ static int get_object(struct ref_array_item *ref, int deref, struct object **objoi->info.sizep=&oi->size;oi->info.typep=&oi->type;}-if(oid_object_info_extended(the_repository,&oi->oid,&oi->info,+if(!have_git_dir()||+oid_object_info_extended(the_repository,&oi->oid,&oi->info,OBJECT_INFO_LOOKUP_REPLACE))returnstrbuf_addf_ret(err,-1,_("missing object %s for %s"),oid_to_hex(&oi->oid),ref->refname);
Would we perhaps want to give the user a hint that the object is not
really missing, but rather that we're not in a repository? E.g.,
something like:
if (!have_git_dir())
return strbuf_addf_ret(err, -1, "format specifier requires a repository");
if (oid_object_info_extended(...))
return ...;
?
-Peff
@@ -1473,7 +1473,8 @@ static int get_object(struct ref_array_item *ref, int deref, struct object **objoi->info.sizep=&oi->size;oi->info.typep=&oi->type;}-if(oid_object_info_extended(the_repository,&oi->oid,&oi->info,+if(!have_git_dir()||+oid_object_info_extended(the_repository,&oi->oid,&oi->info,OBJECT_INFO_LOOKUP_REPLACE))returnstrbuf_addf_ret(err,-1,_("missing object %s for %s"),oid_to_hex(&oi->oid),ref->refname);
Would we perhaps want to give the user a hint that the object is not
really missing, but rather that we're not in a repository? E.g.,
something like:
if (!have_git_dir())
return strbuf_addf_ret(err, -1, "format specifier requires a repository");
if (oid_object_info_extended(...))
return ...;
?
I think it makes sense.
I wanted to preserve the error message, because the description of
'--sort=<key>' in 'Documentation/git-ls-remote.txt' explicitly
mentions it, and I added the condition at this place because I didn't
want to duplicate the construction of the error message.
However, if we go for a more informative error message, then wouldn't
it be better to add this condition in populate_value() before it even
calls get_object()? Then we could also add the problematic format
specifier to the error message (I think, but didn't actually check),
just in case someone specified multiple sort keys.
From: Jeff King <hidden> Date: 2018-09-24 21:30:04
On Mon, Sep 24, 2018 at 11:20:34PM +0200, SZEDER Gábor wrote:
quoted
Would we perhaps want to give the user a hint that the object is not
really missing, but rather that we're not in a repository? E.g.,
something like:
if (!have_git_dir())
return strbuf_addf_ret(err, -1, "format specifier requires a repository");
if (oid_object_info_extended(...))
return ...;
?
I think it makes sense.
I wanted to preserve the error message, because the description of
'--sort=<key>' in 'Documentation/git-ls-remote.txt' explicitly
mentions it, and I added the condition at this place because I didn't
want to duplicate the construction of the error message.
Ah, I didn't realize we actually documented that. And perhaps it is more
consistent, too: you'd get different results from running "ls-remote"
outside a repository versus one that just doesn't have the objects from
the other side.
However, if we go for a more informative error message, then wouldn't
it be better to add this condition in populate_value() before it even
calls get_object()? Then we could also add the problematic format
specifier to the error message (I think, but didn't actually check),
just in case someone specified multiple sort keys.
Yeah, that probably would be a better place. Though your response also
has made me think that maybe just sticking with the "missing object"
response is reasonable. I don't have a strong opinion between the two.
-Peff