This is a continuation of Calvin Wan's (calvinwan@google.com)
patch series [PATCH v5 0/6] cat-file: add --batch-command remote-object-info command at [1].
Sometimes it is useful to get information about an object without having to download
it completely. The server logic for retrieving size has already been implemented and merged in
"a2ba162cda (object-info: support for retrieving object info, 2021-04-20)"[2].
This patch series implement the client option for it.
This patch series add the `remote-object-info` command to `cat-file --batch-command`. This command
allows the client to make an object-info command request to a server
that supports protocol v2. If the server is v2, but does not have
object-info capability, the entire object is fetched and the
relevant object info is returned.
A few questions open for discussions please:
1. In the current implementation, if a user puts `remote-object-info` in protocol v1,
`cat-file --batch-command` will die. Which way do we prefer? "error and exit (i.e. die)"
or "warn and wait for new command".
2. Right now, only the size is supported. If the batch command format
contains objectsize:disk or deltabase, it will die. The question
is about objecttype. In the current implementation, it will die too.
But dying on objecttype breaks the default format. We have changed the
default format to %(objectname) %(objectsize) when remote-object-info is used.
Any suggestions on this approach?
[1] https://lore.kernel.org/git/20220728230210.2952731-1-calvinwan@google.com/#t
[2] https://git.kernel.org/pub/scm/git/git.git/commit/?id=a2ba162cda2acc171c3e36acbbc854792b093cb7
Calvin Wan (5):
fetch-pack: refactor packet writing
fetch-pack: move fetch initialization
serve: advertise object-info feature
transport: add client support for object-info
cat-file: add remote-object-info to batch-command
Eric Ju (1):
cat-file: add declaration of variable i inside its for loop
Documentation/git-cat-file.txt | 22 +-
builtin/cat-file.c | 240 ++++++++++----
fetch-pack.c | 48 ++-
fetch-pack.h | 10 +
object-file.c | 11 +
object-store-ll.h | 3 +
serve.c | 4 +-
t/t1017-cat-file-remote-object-info.sh | 412 +++++++++++++++++++++++++
transport-helper.c | 8 +-
transport.c | 102 +++++-
transport.h | 11 +
11 files changed, 785 insertions(+), 86 deletions(-)
create mode 100755 t/t1017-cat-file-remote-object-info.sh
--
2.45.2
From: Calvin Wan <redacted>
A subsequent patch need to write capabilities for another command.
Refactor write_fetch_command_and_capabilities() to be used by both
fetch and future command.
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
---
fetch-pack.c | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
From: Calvin Wan <redacted>
There are some variables initialized at the start of the
do_fetch_pack_v2() state machine. Currently, they are initialized
in FETCH_CHECK_LOCAL, which is the initial state set at the beginning
of the function.
However, a subsequent patch will allow for another initial state,
while still requiring these initialized variables.
Move the initialization to be before the state machine,
so that they are set regardless of the initial state.
Note that there is no change in behavior, because we're moving code
from the beginning of the first state to just before the execution of
the state machine.
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
---
fetch-pack.c | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
@@ -1676,18 +1676,18 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,reader.me="fetch-pack";}+/* v2 supports these by default */+allow_unadvertised_object_request|=ALLOW_REACHABLE_SHA1;+use_sideband=2;+if(args->depth>0||args->deepen_since||args->deepen_not)+args->deepen=1;+while(state!=FETCH_DONE){switch(state){caseFETCH_CHECK_LOCAL:sort_ref_list(&ref,ref_compare_name);QSORT(sought,nr_sought,cmp_ref_by_name);-/* v2 supports these by default */-allow_unadvertised_object_request|=ALLOW_REACHABLE_SHA1;-use_sideband=2;-if(args->depth>0||args->deepen_since||args->deepen_not)-args->deepen=1;-/* Filter 'ref' by 'sought' and those that aren't local */mark_complete_and_common_ref(negotiator,args,&ref);filter_refs(args,&ref,sought,nr_sought);
From: Calvin Wan <redacted>
In order for a client to know what object-info components a server can
provide, advertise supported object-info features. This will allow a
client to decide whether to query the server for object-info or fetch
as a fallback.
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
---
serve.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
From: Calvin Wan <redacted>
Sometimes it is useful to get information about an object without having
to download it completely. The server logic has already been implemented
as “a2ba162cda (object-info: support for retrieving object info,
2021-04-20)”.
Add client functions to communicate with the server.
The client currently supports requesting a list of object ids with
features 'size' and 'type' from a v2 server. If a server does not
advertise either of the requested features, then the client falls back
to making the request through 'fetch'.
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
---
fetch-pack.c | 24 +++++++++++
fetch-pack.h | 10 +++++
transport-helper.c | 8 +++-
transport.c | 102 ++++++++++++++++++++++++++++++++++++++++++---
transport.h | 11 +++++
5 files changed, 148 insertions(+), 7 deletions(-)
@@ -697,13 +697,17 @@ static int fetch_refs(struct transport *transport,/**Ifwereachhere,thentheserver,theclient,and/orthetransport-*helperdoesnotsupportprotocolv2.--negotiate-onlyrequires-*protocolv2.+*helperdoesnotsupportprotocolv2.--negotiate-onlyandcat-fileremote-object-info+*requireprotocolv2.*/if(data->transport_options.acked_commits){warning(_("--negotiate-only requires protocol v2"));return-1;}+if(transport->smart_options->object_info){+// fail the command explicitly to avoid further commands input+die(_("remote-object-info requires protocol v2"));+}if(!data->get_refs_list_called)get_refs_list_using_list(transport,0);
@@ -363,6 +363,73 @@ static struct ref *handshake(struct transport *transport, int for_push,returnrefs;}+staticintfetch_object_info(structtransport*transport,structobject_info**object_info_data)+{+intsize_index=-1;+structgit_transport_data*data=transport->data;+structobject_info_argsargs;+structpacket_readerreader;++memset(&args,0,sizeof(args));+args.server_options=transport->server_options;+args.object_info_options=transport->smart_options->object_info_options;+args.oids=transport->smart_options->object_info_oids;++connect_setup(transport,0);+packet_reader_init(&reader,data->fd[0],NULL,0,+PACKET_READ_CHOMP_NEWLINE|+PACKET_READ_GENTLE_ON_EOF|+PACKET_READ_DIE_ON_ERR_PACKET);+data->version=discover_version(&reader);++transport->hash_algo=reader.hash_algo;++switch(data->version){+caseprotocol_v2:+if(!server_supports_v2("object-info"))+return-1;+if(unsorted_string_list_has_string(args.object_info_options,"size")+&&!server_supports_feature("object-info","size",0)){+return-1;+}+send_object_info_request(data->fd[1],&args);+break;+caseprotocol_v1:+caseprotocol_v0:+die(_("wrong protocol version. expected v2"));+caseprotocol_unknown_version:+BUG("unknown protocol version");+}++for(size_ti=0;i<args.object_info_options->nr;i++){+if(packet_reader_read(&reader)!=PACKET_READ_NORMAL){+check_stateless_delimiter(transport->stateless_rpc,&reader,"stateless delimiter expected");+return-1;+}+if(unsorted_string_list_has_string(args.object_info_options,reader.line)){+if(!strcmp(reader.line,"size"))+size_index=i;+continue;+}+return-1;+}++for(size_ti=0;packet_reader_read(&reader)==PACKET_READ_NORMAL&&i<args.oids->nr;i++){+structstring_listobject_info_values=STRING_LIST_INIT_DUP;++string_list_split(&object_info_values,reader.line,' ',-1);+if(0<=size_index){+if(!strcmp(object_info_values.items[1+size_index].string,""))+die("object-info: not our ref %s",+object_info_values.items[0].string);+*(*object_info_data)[i].sizep=strtoul(object_info_values.items[1+size_index].string,NULL,10);+}+}+check_stateless_delimiter(transport->stateless_rpc,&reader,"stateless delimiter expected");++return0;+}+staticstructref*get_refs_via_connect(structtransport*transport,intfor_push,structtransport_ls_refs_options*options){
@@ -410,6 +477,7 @@ static int fetch_refs_via_pack(struct transport *transport,structref*refs=NULL;structfetch_pack_argsargs;structref*refs_tmp=NULL;+structref*object_info_refs=xcalloc(1,sizeof(structref));memset(&args,0,sizeof(args));args.uploadpack=data->options.uploadpack;
@@ -436,11 +504,27 @@ static int fetch_refs_via_pack(struct transport *transport,args.server_options=transport->server_options;args.negotiation_tips=data->options.negotiation_tips;args.reject_shallow_remote=transport->smart_options->reject_shallow;--if(!data->finished_handshake){-inti;+args.object_info=transport->smart_options->object_info;++if(transport->smart_options&&transport->smart_options->object_info){+structref*ref=object_info_refs;++if(!fetch_object_info(transport,data->options.object_info_data))+gotocleanup;+args.object_info_data=data->options.object_info_data;+args.quiet=1;+args.no_progress=1;+for(size_ti=0;i<transport->smart_options->object_info_oids->nr;i++){+structref*temp_ref=xcalloc(1,sizeof(structref));+temp_ref->old_oid=*(transport->smart_options->object_info_oids->oid+i);+temp_ref->exact_oid=1;+ref->next=temp_ref;+ref=ref->next;+}+transport->remote_refs=object_info_refs->next;+}elseif(!data->finished_handshake){intmust_list_refs=0;-for(i=0;i<nr_heads;i++){+for(inti=0;i<nr_heads;i++){if(!to_fetch[i]->exact_oid){must_list_refs=1;break;
@@ -478,11 +562,18 @@ static int fetch_refs_via_pack(struct transport *transport,&transport->pack_lockfiles,data->version);data->finished_handshake=0;+if(args.object_info){+structref*ref_cpy_reader=object_info_refs->next;+for(inti=0;ref_cpy_reader;i++){+oid_object_info_extended(the_repository,&ref_cpy_reader->old_oid,&(*args.object_info_data)[i],OBJECT_INFO_LOOKUP_REPLACE);+ref_cpy_reader=ref_cpy_reader->next;+}+}data->options.self_contained_and_connected=args.self_contained_and_connected;data->options.connectivity_checked=args.connectivity_checked;-if(!refs)+if(!refs&&!args.object_info)ret=-1;if(report_unmatched_refs(to_fetch,nr_heads))ret=-1;
@@ -498,6 +589,7 @@ static int fetch_refs_via_pack(struct transport *transport,free_refs(refs_tmp);free_refs(refs);list_objects_filter_release(&args.filter_options);+free_refs(object_info_refs);returnret;}
Some code declares variable i and only uses it
in a for loop, not in any other logic outside the loop.
Change the declaration of i to be inside the for loop for readability.
Signed-off-by: Eric Ju <redacted>
Helped-by: Christian Couder [off-list ref]
---
builtin/cat-file.c | 11 +++--------
1 file changed, 3 insertions(+), 8 deletions(-)
@@ -668,12 +668,10 @@ static void dispatch_calls(struct batch_options *opt,structqueued_cmd*cmd,intnr){-inti;-if(!opt->buffer_output)die(_("flush is only for --buffer mode"));-for(i=0;i<nr;i++)+for(inti=0;i<nr;i++)cmd[i].fn(opt,cmd[i].line,output,data);fflush(stdout);
From: Calvin Wan <redacted>
Since the `info` command in cat-file --batch-command prints object info
for a given object, it is natural to add another command in cat-file
--batch-command to print object info for a given object from a remote.
Add `remote-object-info` to cat-file --batch-command.
While `info` takes object ids one at a time, this creates overhead when
making requests to a server so `remote-object-info` instead can take
multiple object ids at once.
cat-file --batch-command is generally implemented in the following
manner:
- Receive and parse input from user
- Call respective function attached to command
- Set batch mode state, get object info, print object info
In --buffer mode, this changes to:
- Receive and parse input from user
- Store respective function attached to command in a queue
- After flush, loop through commands in queue
- Call respective function attached to command
- Set batch mode state, get object info, print object info
Notice how the getting and printing of object info is accomplished one
at a time. As described above, this creates a problem for making
requests to a server. Therefore, `remote-object-info` is implemented in
the following manner:
- Receive and parse input from user
If command is `remote-object-info`:
- Get object info from remote
- Loop through object info
- Call respective function attached to `info`
- Set batch mode state, use passed in object info, print object
info
Else:
- Call respective function attached to command
- Parse input, get object info, print object info
And finally for --buffer mode `remote-object-info`:
- Receive and parse input from user
- Store respective function attached to command in a queue
- After flush, loop through commands in queue:
If command is `remote-object-info`:
- Get object info from remote
- Loop through object info
- Call respective function attached to `info`
- Set batch mode state, use passed in object info, print
object info
Else:
- Call respective function attached to command
- Set batch mode state, get object info, print object info
To summarize, `remote-object-info` gets object info from the remote and
then generates multiple `info` commands with the object info passed in.
In order for remote-object-info to avoid remote communication overhead
in the non-buffer mode, the objects are passed in as such:
remote-object-info <remote> <oid> <oid> ... <oid>
rather than
remote-object-info <remote> <oid>
remote-object-info <remote> <oid>
...
remote-object-info <remote> <oid>
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
---
Documentation/git-cat-file.txt | 22 +-
builtin/cat-file.c | 231 ++++++++++----
object-file.c | 11 +
object-store-ll.h | 3 +
t/t1017-cat-file-remote-object-info.sh | 412 +++++++++++++++++++++++++
5 files changed, 620 insertions(+), 59 deletions(-)
create mode 100755 t/t1017-cat-file-remote-object-info.sh
@@ -149,6 +149,12 @@ info <object>:: Print object info for object reference `<object>`. This corresponds to the output of `--batch-check`.+remote-object-info <remote> <object>...::+ Print object info for object references `<object>` at specified <remote> without+ downloading object from remote.+ Error when no object references is provided.+ This command may be combined with `--buffer`.+ flush:: Used with `--buffer` to execute all preceding commands that were issued since the beginning or since the last flush was issued. When `--buffer`
@@ -290,7 +296,8 @@ newline. The available atoms are: The full hex representation of the object name. `objecttype`::- The type of the object (the same as `cat-file -t` reports).+ The type of the object (the same as `cat-file -t` reports). See+ `CAVEATS` below. Not supported by `remote-object-info`. `objectsize`:: The size, in bytes, of the object (the same as `cat-file -s`
@@ -298,13 +305,14 @@ newline. The available atoms are: `objectsize:disk`:: The size, in bytes, that the object takes up on disk. See the- note about on-disk sizes in the `CAVEATS` section below.+ note about on-disk sizes in the `CAVEATS` section below. Not+ supported by `remote-object-info`. `deltabase`:: If the object is stored as a delta on-disk, this expands to the full hex representation of the delta base object name. Otherwise, expands to the null OID (all zeroes). See `CAVEATS`- below.+ below. Not supported by `remote-object-info`. `rest`:: If this atom is used in the output string, input lines are split
@@ -314,7 +322,9 @@ newline. The available atoms are: line) are output in place of the `%(rest)` atom. If no format is specified, the default format is `%(objectname)-%(objecttype) %(objectsize)`.+%(objecttype) %(objectsize)`, except remote-object-info command who uses+`%(objectname) %(objectsize)` for now because "%(objecttype)" is not supported yet.+When "%(objecttype)" is supported, default format should be unified. If `--batch` is specified, or if `--batch-command` is used with the `contents` command, the object information is followed by the object contents (consisting
@@ -396,6 +406,10 @@ scripting purposes. CAVEATS -------+Note that since objecttype, objectsize:disk and deltabase are currently not supported by the+remote-object-info, git will error and exit when they are in the format string.++ Note that the sizes of objects on disk are reported accurately, but care should be taken in drawing conclusions about which refs or objects are responsible for disk usage. The size of a packed non-delta object may be
@@ -526,51 +533,118 @@ static void batch_one_object(const char *obj_name,(opt->follow_symlinks?GET_OID_FOLLOW_SYMLINKS:0);enumget_oid_resultresult;-result=get_oid_with_context(the_repository,obj_name,-flags,&data->oid,&ctx);-if(result!=FOUND){-switch(result){-caseMISSING_OBJECT:-printf("%s missing%c",obj_name,opt->output_delim);-break;-caseSHORT_NAME_AMBIGUOUS:-printf("%s ambiguous%c",obj_name,opt->output_delim);-break;-caseDANGLING_SYMLINK:-printf("dangling %"PRIuMAX"%c%s%c",-(uintmax_t)strlen(obj_name),-opt->output_delim,obj_name,opt->output_delim);-break;-caseSYMLINK_LOOP:-printf("loop %"PRIuMAX"%c%s%c",-(uintmax_t)strlen(obj_name),-opt->output_delim,obj_name,opt->output_delim);-break;-caseNOT_DIR:-printf("notdir %"PRIuMAX"%c%s%c",-(uintmax_t)strlen(obj_name),-opt->output_delim,obj_name,opt->output_delim);-break;-default:-BUG("unknown get_sha1_with_context result %d\n",-result);-break;+if(!opt->use_remote_info){+result=get_oid_with_context(the_repository,obj_name,+flags,&data->oid,&ctx);+if(result!=FOUND){+switch(result){+caseMISSING_OBJECT:+printf("%s missing%c",obj_name,opt->output_delim);+break;+caseSHORT_NAME_AMBIGUOUS:+printf("%s ambiguous%c",obj_name,opt->output_delim);+break;+caseDANGLING_SYMLINK:+printf("dangling %"PRIuMAX"%c%s%c",+(uintmax_t)strlen(obj_name),+opt->output_delim,obj_name,opt->output_delim);+break;+caseSYMLINK_LOOP:+printf("loop %"PRIuMAX"%c%s%c",+(uintmax_t)strlen(obj_name),+opt->output_delim,obj_name,opt->output_delim);+break;+caseNOT_DIR:+printf("notdir %"PRIuMAX"%c%s%c",+(uintmax_t)strlen(obj_name),+opt->output_delim,obj_name,opt->output_delim);+break;+default:+BUG("unknown get_sha1_with_context result %d\n",+result);+break;+}+fflush(stdout);+return;}-fflush(stdout);-return;-}-if(ctx.mode==0){-printf("symlink %"PRIuMAX"%c%s%c",-(uintmax_t)ctx.symlink_path.len,-opt->output_delim,ctx.symlink_path.buf,opt->output_delim);-fflush(stdout);-return;+if(ctx.mode==0){+printf("symlink %"PRIuMAX"%c%s%c",+(uintmax_t)ctx.symlink_path.len,+opt->output_delim,ctx.symlink_path.buf,opt->output_delim);+fflush(stdout);+return;+}}batch_object_write(obj_name,scratch,opt,data,NULL,0);}+staticintget_remote_info(structbatch_options*opt,intargc,constchar**argv)+{+intretval=0;+structremote*remote=NULL;+structobject_idoid;+structstring_listobject_info_options=STRING_LIST_INIT_NODUP;+staticstructtransport*gtransport;++/*+*Changetheformatto"%(objectname) %(objectsize)"when+*remote-object-infocommandisused.Oncewestartsupportingobjecttype+*thedefaultformatshouldchangetoDEFAULT_FORMAT+*/+if(!opt->format){+opt->format="%(objectname) %(objectsize)";+}++remote=remote_get(argv[0]);+if(!remote)+die(_("must supply valid remote when using remote-object-info"));+oid_array_clear(&object_info_oids);+for(size_ti=1;i<argc;i++){+if(get_oid_hex(argv[i],&oid))+die(_("Not a valid object name %s"),argv[i]);+oid_array_append(&object_info_oids,&oid);+}++gtransport=transport_get(remote,NULL);+if(gtransport->smart_options){+intinclude_size=0;++CALLOC_ARRAY(remote_object_info,object_info_oids.nr);+gtransport->smart_options->object_info=1;+gtransport->smart_options->object_info_oids=&object_info_oids;+/*+*'size'istheonlyoptioncurrentlysupported.+*Otheroptionsthatarepassedintheformatwillexitwitherror.+*/+if(strstr(opt->format,"%(objectsize)")){+string_list_append(&object_info_options,"size");+include_size=1;+}+if(strstr(opt->format,"%(objecttype)")){+die(_("objecttype is currently not supported with remote-object-info"));+}+if(strstr(opt->format,"%(objectsize:disk)"))+die(_("objectsize:disk is currently not supported with remote-object-info"));+if(strstr(opt->format,"%(deltabase)"))+die(_("deltabase is currently not supported with remote-object-info"));+if(object_info_options.nr>0){+gtransport->smart_options->object_info_options=&object_info_options;+for(size_ti=0;i<object_info_oids.nr;i++){+if(include_size)+remote_object_info[i].sizep=xcalloc(1,sizeof(long));+}+gtransport->smart_options->object_info_data=&remote_object_info;+retval=transport_fetch_refs(gtransport,NULL);+}+}else{+retval=-1;+}++returnretval;+}+structobject_cb_data{structbatch_options*opt;structexpand_data*expand;
@@ -671,8 +795,12 @@ static void dispatch_calls(struct batch_options *opt,if(!opt->buffer_output)die(_("flush is only for --buffer mode"));-for(inti=0;i<nr;i++)-cmd[i].fn(opt,cmd[i].line,output,data);+for(inti=0;i<nr;i++){+if(!strcmp(cmd[i].name,"remote-object-info"))+parse_remote_info(opt,cmd[i].line,output,data,NULL,&cmd[i]);+else+cmd[i].fn(opt,cmd[i].line,output,data);+}fflush(stdout);}
@@ -0,0 +1,412 @@+#!/bin/sh++test_description='git cat-file --batch-command with remote-object-info command'++GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main+exportGIT_TEST_DEFAULT_INITIAL_BRANCH_NAME++../test-lib.sh++echo_without_newline(){+printf'%s'"$*"+}++strlen(){+echo_without_newline"$1"|wc-c|sed-e's/^ *//'+}++hello_content="Hello World"+hello_size=$(strlen"$hello_content")+hello_oid=$(echo_without_newline"$hello_content"|githash-object--stdin)++tree_size=$(($(test_oidrawsz)+13))++commit_message="Initial commit"+commit_size=$(($(test_oidhexsz)+137))++tag_header_without_oid="type blob+taghellotag+tagger$GIT_COMMITTER_NAME<$GIT_COMMITTER_EMAIL>"+tag_header_without_timestamp="object $hello_oid+$tag_header_without_oid"+tag_description="This is a tag"+tag_content="$tag_header_without_timestamp 0 +0000++$tag_description"++tag_oid=$(echo_without_newline"$tag_content"|githash-object-ttag--stdin-w)+tag_size=$(strlen"$tag_content")++# This section tests --batch-command with remote-object-info command+# Since "%(objecttype)" is currently not supported by the command remote-object-info ,+# the filters are set to "%(objectname) %(objectsize)".+# Tests with the default filter are used to test the fallback to 'fetch' command+++# Test --batch-command remote-object-info with 'git://' transport++."$TEST_DIRECTORY"/lib-git-daemon.sh+start_git_daemon--export-all--enable=receive-pack+daemon_parent=$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent++test_expect_success'create repo to be served by git-daemon''+gitinit"$daemon_parent"&&++echo_without_newline"$hello_content">$daemon_parent/hello&&+git-C"$daemon_parent"update-index--addhello&&+git-C"$daemon_parent"configtransfer.advertiseobjectinfotrue+'++set_transport_variables(){+hello_sha1=$(echo_without_newline"$hello_content"|githash-object--stdin)+tree_sha1=$(git-C"$1"write-tree)+commit_sha1=$(echo_without_newline"$commit_message"|git-C"$1"commit-tree$tree_sha1)+tag_sha1=$(echo_without_newline"$tag_content"|git-C"$1"hash-object-ttag--stdin-w)+tag_size=$(strlen"$tag_content")+}+++test_expect_success'batch-command remote-object-info git://''+(+set_transport_variables"$daemon_parent"&&+cd"$daemon_parent"&&++echo"$hello_sha1$hello_size">expect&&+echo"$tree_sha1$tree_size">>expect&&+echo"$commit_sha1$commit_size">>expect&&+echo"$tag_sha1$tag_size">>expect&&+gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+remote-object-info"$GIT_DAEMON_URL/parent"$hello_sha1+remote-object-info"$GIT_DAEMON_URL/parent"$tree_sha1+remote-object-info"$GIT_DAEMON_URL/parent"$commit_sha1+remote-object-info"$GIT_DAEMON_URL/parent"$tag_sha1+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command remote-object-info git:// multiple sha1 per line''+(+set_transport_variables"$daemon_parent"&&+cd"$daemon_parent"&&++echo"$hello_sha1$hello_size">expect&&+echo"$tree_sha1$tree_size">>expect&&+echo"$commit_sha1$commit_size">>expect&&+echo"$tag_sha1$tag_size">>expect&&+gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+remote-object-info"$GIT_DAEMON_URL/parent"$hello_sha1$tree_sha1$commit_sha1$tag_sha1+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command remote-object-info http:// default filter''+(+set_transport_variables"$daemon_parent"&&+cd"$daemon_parent"&&++echo"$hello_sha1$hello_size">expect&&+echo"$tree_sha1$tree_size">>expect&&+echo"$commit_sha1$commit_size">>expect&&+echo"$tag_sha1$tag_size">>expect&&+GIT_TRACE_PACKET=1gitcat-file--batch-command>actual<<-EOF&&+remote-object-info"$GIT_DAEMON_URL/parent"$hello_sha1$tree_sha1+remote-object-info"$GIT_DAEMON_URL/parent"$commit_sha1$tag_sha1+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command --buffer remote-object-info git://''+(+set_transport_variables"$daemon_parent"&&+cd"$daemon_parent"&&++echo"$hello_sha1$hello_size">expect&&+echo"$tree_sha1$tree_size">>expect&&+echo"$commit_sha1$commit_size">>expect&&+echo"$tag_sha1$tag_size">>expect&&+gitcat-file--batch-command="%(objectname) %(objectsize)"--buffer>actual<<-EOF&&+remote-object-info"$GIT_DAEMON_URL/parent"$hello_sha1$tree_sha1+remote-object-info"$GIT_DAEMON_URL/parent"$commit_sha1$tag_sha1+flush+EOF+test_cmpexpectactual+)+'++stop_git_daemon++# Test --batch-command remote-object-info with 'http://' transport++."$TEST_DIRECTORY"/lib-httpd.sh+start_httpd++test_expect_success'create repo to be served by http:// transport''+gitinit"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+git-C"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"confighttp.receivepacktrue&&+git-C"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"configtransfer.advertiseobjectinfotrue&&+echo_without_newline"$hello_content">$HTTPD_DOCUMENT_ROOT_PATH/http_parent/hello&&+git-C"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"update-index--addhello+'+++test_expect_success'batch-command remote-object-info http://''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&++echo"$hello_sha1$hello_size">expect&&+echo"$tree_sha1$tree_size">>expect&&+echo"$commit_sha1$commit_size">>expect&&+echo"$tag_sha1$tag_size">>expect&&+gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_sha1+remote-object-info"$HTTPD_URL/smart/http_parent"$tree_sha1+remote-object-info"$HTTPD_URL/smart/http_parent"$commit_sha1+remote-object-info"$HTTPD_URL/smart/http_parent"$tag_sha1+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command remote-object-info http:// one line''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&++echo"$hello_sha1$hello_size">expect&&+echo"$tree_sha1$tree_size">>expect&&+echo"$commit_sha1$commit_size">>expect&&+echo"$tag_sha1$tag_size">>expect&&+gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_sha1$tree_sha1$commit_sha1$tag_sha1+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command --buffer remote-object-info http://''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&++echo"$hello_sha1$hello_size">expect&&+echo"$tree_sha1$tree_size">>expect&&+echo"$commit_sha1$commit_size">>expect&&+echo"$tag_sha1$tag_size">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)"--buffer>actual<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_sha1$tree_sha1+remote-object-info"$HTTPD_URL/smart/http_parent"$commit_sha1$tag_sha1+flush+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command remote-object-info http:// default filter''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&++echo"$hello_sha1$hello_size">expect&&+echo"$tree_sha1$tree_size">>expect&&+echo"$commit_sha1$commit_size">>expect&&+echo"$tag_sha1$tag_size">>expect&&++gitcat-file--batch-command>actual<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_sha1$tree_sha1+remote-object-info"$HTTPD_URL/smart/http_parent"$commit_sha1$tag_sha1+EOF+test_cmpexpectactual+)+'++test_expect_success'remote-object-info fails on unspported filter option (objectsize:disk)''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&++test_must_failgitcat-file--batch-command="%(objectsize:disk)"2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_sha1+EOF+test_grep"objectsize:disk is currently not supported with remote-object-info"err+)+'++test_expect_success'remote-object-info fails on unspported filter option (deltabase)''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&++test_must_failgitcat-file--batch-command="%(deltabase)"2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_sha1+EOF+test_grep"deltabase is currently not supported with remote-object-info"err+)+'++test_expect_success'remote-object-info fails on server with legacy protocol''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&++test_must_failgit-cprotocol.version=0cat-file--batch-command="%(objectname) %(objectsize)"2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_sha1+EOF+test_grep"remote-object-info requires protocol v2"err+)+'++test_expect_success'remote-object-info fails on server with legacy protocol fallback''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&++test_must_failgit-cprotocol.version=0cat-file--batch-command2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_sha1+EOF+test_grep"remote-object-info requires protocol v2"err+)+'++test_expect_success'remote-object-info fails on malformed OID''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+malformed_object_id="this_id_is_not_valid"&&++test_must_failgitcat-file--batch-command="%(objectname) %(objectsize)"2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$malformed_object_id+EOF+test_grep"Not a valid object name '$malformed_object_id'"err+)+'++test_expect_success'remote-object-info fails on malformed OID fallback''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+malformed_object_id="this_id_is_not_valid"&&++test_must_failgitcat-file--batch-command2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$malformed_object_id+EOF+test_grep"Not a valid object name '$malformed_object_id'"err+)+'++test_expect_success'remote-object-info fails on missing OID''+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+gitclone"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"missing_oid_repo&&+test_commit-Cmissing_oid_repomessage1c.txt&&+(+cdmissing_oid_repo&&++object_id=$(gitrev-parsemessage1:c.txt)&&+test_must_failgitcat-file--batch-command="%(objectname) %(objectsize)"2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$object_id+EOF+test_grep"object-info: not our ref $object_id"err+)+'++# shellcheck disable=SC2016+test_expect_success'remote-object-info fails on missing OID fallback''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cdmissing_oid_repo&&+object_id=$(gitrev-parsemessage1:c.txt)&&+test_must_failgitcat-file--batch-command2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$object_id+EOF+test_grep"fatal: object-info: not our ref $object_id"err+)+'++# Test --batch-command remote-object-info with 'file://' transport++# shellcheck disable=SC2016+test_expect_success'create repo to be served by file:// transport''+gitinitserver&&+git-Cserverconfigprotocol.version2&&+git-Cserverconfigtransfer.advertiseobjectinfotrue&&+echo_without_newline"$hello_content">server/hello&&+git-Cserverupdate-index--addhello+'+++test_expect_success'batch-command remote-object-info file://''+(+set_transport_variables"server"&&+cdserver&&++echo"$hello_sha1$hello_size">expect&&+echo"$tree_sha1$tree_size">>expect&&+echo"$commit_sha1$commit_size">>expect&&+echo"$tag_sha1$tag_size">>expect&&+gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+remote-object-info"file://$(pwd)"$hello_sha1+remote-object-info"file://$(pwd)"$tree_sha1+remote-object-info"file://$(pwd)"$commit_sha1+remote-object-info"file://$(pwd)"$tag_sha1+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command remote-object-info file:// multiple sha1 per line''+(+set_transport_variables"server"&&+cdserver&&++echo"$hello_sha1$hello_size">expect&&+echo"$tree_sha1$tree_size">>expect&&+echo"$commit_sha1$commit_size">>expect&&+echo"$tag_sha1$tag_size">>expect&&+gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+remote-object-info"file://$(pwd)"$hello_sha1$tree_sha1$commit_sha1$tag_sha1+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command --buffer remote-object-info file://''+(+set_transport_variables"server"&&+cdserver&&++echo"$hello_sha1$hello_size">expect&&+echo"$tree_sha1$tree_size">>expect&&+echo"$commit_sha1$commit_size">>expect&&+echo"$tag_sha1$tag_size">>expect&&+gitcat-file--batch-command="%(objectname) %(objectsize)"--buffer>actual<<-EOF&&+remote-object-info"file://$(pwd)"$hello_sha1$tree_sha1+remote-object-info"file://$(pwd)"$commit_sha1$tag_sha1+flush+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command remote-object-info file:// default filter''+(+set_transport_variables"server"&&+cdserver&&++echo"$hello_sha1$hello_size">expect&&+echo"$tree_sha1$tree_size">>expect&&+echo"$commit_sha1$commit_size">>expect&&+echo"$tag_sha1$tag_size">>expect&&++gitcat-file--batch-command>actual<<-EOF&&+remote-object-info"file://$(pwd)"$hello_sha1$tree_sha1+remote-object-info"file://$(pwd)"$commit_sha1$tag_sha1+EOF+test_cmpexpectactual+)+'++test_done
From: Calvin Wan <redacted>
A subsequent patch need to write capabilities for another command.
s/need/needs
Refactor write_fetch_command_and_capabilities() to be used by both
fetch and future command.
Nit: mostly from my lack of understanding, but until I read the code, I
couldn't understand what 'command' meant in this para. Maybe some
preface would be nice here.
quoted hunk
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
---
fetch-pack.c | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
Right, this commit in itself looks good. But I was curious why we need
this, so I did a sneak peak into the following commits.
To summarize, we want to call:
`write_command_and_capabilities(..., "object-info");`
in the upcoming patches to get the object-info details from the server.
But isn't this function too specific to the "fetch" command to be
generalized to be for "object-info" too?
Wouldn't it make sense to add a custom function for 'object-info' in
'connect.c'? Like how we currently have `get_remote_bundle_uri()` for
'bundle-uri' and `get_remote_refs` for 'ls-refs'?
From: Peijian Ju <hidden> Date: 2024-07-08 15:17:57
On Thu, Jul 4, 2024 at 1:00 PM Karthik Nayak [off-list ref] wrote:
Eric Ju [off-list ref] writes:
quoted
From: Calvin Wan <redacted>
A subsequent patch need to write capabilities for another command.
s/need/needs
Thank you. Fixed in v2.
quoted
Refactor write_fetch_command_and_capabilities() to be used by both
fetch and future command.
Nit: mostly from my lack of understanding, but until I read the code, I
couldn't understand what 'command' meant in this para. Maybe some
preface would be nice here.
Thank you. I will add this in v2 commit message.
Here "command" means the "operations" supported by Git’s wire protocol
https://git-scm.com/docs/protocol-v2. An example would be a
git's subcommand, such as git-fetch(1); or an operation supported by
the server side such as "object-info" implemented at "a2ba162cda
(object-info: support for retrieving object info, 2021-04-20)".
quoted
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
---
fetch-pack.c | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
Right, this commit in itself looks good. But I was curious why we need
this, so I did a sneak peak into the following commits.
To summarize, we want to call:
`write_command_and_capabilities(..., "object-info");`
in the upcoming patches to get the object-info details from the server.
But isn't this function too specific to the "fetch" command to be
generalized to be for "object-info" too?
Wouldn't it make sense to add a custom function for 'object-info' in
'connect.c'? Like how we currently have `get_remote_bundle_uri()` for
'bundle-uri' and `get_remote_refs` for 'ls-refs'?
Thank you. I am reading through the old comments left by Taylor
at https://lore.kernel.org/git/YkOPyc9tUfe2Tozx@nand.local/
" Makes obvious sense, and this was something that jumped out to me when I
looked at the first and second versions of this patch. I'm glad that
this is getting factored out."
It seems refactoring this into a more general function is on purpose.
It is encouraged to use this general function to request capability
rather than adding a custom function.
Taylor’s comment was 2 years ago, but I think refactoring this into a
more general function to
enforce DRY still makes sense.
From: Calvin Wan <redacted>
Since the `info` command in cat-file --batch-command prints object info
for a given object, it is natural to add another command in cat-file
--batch-command to print object info for a given object from a remote.
Add `remote-object-info` to cat-file --batch-command.
While `info` takes object ids one at a time, this creates overhead when
making requests to a server so `remote-object-info` instead can take
multiple object ids at once.
cat-file --batch-command is generally implemented in the following
manner:
- Receive and parse input from user
- Call respective function attached to command
- Set batch mode state, get object info, print object info
In --buffer mode, this changes to:
- Receive and parse input from user
- Store respective function attached to command in a queue
- After flush, loop through commands in queue
- Call respective function attached to command
- Set batch mode state, get object info, print object info
So the problem is that there is overhead associated with getting object
info from the remote. Therefore, remote-object-info also supports
batching objects together. This seems reasonable.
Notice how the getting and printing of object info is accomplished one
at a time. As described above, this creates a problem for making
requests to a server. Therefore, `remote-object-info` is implemented in
the following manner:
- Receive and parse input from user
If command is `remote-object-info`:
- Get object info from remote
- Loop through object info
- Call respective function attached to `info`
- Set batch mode state, use passed in object info, print object
info
Else:
- Call respective function attached to command
- Parse input, get object info, print object info
And finally for --buffer mode `remote-object-info`:
- Receive and parse input from user
- Store respective function attached to command in a queue
- After flush, loop through commands in queue:
If command is `remote-object-info`:
- Get object info from remote
- Loop through object info
- Call respective function attached to `info`
- Set batch mode state, use passed in object info, print
object info
Else:
- Call respective function attached to command
- Set batch mode state, get object info, print object info
To summarize, `remote-object-info` gets object info from the remote and
then generates multiple `info` commands with the object info passed in.
In order for remote-object-info to avoid remote communication overhead
in the non-buffer mode, the objects are passed in as such:
Even in non-buffer mode, having separate remote-object-info commands
would result in additional overhead correct? From my understanding each
command is executed sequently, so multiples of remote-object-info would
always result in additional overhead.
remote-object-info <remote> <oid> <oid> ... <oid>
rather than
remote-object-info <remote> <oid>
remote-object-info <remote> <oid>
...
remote-object-info <remote> <oid>
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
I think the sign-offs are supposed to go at the bottom.
[snip]
quoted hunk
@@ -526,51 +533,118 @@ static void batch_one_object(const char *obj_name, (opt->follow_symlinks ? GET_OID_FOLLOW_SYMLINKS : 0); enum get_oid_result result;- result = get_oid_with_context(the_repository, obj_name,- flags, &data->oid, &ctx);- if (result != FOUND) {- switch (result) {- case MISSING_OBJECT:- printf("%s missing%c", obj_name, opt->output_delim);- break;- case SHORT_NAME_AMBIGUOUS:- printf("%s ambiguous%c", obj_name, opt->output_delim);- break;- case DANGLING_SYMLINK:- printf("dangling %"PRIuMAX"%c%s%c",- (uintmax_t)strlen(obj_name),- opt->output_delim, obj_name, opt->output_delim);- break;- case SYMLINK_LOOP:- printf("loop %"PRIuMAX"%c%s%c",- (uintmax_t)strlen(obj_name),- opt->output_delim, obj_name, opt->output_delim);- break;- case NOT_DIR:- printf("notdir %"PRIuMAX"%c%s%c",- (uintmax_t)strlen(obj_name),- opt->output_delim, obj_name, opt->output_delim);- break;- default:- BUG("unknown get_sha1_with_context result %d\n",- result);- break;+ if (!opt->use_remote_info) {
When using the remote-object-info command, the object in question is
supposed to be on the remote and may not exist locally. Therefore we
skip over `get_oid_with_context()`.
+ result = get_oid_with_context(the_repository, obj_name,
+ flags, &data->oid, &ctx);
+ if (result != FOUND) {
+ switch (result) {
+ case MISSING_OBJECT:
+ printf("%s missing%c", obj_name, opt->output_delim);
+ break;
+ case SHORT_NAME_AMBIGUOUS:
+ printf("%s ambiguous%c", obj_name, opt->output_delim);
+ break;
+ case DANGLING_SYMLINK:
+ printf("dangling %"PRIuMAX"%c%s%c",
+ (uintmax_t)strlen(obj_name),
+ opt->output_delim, obj_name, opt->output_delim);
+ break;
+ case SYMLINK_LOOP:
+ printf("loop %"PRIuMAX"%c%s%c",
+ (uintmax_t)strlen(obj_name),
+ opt->output_delim, obj_name, opt->output_delim);
+ break;
+ case NOT_DIR:
+ printf("notdir %"PRIuMAX"%c%s%c",
+ (uintmax_t)strlen(obj_name),
+ opt->output_delim, obj_name, opt->output_delim);
+ break;
+ default:
+ BUG("unknown get_sha1_with_context result %d\n",
+ result);
+ break;
+ }
+ fflush(stdout);
+ return;
}
- fflush(stdout);
- return;
- }
- if (ctx.mode == 0) {
- printf("symlink %"PRIuMAX"%c%s%c",
- (uintmax_t)ctx.symlink_path.len,
- opt->output_delim, ctx.symlink_path.buf, opt->output_delim);
- fflush(stdout);
- return;
+ if (ctx.mode == 0) {
+ printf("symlink %"PRIuMAX"%c%s%c",
+ (uintmax_t)ctx.symlink_path.len,
+ opt->output_delim, ctx.symlink_path.buf, opt->output_delim);
+ fflush(stdout);
+ return;
+ }
}
batch_object_write(obj_name, scratch, opt, data, NULL, 0);
}
+static int get_remote_info(struct batch_options *opt, int argc, const char **argv)
+{
+ int retval = 0;
+ struct remote *remote = NULL;
+ struct object_id oid;
+ struct string_list object_info_options = STRING_LIST_INIT_NODUP;
+ static struct transport *gtransport;
+
+ /*
+ * Change the format to "%(objectname) %(objectsize)" when
+ * remote-object-info command is used. Once we start supporting objecttype
+ * the default format should change to DEFAULT_FORMAT
+ */
+ if (!opt->format) {
+ opt->format = "%(objectname) %(objectsize)";
+ }
We should omit the parenthesis for single line if statements.
+
+ remote = remote_get(argv[0]);
+ if (!remote)
+ die(_("must supply valid remote when using remote-object-info"));
+ oid_array_clear(&object_info_oids);
+ for (size_t i = 1; i < argc; i++) {
+ if (get_oid_hex(argv[i], &oid))
+ die(_("Not a valid object name %s"), argv[i]);
+ oid_array_append(&object_info_oids, &oid);
+ }
+
+ gtransport = transport_get(remote, NULL);
+ if (gtransport->smart_options) {
+ int include_size = 0;
+
+ CALLOC_ARRAY(remote_object_info, object_info_oids.nr);
+ gtransport->smart_options->object_info = 1;
+ gtransport->smart_options->object_info_oids = &object_info_oids;
+ /*
+ * 'size' is the only option currently supported.
+ * Other options that are passed in the format will exit with error.
+ */
+ if (strstr(opt->format, "%(objectsize)")) {
+ string_list_append(&object_info_options, "size");
+ include_size = 1;
+ }
+ if (strstr(opt->format, "%(objecttype)")) {
+ die(_("objecttype is currently not supported with remote-object-info"));
+ }
Another single line if statement above that should omit the parenthesis.
quoted hunk
+ if (strstr(opt->format, "%(objectsize:disk)"))
+ die(_("objectsize:disk is currently not supported with remote-object-info"));
+ if (strstr(opt->format, "%(deltabase)"))
+ die(_("deltabase is currently not supported with remote-object-info"));
+ if (object_info_options.nr > 0) {
+ gtransport->smart_options->object_info_options = &object_info_options;
+ for (size_t i = 0; i < object_info_oids.nr; i++) {
+ if (include_size)
+ remote_object_info[i].sizep = xcalloc(1, sizeof(long));
+ }
+ gtransport->smart_options->object_info_data = &remote_object_info;
+ retval = transport_fetch_refs(gtransport, NULL);
+ }
+ } else {
+ retval = -1;
+ }
+
+ return retval;
+}
+
struct object_cb_data {
struct batch_options *opt;
struct expand_data *expand;
It seems a little confusing to me that `parse_remote_info()` accepts
both a `parse_cmd` and `queued_cmd`, but only expects to use one or the
other. It looks like this is done because `dispatch_calls()` already
accepts `queued_cmd`, but now needs to call `parse_remote_info()`.
Since it is only the underlying command function that is needed by
`parse_remote_info()`
@@ -671,8 +795,12 @@ static void dispatch_calls(struct batch_options *opt, if (!opt->buffer_output) die(_("flush is only for --buffer mode"));- for (int i = 0; i < nr; i++)- cmd[i].fn(opt, cmd[i].line, output, data);+ for (int i = 0; i < nr; i++) {+ if (!strcmp(cmd[i].name, "remote-object-info"))+ parse_remote_info(opt, cmd[i].line, output, data, NULL, &cmd[i]);
If we adapt `parse_remote_info()` to accept the command function we
could pass cmd->fn here instead.
@@ -436,11 +504,27 @@ static int fetch_refs_via_pack(struct transport *transport,args.server_options=transport->server_options;args.negotiation_tips=data->options.negotiation_tips;args.reject_shallow_remote=transport->smart_options->reject_shallow;--if(!data->finished_handshake){-inti;+args.object_info=transport->smart_options->object_info;++if(transport->smart_options&&transport->smart_options->object_info){+structref*ref=object_info_refs;++if(!fetch_object_info(transport,data->options.object_info_data))+gotocleanup;+args.object_info_data=data->options.object_info_data;+args.quiet=1;+args.no_progress=1;+for(size_ti=0;i<transport->smart_options->object_info_oids->nr;i++){+structref*temp_ref=xcalloc(1,sizeof(structref));+temp_ref->old_oid=*(transport->smart_options->object_info_oids->oid+i);
Any reason why you're not using the subscript operator (square brackets)
like this:
+ temp_ref->old_oid = transport->smart_options->object_info_oids->oid[i];
@@ -479,7 +479,7 @@ static int fetch_refs_via_pack(struct transport *transport,structref*refs=NULL;structfetch_pack_argsargs;structref*refs_tmp=NULL;-structref*object_info_refs=xcalloc(1,sizeof(structref));+structref*object_info_refs=NULL;memset(&args,0,sizeof(args));args.uploadpack=data->options.uploadpack;
@@ -509,7 +509,7 @@ static int fetch_refs_via_pack(struct transport *transport,args.object_info=transport->smart_options->object_info;if(transport->smart_options&&transport->smart_options->object_info){-structref*ref=object_info_refs;+structref*ref=object_info_refs=xcalloc(1,sizeof(structref));if(!fetch_object_info(transport,data->options.object_info_data))gotocleanup;
@@ -517,13 +517,12 @@ static int fetch_refs_via_pack(struct transport *transport,args.quiet=1;args.no_progress=1;for(size_ti=0;i<transport->smart_options->object_info_oids->nr;i++){-structref*temp_ref=xcalloc(1,sizeof(structref));-temp_ref->old_oid=*(transport->smart_options->object_info_oids->oid+i);-temp_ref->exact_oid=1;-ref->next=temp_ref;+ref->old_oid=transport->smart_options->object_info_oids->oid[i];+ref->exact_oid=1;+ref->next=xcalloc(1,sizeof(structref));ref=ref->next;}-transport->remote_refs=object_info_refs->next;+transport->remote_refs=object_info_refs;}elseif(!data->finished_handshake){intmust_list_refs=0;for(inti=0;i<nr_heads;i++){
@@ -565,7 +564,7 @@ static int fetch_refs_via_pack(struct transport *transport,data->finished_handshake=0;if(args.object_info){-structref*ref_cpy_reader=object_info_refs->next;+structref*ref_cpy_reader=object_info_refs;for(inti=0;ref_cpy_reader;i++){oid_object_info_extended(the_repository,&ref_cpy_reader->old_oid,&(*args.object_info_data)[i],OBJECT_INFO_LOOKUP_REPLACE);ref_cpy_reader=ref_cpy_reader->next;----8<-----8<----
To be honest, I'm not sure it works, because fetch_object_info() always
seem to return a non-zero value. I'm not sure this is due to missing
code coverage, or a bug. I guess it's worth looking into.
--
Toon
...
+static int get_remote_info(struct batch_options *opt, int argc, const char **argv)
+{
+ int retval = 0;
+ struct remote *remote = NULL;
+ struct object_id oid;
+ struct string_list object_info_options = STRING_LIST_INIT_NODUP;
+ static struct transport *gtransport;
+
+ /*
+ * Change the format to "%(objectname) %(objectsize)" when
+ * remote-object-info command is used. Once we start supporting objecttype
+ * the default format should change to DEFAULT_FORMAT
+ */
I believe this comment has become outdated, or got moved around
incorrectly.
...
+stop_git_daemon
+
+# Test --batch-command remote-object-info with 'http://' transport
+
+. "$TEST_DIRECTORY"/lib-httpd.sh
+start_httpd
start_httpd skips the remainder of the tests if it fails to start the
httpd server. That's why I see various other tests which have this at
the end:
# DO NOT add non-httpd-specific tests here, because the last part of this
# test script is only executed when httpd is available and enabled.
So I would suggest to add this comment as well, and move the file://
tests above start_httpd.
--
Toon
Right, this commit in itself looks good. But I was curious why we need
this, so I did a sneak peak into the following commits.
To summarize, we want to call:
`write_command_and_capabilities(..., "object-info");`
in the upcoming patches to get the object-info details from the server.
But isn't this function too specific to the "fetch" command to be
generalized to be for "object-info" too?
Wouldn't it make sense to add a custom function for 'object-info' in
'connect.c'? Like how we currently have `get_remote_bundle_uri()` for
'bundle-uri' and `get_remote_refs` for 'ls-refs'?
Thank you. I am reading through the old comments left by Taylor
at https://lore.kernel.org/git/YkOPyc9tUfe2Tozx@nand.local/
" Makes obvious sense, and this was something that jumped out to me when I
looked at the first and second versions of this patch. I'm glad that
this is getting factored out."
It seems refactoring this into a more general function is on purpose.
It is encouraged to use this general function to request capability
rather than adding a custom function.
Taylor’s comment was 2 years ago, but I think refactoring this into a
more general function to
enforce DRY still makes sense.
It would make sense then to move the existing users to also use
`write_command_and_capabilities` eventually. I guess this could be done
in a follow up series.
Then I would say `write_command_and_capabilities()` should be moved to
`transport.c`, no?
From: Calvin Wan <redacted>
Sometimes it is useful to get information about an object without having
to download it completely. The server logic has already been implemented
as “a2ba162cda (object-info: support for retrieving object info,
Nit: s/as/in
2021-04-20)”.
Add client functions to communicate with the server.
The client currently supports requesting a list of object ids with
features 'size' and 'type' from a v2 server. If a server does not
But do we support type? I thought we only added support for 'size'.
quoted hunk
advertise either of the requested features, then the client falls back
to making the request through 'fetch'.
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
---
fetch-pack.c | 24 +++++++++++
fetch-pack.h | 10 +++++
transport-helper.c | 8 +++-
transport.c | 102 ++++++++++++++++++++++++++++++++++++++++++---
transport.h | 11 +++++
5 files changed, 148 insertions(+), 7 deletions(-)
@@ -697,13 +697,17 @@ static int fetch_refs(struct transport *transport,/**Ifwereachhere,thentheserver,theclient,and/orthetransport-*helperdoesnotsupportprotocolv2.--negotiate-onlyrequires-*protocolv2.+*helperdoesnotsupportprotocolv2.--negotiate-onlyandcat-fileremote-object-info+*requireprotocolv2.*/if(data->transport_options.acked_commits){warning(_("--negotiate-only requires protocol v2"));return-1;}+if(transport->smart_options->object_info){+// fail the command explicitly to avoid further commands input+die(_("remote-object-info requires protocol v2"));+}if(!data->get_refs_list_called)get_refs_list_using_list(transport,0);
Perhaps `*object_info_data[i]->sizep =
strtoul(object_info_values.items[1 + size_index].string, NULL, 10);`?
So, this is allocated in 'cat-file' and set here? Wouldn't it be nicer
to also do the alloc here?
Some code declares variable i and only uses it
in a for loop, not in any other logic outside the loop.
Change the declaration of i to be inside the for loop for readability.
If we're doing this anyways, we could replace the 'int' with 'size_t'
too.
[snip]
From: Calvin Wan <redacted>
Since the `info` command in cat-file --batch-command prints object info
for a given object, it is natural to add another command in cat-file
--batch-command to print object info for a given object from a remote.
Add `remote-object-info` to cat-file --batch-command.
While `info` takes object ids one at a time, this creates overhead when
making requests to a server so `remote-object-info` instead can take
multiple object ids at once.
cat-file --batch-command is generally implemented in the following
manner:
- Receive and parse input from user
So this refers input delimited by newline or '\0'.
- Call respective function attached to command
- Set batch mode state, get object info, print object info
Doesn't the batch mode get set before the input parsing begins?
In --buffer mode, this changes to:
- Receive and parse input from user
- Store respective function attached to command in a queue
- After flush, loop through commands in queue
- Call respective function attached to command
- Set batch mode state, get object info, print object info
Notice how the getting and printing of object info is accomplished one
at a time. As described above, this creates a problem for making
requests to a server. Therefore, `remote-object-info` is implemented in
the following manner:
- Receive and parse input from user
If command is `remote-object-info`:
- Get object info from remote
- Loop through object info
- Call respective function attached to `info`
- Set batch mode state, use passed in object info, print object
info
Else:
- Call respective function attached to command
- Parse input, get object info, print object info
So this is because we want 'remote-object-info' to also use
'parse_cmd_info' similar to 'info'. But I'm not understanding why,
especially since 'parse_cmd_info' calls 'batch_one_object', and we skip
most of that code for 'remote-object-info'.
Wouldn't it be cleaner to just define our own 'batch_remote_object' and
create 'parse_cmd_remote_info' ?
quoted hunk
And finally for --buffer mode `remote-object-info`:
- Receive and parse input from user
- Store respective function attached to command in a queue
- After flush, loop through commands in queue:
If command is `remote-object-info`:
- Get object info from remote
- Loop through object info
- Call respective function attached to `info`
- Set batch mode state, use passed in object info, print
object info
Else:
- Call respective function attached to command
- Set batch mode state, get object info, print object info
To summarize, `remote-object-info` gets object info from the remote and
then generates multiple `info` commands with the object info passed in.
In order for remote-object-info to avoid remote communication overhead
in the non-buffer mode, the objects are passed in as such:
remote-object-info <remote> <oid> <oid> ... <oid>
rather than
remote-object-info <remote> <oid>
remote-object-info <remote> <oid>
...
remote-object-info <remote> <oid>
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
---
Documentation/git-cat-file.txt | 22 +-
builtin/cat-file.c | 231 ++++++++++----
object-file.c | 11 +
object-store-ll.h | 3 +
t/t1017-cat-file-remote-object-info.sh | 412 +++++++++++++++++++++++++
5 files changed, 620 insertions(+), 59 deletions(-)
create mode 100755 t/t1017-cat-file-remote-object-info.sh
@@ -149,6 +149,12 @@ info <object>:: Print object info for object reference `<object>`. This corresponds to the output of `--batch-check`.+remote-object-info <remote> <object>...::+ Print object info for object references `<object>` at specified <remote> without+ downloading object from remote.+ Error when no object references is provided.+ This command may be combined with `--buffer`.+ flush:: Used with `--buffer` to execute all preceding commands that were issued since the beginning or since the last flush was issued. When `--buffer`
@@ -290,7 +296,8 @@ newline. The available atoms are: The full hex representation of the object name. `objecttype`::- The type of the object (the same as `cat-file -t` reports).+ The type of the object (the same as `cat-file -t` reports). See+ `CAVEATS` below. Not supported by `remote-object-info`. `objectsize`:: The size, in bytes, of the object (the same as `cat-file -s`
@@ -298,13 +305,14 @@ newline. The available atoms are: `objectsize:disk`:: The size, in bytes, that the object takes up on disk. See the- note about on-disk sizes in the `CAVEATS` section below.+ note about on-disk sizes in the `CAVEATS` section below. Not+ supported by `remote-object-info`. `deltabase`:: If the object is stored as a delta on-disk, this expands to the full hex representation of the delta base object name. Otherwise, expands to the null OID (all zeroes). See `CAVEATS`- below.+ below. Not supported by `remote-object-info`. `rest`:: If this atom is used in the output string, input lines are split
@@ -314,7 +322,9 @@ newline. The available atoms are: line) are output in place of the `%(rest)` atom. If no format is specified, the default format is `%(objectname)-%(objecttype) %(objectsize)`.+%(objecttype) %(objectsize)`, except remote-object-info command who uses+`%(objectname) %(objectsize)` for now because "%(objecttype)" is not supported yet.+When "%(objecttype)" is supported, default format should be unified. If `--batch` is specified, or if `--batch-command` is used with the `contents` command, the object information is followed by the object contents (consisting
@@ -396,6 +406,10 @@ scripting purposes. CAVEATS -------+Note that since objecttype, objectsize:disk and deltabase are currently not supported by the+remote-object-info, git will error and exit when they are in the format string.++ Note that the sizes of objects on disk are reported accurately, but care should be taken in drawing conclusions about which refs or objects are responsible for disk usage. The size of a packed non-delta object may be
Shouldn't we call `transport_disconnect(transport);`?
+ /*
+ * Change the format to "%(objectname) %(objectsize)" when
+ * remote-object-info command is used. Once we start supporting objecttype
+ * the default format should change to DEFAULT_FORMAT
+ */
+ if (!opt->format) {
+ opt->format = "%(objectname) %(objectsize)";
+ }
+
+ remote = remote_get(argv[0]);
+ if (!remote)
+ die(_("must supply valid remote when using remote-object-info"));
+ oid_array_clear(&object_info_oids);
+ for (size_t i = 1; i < argc; i++) {
+ if (get_oid_hex(argv[i], &oid))
+ die(_("Not a valid object name %s"), argv[i]);
+ oid_array_append(&object_info_oids, &oid);
+ }
+
+ gtransport = transport_get(remote, NULL);
+ if (gtransport->smart_options) {
+ int include_size = 0;
+
+ CALLOC_ARRAY(remote_object_info, object_info_oids.nr);
+ gtransport->smart_options->object_info = 1;
+ gtransport->smart_options->object_info_oids = &object_info_oids;
+ /*
+ * 'size' is the only option currently supported.
+ * Other options that are passed in the format will exit with error.
+ */
+ if (strstr(opt->format, "%(objectsize)")) {
+ string_list_append(&object_info_options, "size");
+ include_size = 1;
+ }
+ if (strstr(opt->format, "%(objecttype)")) {
+ die(_("objecttype is currently not supported with remote-object-info"));
+ }
+ if (strstr(opt->format, "%(objectsize:disk)"))
+ die(_("objectsize:disk is currently not supported with remote-object-info"));
+ if (strstr(opt->format, "%(deltabase)"))
+ die(_("deltabase is currently not supported with remote-object-info"));
This whole block could be replaced by an else..
if (strstr(opt->format, "%(objectsize)")) {
string_list_append(&object_info_options, "size");
include_size = 1;
} else {
die(_("%s is currently not supported with remote-object-info", opt->format));
}
Some more tests I'd like to see
- Testing against the '-Z' option.
- Testing the fallback to fetch whole object when the server doesn't
support 'remote-object-info'.
Thanks
From: Peijian Ju <hidden> Date: 2024-07-12 17:41:37
On Mon, Jul 8, 2024 at 9:51 PM Justin Tobler [off-list ref] wrote:
On 24/06/28 03:05PM, Eric Ju wrote:
quoted
From: Calvin Wan <redacted>
Since the `info` command in cat-file --batch-command prints object info
for a given object, it is natural to add another command in cat-file
--batch-command to print object info for a given object from a remote.
Add `remote-object-info` to cat-file --batch-command.
While `info` takes object ids one at a time, this creates overhead when
making requests to a server so `remote-object-info` instead can take
multiple object ids at once.
cat-file --batch-command is generally implemented in the following
manner:
- Receive and parse input from user
- Call respective function attached to command
- Set batch mode state, get object info, print object info
In --buffer mode, this changes to:
- Receive and parse input from user
- Store respective function attached to command in a queue
- After flush, loop through commands in queue
- Call respective function attached to command
- Set batch mode state, get object info, print object info
So the problem is that there is overhead associated with getting object
info from the remote. Therefore, remote-object-info also supports
batching objects together. This seems reasonable.
Thank you, Justin. Yes, you are right, whenever remote-object-info is
called there is an overhead. I will explain where this overhead
happens in the following reply.
quoted
Notice how the getting and printing of object info is accomplished one
at a time. As described above, this creates a problem for making
requests to a server. Therefore, `remote-object-info` is implemented in
the following manner:
- Receive and parse input from user
If command is `remote-object-info`:
- Get object info from remote
- Loop through object info
- Call respective function attached to `info`
- Set batch mode state, use passed in object info, print object
info
Else:
- Call respective function attached to command
- Parse input, get object info, print object info
And finally for --buffer mode `remote-object-info`:
- Receive and parse input from user
- Store respective function attached to command in a queue
- After flush, loop through commands in queue:
If command is `remote-object-info`:
- Get object info from remote
- Loop through object info
- Call respective function attached to `info`
- Set batch mode state, use passed in object info, print
object info
Else:
- Call respective function attached to command
- Set batch mode state, get object info, print object info
To summarize, `remote-object-info` gets object info from the remote and
then generates multiple `info` commands with the object info passed in.
In order for remote-object-info to avoid remote communication overhead
in the non-buffer mode, the objects are passed in as such:
Even in non-buffer mode, having separate remote-object-info commands
would result in additional overhead correct? From my understanding each
command is executed sequently, so multiples of remote-object-info would
always result in additional overhead.
Thank you. No matter what mode it is (buffer or non-buffer), the
overhead of remote-object-info is always there. To my understanding,
there are two parts in the overhead:
1. Setting up a connection. This is happening in `connect_setup()` in
`fetch_object_info()` function.
2. Sending request buf. This includes initializing the packet reader
in `packet_reader_init()` and putting OIDs in the request buff in
`send_object_info_request()`. Both of them are called in the
`fetch_object_info()` function.
It would be more efficient to send multiple OIDs over one request
packet in one connection in the form of remote-object-info <remote>
<oid> <oid> ... <oid>
quoted
remote-object-info <remote> <oid> <oid> ... <oid>
rather than
remote-object-info <remote> <oid>
remote-object-info <remote> <oid>
...
remote-object-info <remote> <oid>
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
I think the sign-offs are supposed to go at the bottom.
Thank you. I am fixing it in v2.
[snip]
quoted
@@ -526,51 +533,118 @@ static void batch_one_object(const char *obj_name, (opt->follow_symlinks ? GET_OID_FOLLOW_SYMLINKS : 0); enum get_oid_result result;- result = get_oid_with_context(the_repository, obj_name,- flags, &data->oid, &ctx);- if (result != FOUND) {- switch (result) {- case MISSING_OBJECT:- printf("%s missing%c", obj_name, opt->output_delim);- break;- case SHORT_NAME_AMBIGUOUS:- printf("%s ambiguous%c", obj_name, opt->output_delim);- break;- case DANGLING_SYMLINK:- printf("dangling %"PRIuMAX"%c%s%c",- (uintmax_t)strlen(obj_name),- opt->output_delim, obj_name, opt->output_delim);- break;- case SYMLINK_LOOP:- printf("loop %"PRIuMAX"%c%s%c",- (uintmax_t)strlen(obj_name),- opt->output_delim, obj_name, opt->output_delim);- break;- case NOT_DIR:- printf("notdir %"PRIuMAX"%c%s%c",- (uintmax_t)strlen(obj_name),- opt->output_delim, obj_name, opt->output_delim);- break;- default:- BUG("unknown get_sha1_with_context result %d\n",- result);- break;+ if (!opt->use_remote_info) {
When using the remote-object-info command, the object in question is
supposed to be on the remote and may not exist locally. Therefore we
skip over `get_oid_with_context()`.
Thank you. Yes, that is the reason. I reworded your comment and added
it to the code in v2 to make it easier to follow.
quoted
+ result = get_oid_with_context(the_repository, obj_name,
+ flags, &data->oid, &ctx);
+ if (result != FOUND) {
+ switch (result) {
+ case MISSING_OBJECT:
+ printf("%s missing%c", obj_name, opt->output_delim);
+ break;
+ case SHORT_NAME_AMBIGUOUS:
+ printf("%s ambiguous%c", obj_name, opt->output_delim);
+ break;
+ case DANGLING_SYMLINK:
+ printf("dangling %"PRIuMAX"%c%s%c",
+ (uintmax_t)strlen(obj_name),
+ opt->output_delim, obj_name, opt->output_delim);
+ break;
+ case SYMLINK_LOOP:
+ printf("loop %"PRIuMAX"%c%s%c",
+ (uintmax_t)strlen(obj_name),
+ opt->output_delim, obj_name, opt->output_delim);
+ break;
+ case NOT_DIR:
+ printf("notdir %"PRIuMAX"%c%s%c",
+ (uintmax_t)strlen(obj_name),
+ opt->output_delim, obj_name, opt->output_delim);
+ break;
+ default:
+ BUG("unknown get_sha1_with_context result %d\n",
+ result);
+ break;
+ }
+ fflush(stdout);
+ return;
}
- fflush(stdout);
- return;
- }
- if (ctx.mode == 0) {
- printf("symlink %"PRIuMAX"%c%s%c",
- (uintmax_t)ctx.symlink_path.len,
- opt->output_delim, ctx.symlink_path.buf, opt->output_delim);
- fflush(stdout);
- return;
+ if (ctx.mode == 0) {
+ printf("symlink %"PRIuMAX"%c%s%c",
+ (uintmax_t)ctx.symlink_path.len,
+ opt->output_delim, ctx.symlink_path.buf, opt->output_delim);
+ fflush(stdout);
+ return;
+ }
}
batch_object_write(obj_name, scratch, opt, data, NULL, 0);
}
+static int get_remote_info(struct batch_options *opt, int argc, const char **argv)
+{
+ int retval = 0;
+ struct remote *remote = NULL;
+ struct object_id oid;
+ struct string_list object_info_options = STRING_LIST_INIT_NODUP;
+ static struct transport *gtransport;
+
+ /*
+ * Change the format to "%(objectname) %(objectsize)" when
+ * remote-object-info command is used. Once we start supporting objecttype
+ * the default format should change to DEFAULT_FORMAT
+ */
+ if (!opt->format) {
+ opt->format = "%(objectname) %(objectsize)";
+ }
We should omit the parenthesis for single line if statements.
Thank you. Fixed in V2.
quoted
+
+ remote = remote_get(argv[0]);
+ if (!remote)
+ die(_("must supply valid remote when using remote-object-info"));
+ oid_array_clear(&object_info_oids);
+ for (size_t i = 1; i < argc; i++) {
+ if (get_oid_hex(argv[i], &oid))
+ die(_("Not a valid object name %s"), argv[i]);
+ oid_array_append(&object_info_oids, &oid);
+ }
+
+ gtransport = transport_get(remote, NULL);
+ if (gtransport->smart_options) {
+ int include_size = 0;
+
+ CALLOC_ARRAY(remote_object_info, object_info_oids.nr);
+ gtransport->smart_options->object_info = 1;
+ gtransport->smart_options->object_info_oids = &object_info_oids;
+ /*
+ * 'size' is the only option currently supported.
+ * Other options that are passed in the format will exit with error.
+ */
+ if (strstr(opt->format, "%(objectsize)")) {
+ string_list_append(&object_info_options, "size");
+ include_size = 1;
+ }
+ if (strstr(opt->format, "%(objecttype)")) {
+ die(_("objecttype is currently not supported with remote-object-info"));
+ }
Another single line if statement above that should omit the parenthesis.
Thank you. Fixed in V2.
quoted
+ if (strstr(opt->format, "%(objectsize:disk)"))
+ die(_("objectsize:disk is currently not supported with remote-object-info"));
+ if (strstr(opt->format, "%(deltabase)"))
+ die(_("deltabase is currently not supported with remote-object-info"));
+ if (object_info_options.nr > 0) {
+ gtransport->smart_options->object_info_options = &object_info_options;
+ for (size_t i = 0; i < object_info_oids.nr; i++) {
+ if (include_size)
+ remote_object_info[i].sizep = xcalloc(1, sizeof(long));
+ }
+ gtransport->smart_options->object_info_data = &remote_object_info;
+ retval = transport_fetch_refs(gtransport, NULL);
+ }
+ } else {
+ retval = -1;
+ }
+
+ return retval;
+}
+
struct object_cb_data {
struct batch_options *opt;
struct expand_data *expand;
Since special handling is needed for the remote-object-info command, we
record the queued command names to check against later.
Yes. We need to compare the function name to do special handling
later. But I think we can have a better solution here instead of doing
a name comparison. Please see my reply below.
It seems a little confusing to me that `parse_remote_info()` accepts
both a `parse_cmd` and `queued_cmd`, but only expects to use one or the
other. It looks like this is done because `dispatch_calls()` already
accepts `queued_cmd`, but now needs to call `parse_remote_info()`.
Since it is only the underlying command function that is needed by
`parse_remote_info()`
Thank you. I agree. I did some refactoring. Please see me reply below.
@@ -671,8 +795,12 @@ static void dispatch_calls(struct batch_options *opt, if (!opt->buffer_output) die(_("flush is only for --buffer mode"));- for (int i = 0; i < nr; i++)- cmd[i].fn(opt, cmd[i].line, output, data);+ for (int i = 0; i < nr; i++) {+ if (!strcmp(cmd[i].name, "remote-object-info"))+ parse_remote_info(opt, cmd[i].line, output, data, NULL, &cmd[i]);
If we adapt `parse_remote_info()` to accept the command function we
could pass cmd->fn here instead.
Thank you. I think I can push it a bit further.
Under the hood, parse_remote_info will use parse_cmd_info to print the
retrieved information to the client. That is why it had this line
originally:
...
{ "remote-object-info", parse_cmd_info, 1 },
...
Inspired by your comment, I am thinking if I can adapt
parse_remote_info() 's signature to the same as parse_cmd_info(). It
would make the code cleaner. To be specific. I can
1. get rid of name cooperation in
...
if (!strcmp(cmd[i].name, "remote-object-info"))
parse_remote_info(opt, cmd[i].line, output, data, NULL, &cmd[I]);
else
cmd[i].fn(opt, cmd[i].line, output, data);
...
and I can just use `cmd[i].fn(opt, cmd[i].line, output, data)`
2. get rid of
...
if (p_cmd)
p_cmd->fn(opt, argv[i+1], output, data);
else
q_cmd->fn(opt, argv[i+1], output, data);
...
I will make this change in V2.
On Mon, Jul 8, 2024 at 9:51 PM Justin Tobler [off-list ref] wrote:
On 24/06/28 03:05PM, Eric Ju wrote:
quoted
From: Calvin Wan <redacted>
Since the `info` command in cat-file --batch-command prints object info
for a given object, it is natural to add another command in cat-file
--batch-command to print object info for a given object from a remote.
Add `remote-object-info` to cat-file --batch-command.
While `info` takes object ids one at a time, this creates overhead when
making requests to a server so `remote-object-info` instead can take
multiple object ids at once.
cat-file --batch-command is generally implemented in the following
manner:
- Receive and parse input from user
- Call respective function attached to command
- Set batch mode state, get object info, print object info
In --buffer mode, this changes to:
- Receive and parse input from user
- Store respective function attached to command in a queue
- After flush, loop through commands in queue
- Call respective function attached to command
- Set batch mode state, get object info, print object info
So the problem is that there is overhead associated with getting object
info from the remote. Therefore, remote-object-info also supports
batching objects together. This seems reasonable.
quoted
Notice how the getting and printing of object info is accomplished one
at a time. As described above, this creates a problem for making
requests to a server. Therefore, `remote-object-info` is implemented in
the following manner:
- Receive and parse input from user
If command is `remote-object-info`:
- Get object info from remote
- Loop through object info
- Call respective function attached to `info`
- Set batch mode state, use passed in object info, print object
info
Else:
- Call respective function attached to command
- Parse input, get object info, print object info
And finally for --buffer mode `remote-object-info`:
- Receive and parse input from user
- Store respective function attached to command in a queue
- After flush, loop through commands in queue:
If command is `remote-object-info`:
- Get object info from remote
- Loop through object info
- Call respective function attached to `info`
- Set batch mode state, use passed in object info, print
object info
Else:
- Call respective function attached to command
- Set batch mode state, get object info, print object info
To summarize, `remote-object-info` gets object info from the remote and
then generates multiple `info` commands with the object info passed in.
In order for remote-object-info to avoid remote communication overhead
in the non-buffer mode, the objects are passed in as such:
Even in non-buffer mode, having separate remote-object-info commands
would result in additional overhead correct? From my understanding each
command is executed sequently, so multiples of remote-object-info would
always result in additional overhead.
quoted
remote-object-info <remote> <oid> <oid> ... <oid>
rather than
remote-object-info <remote> <oid>
remote-object-info <remote> <oid>
...
remote-object-info <remote> <oid>
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
I think the sign-offs are supposed to go at the bottom.
[snip]
quoted
@@ -526,51 +533,118 @@ static void batch_one_object(const char *obj_name, (opt->follow_symlinks ? GET_OID_FOLLOW_SYMLINKS : 0); enum get_oid_result result;- result = get_oid_with_context(the_repository, obj_name,- flags, &data->oid, &ctx);- if (result != FOUND) {- switch (result) {- case MISSING_OBJECT:- printf("%s missing%c", obj_name, opt->output_delim);- break;- case SHORT_NAME_AMBIGUOUS:- printf("%s ambiguous%c", obj_name, opt->output_delim);- break;- case DANGLING_SYMLINK:- printf("dangling %"PRIuMAX"%c%s%c",- (uintmax_t)strlen(obj_name),- opt->output_delim, obj_name, opt->output_delim);- break;- case SYMLINK_LOOP:- printf("loop %"PRIuMAX"%c%s%c",- (uintmax_t)strlen(obj_name),- opt->output_delim, obj_name, opt->output_delim);- break;- case NOT_DIR:- printf("notdir %"PRIuMAX"%c%s%c",- (uintmax_t)strlen(obj_name),- opt->output_delim, obj_name, opt->output_delim);- break;- default:- BUG("unknown get_sha1_with_context result %d\n",- result);- break;+ if (!opt->use_remote_info) {
When using the remote-object-info command, the object in question is
supposed to be on the remote and may not exist locally. Therefore we
skip over `get_oid_with_context()`.
quoted
+ result = get_oid_with_context(the_repository, obj_name,
+ flags, &data->oid, &ctx);
+ if (result != FOUND) {
+ switch (result) {
+ case MISSING_OBJECT:
+ printf("%s missing%c", obj_name, opt->output_delim);
+ break;
+ case SHORT_NAME_AMBIGUOUS:
+ printf("%s ambiguous%c", obj_name, opt->output_delim);
+ break;
+ case DANGLING_SYMLINK:
+ printf("dangling %"PRIuMAX"%c%s%c",
+ (uintmax_t)strlen(obj_name),
+ opt->output_delim, obj_name, opt->output_delim);
+ break;
+ case SYMLINK_LOOP:
+ printf("loop %"PRIuMAX"%c%s%c",
+ (uintmax_t)strlen(obj_name),
+ opt->output_delim, obj_name, opt->output_delim);
+ break;
+ case NOT_DIR:
+ printf("notdir %"PRIuMAX"%c%s%c",
+ (uintmax_t)strlen(obj_name),
+ opt->output_delim, obj_name, opt->output_delim);
+ break;
+ default:
+ BUG("unknown get_sha1_with_context result %d\n",
+ result);
+ break;
+ }
+ fflush(stdout);
+ return;
}
- fflush(stdout);
- return;
- }
- if (ctx.mode == 0) {
- printf("symlink %"PRIuMAX"%c%s%c",
- (uintmax_t)ctx.symlink_path.len,
- opt->output_delim, ctx.symlink_path.buf, opt->output_delim);
- fflush(stdout);
- return;
+ if (ctx.mode == 0) {
+ printf("symlink %"PRIuMAX"%c%s%c",
+ (uintmax_t)ctx.symlink_path.len,
+ opt->output_delim, ctx.symlink_path.buf, opt->output_delim);
+ fflush(stdout);
+ return;
+ }
}
batch_object_write(obj_name, scratch, opt, data, NULL, 0);
}
+static int get_remote_info(struct batch_options *opt, int argc, const char **argv)
+{
+ int retval = 0;
+ struct remote *remote = NULL;
+ struct object_id oid;
+ struct string_list object_info_options = STRING_LIST_INIT_NODUP;
+ static struct transport *gtransport;
+
+ /*
+ * Change the format to "%(objectname) %(objectsize)" when
+ * remote-object-info command is used. Once we start supporting objecttype
+ * the default format should change to DEFAULT_FORMAT
+ */
+ if (!opt->format) {
+ opt->format = "%(objectname) %(objectsize)";
+ }
We should omit the parenthesis for single line if statements.
quoted
+
+ remote = remote_get(argv[0]);
+ if (!remote)
+ die(_("must supply valid remote when using remote-object-info"));
+ oid_array_clear(&object_info_oids);
+ for (size_t i = 1; i < argc; i++) {
+ if (get_oid_hex(argv[i], &oid))
+ die(_("Not a valid object name %s"), argv[i]);
+ oid_array_append(&object_info_oids, &oid);
+ }
+
+ gtransport = transport_get(remote, NULL);
+ if (gtransport->smart_options) {
+ int include_size = 0;
+
+ CALLOC_ARRAY(remote_object_info, object_info_oids.nr);
+ gtransport->smart_options->object_info = 1;
+ gtransport->smart_options->object_info_oids = &object_info_oids;
+ /*
+ * 'size' is the only option currently supported.
+ * Other options that are passed in the format will exit with error.
+ */
+ if (strstr(opt->format, "%(objectsize)")) {
+ string_list_append(&object_info_options, "size");
+ include_size = 1;
+ }
+ if (strstr(opt->format, "%(objecttype)")) {
+ die(_("objecttype is currently not supported with remote-object-info"));
+ }
Another single line if statement above that should omit the parenthesis.
quoted
+ if (strstr(opt->format, "%(objectsize:disk)"))
+ die(_("objectsize:disk is currently not supported with remote-object-info"));
+ if (strstr(opt->format, "%(deltabase)"))
+ die(_("deltabase is currently not supported with remote-object-info"));
+ if (object_info_options.nr > 0) {
+ gtransport->smart_options->object_info_options = &object_info_options;
+ for (size_t i = 0; i < object_info_oids.nr; i++) {
+ if (include_size)
+ remote_object_info[i].sizep = xcalloc(1, sizeof(long));
+ }
+ gtransport->smart_options->object_info_data = &remote_object_info;
+ retval = transport_fetch_refs(gtransport, NULL);
+ }
+ } else {
+ retval = -1;
+ }
+
+ return retval;
+}
+
struct object_cb_data {
struct batch_options *opt;
struct expand_data *expand;
It seems a little confusing to me that `parse_remote_info()` accepts
both a `parse_cmd` and `queued_cmd`, but only expects to use one or the
other. It looks like this is done because `dispatch_calls()` already
accepts `queued_cmd`, but now needs to call `parse_remote_info()`.
Since it is only the underlying command function that is needed by
`parse_remote_info()`
@@ -671,8 +795,12 @@ static void dispatch_calls(struct batch_options *opt, if (!opt->buffer_output) die(_("flush is only for --buffer mode"));- for (int i = 0; i < nr; i++)- cmd[i].fn(opt, cmd[i].line, output, data);+ for (int i = 0; i < nr; i++) {+ if (!strcmp(cmd[i].name, "remote-object-info"))+ parse_remote_info(opt, cmd[i].line, output, data, NULL, &cmd[i]);
If we adapt `parse_remote_info()` to accept the command function we
could pass cmd->fn here instead.
@@ -436,11 +504,27 @@ static int fetch_refs_via_pack(struct transport *transport,args.server_options=transport->server_options;args.negotiation_tips=data->options.negotiation_tips;args.reject_shallow_remote=transport->smart_options->reject_shallow;--if(!data->finished_handshake){-inti;+args.object_info=transport->smart_options->object_info;++if(transport->smart_options&&transport->smart_options->object_info){+structref*ref=object_info_refs;++if(!fetch_object_info(transport,data->options.object_info_data))+gotocleanup;+args.object_info_data=data->options.object_info_data;+args.quiet=1;+args.no_progress=1;+for(size_ti=0;i<transport->smart_options->object_info_oids->nr;i++){+structref*temp_ref=xcalloc(1,sizeof(structref));+temp_ref->old_oid=*(transport->smart_options->object_info_oids->oid+i);
Any reason why you're not using the subscript operator (square brackets)
like this:
+ temp_ref->old_oid = transport->smart_options->object_info_oids->oid[I];
I find it a bit weird you're allocating object_info_refs, only to use it
to point to the next. Can I suggest a little refactor:
Thank you. I have to agree that the old implementation of iterating on
the object_info_refs linked list is a bit obscure.
Your suggestion is easier to follow. I am replacing the old logic in V2.
@@ -479,7 +479,7 @@ static int fetch_refs_via_pack(struct transport *transport,structref*refs=NULL;structfetch_pack_argsargs;structref*refs_tmp=NULL;-structref*object_info_refs=xcalloc(1,sizeof(structref));+structref*object_info_refs=NULL;memset(&args,0,sizeof(args));args.uploadpack=data->options.uploadpack;
@@ -509,7 +509,7 @@ static int fetch_refs_via_pack(struct transport *transport,args.object_info=transport->smart_options->object_info;if(transport->smart_options&&transport->smart_options->object_info){-structref*ref=object_info_refs;+structref*ref=object_info_refs=xcalloc(1,sizeof(structref));if(!fetch_object_info(transport,data->options.object_info_data))gotocleanup;
@@ -517,13 +517,12 @@ static int fetch_refs_via_pack(struct transport *transport,args.quiet=1;args.no_progress=1;for(size_ti=0;i<transport->smart_options->object_info_oids->nr;i++){-structref*temp_ref=xcalloc(1,sizeof(structref));-temp_ref->old_oid=*(transport->smart_options->object_info_oids->oid+i);-temp_ref->exact_oid=1;-ref->next=temp_ref;+ref->old_oid=transport->smart_options->object_info_oids->oid[i];+ref->exact_oid=1;+ref->next=xcalloc(1,sizeof(structref));ref=ref->next;}-transport->remote_refs=object_info_refs->next;+transport->remote_refs=object_info_refs;}elseif(!data->finished_handshake){intmust_list_refs=0;for(inti=0;i<nr_heads;i++){
@@ -565,7 +564,7 @@ static int fetch_refs_via_pack(struct transport *transport,data->finished_handshake=0;if(args.object_info){-structref*ref_cpy_reader=object_info_refs->next;+structref*ref_cpy_reader=object_info_refs;for(inti=0;ref_cpy_reader;i++){oid_object_info_extended(the_repository,&ref_cpy_reader->old_oid,&(*args.object_info_data)[i],OBJECT_INFO_LOOKUP_REPLACE);ref_cpy_reader=ref_cpy_reader->next;----8<-----8<----
To be honest, I'm not sure it works, because fetch_object_info() always
seem to return a non-zero value. I'm not sure this is due to missing
code coverage, or a bug. I guess it's worth looking into.
Thank you. I tested your suggestion and it is working. I can confirm
it when I did the following with my debugger
1. pause on a test case of t/t1017-cat-file-remote-object-info.sh
2. git cat-file "--batch-command=%(objectname) %(objectsize)"
3. remote-object-info http://127.0.0.1:11017/smart/http_parent
5e1c309dae7f45e0f39b1bf3ac3cd9db12e7d689
I set breakpoints all along and see that fetch_object_info() returned zero
Would you mind sharing your test steps with me? I would love to dig deeper.
...
+static int get_remote_info(struct batch_options *opt, int argc, const char **argv)
+{
+ int retval = 0;
+ struct remote *remote = NULL;
+ struct object_id oid;
+ struct string_list object_info_options = STRING_LIST_INIT_NODUP;
+ static struct transport *gtransport;
+
+ /*
+ * Change the format to "%(objectname) %(objectsize)" when
+ * remote-object-info command is used. Once we start supporting objecttype
+ * the default format should change to DEFAULT_FORMAT
+ */
I believe this comment has become outdated, or got moved around
incorrectly.
Thank you Toon. Sorry, I didn't get it. This comment is not outdated.
It is before this code
if (!opt->format) {
opt->format = "%(objectname) %(objectsize)";
}
And this is related to my 2nd open question in the cover letter
2. Right now, only the size is supported. If the batch command format
contains objectsize:disk or deltabase, it will die. The question
is about objecttype. In the current implementation, it will die too.
But dying on objecttype breaks the default format. We have changed the
default format to %(objectname) %(objectsize) when
remote-object-info is used.
Any suggestions on this approach?
...
+stop_git_daemon
+
+# Test --batch-command remote-object-info with 'http://' transport
+
+. "$TEST_DIRECTORY"/lib-httpd.sh
+start_httpd
start_httpd skips the remainder of the tests if it fails to start the
httpd server. That's why I see various other tests which have this at
the end:
# DO NOT add non-httpd-specific tests here, because the last part of this
# test script is only executed when httpd is available and enabled.
So I would suggest to add this comment as well, and move the file://
tests above start_httpd.
From: Peijian Ju <hidden> Date: 2024-07-15 16:40:40
On Wed, Jul 10, 2024 at 5:39 AM Karthik Nayak [off-list ref] wrote:
Peijian Ju [off-list ref] writes:
[snip]
quoted
quoted
Right, this commit in itself looks good. But I was curious why we need
this, so I did a sneak peak into the following commits.
To summarize, we want to call:
`write_command_and_capabilities(..., "object-info");`
in the upcoming patches to get the object-info details from the server.
But isn't this function too specific to the "fetch" command to be
generalized to be for "object-info" too?
Wouldn't it make sense to add a custom function for 'object-info' in
'connect.c'? Like how we currently have `get_remote_bundle_uri()` for
'bundle-uri' and `get_remote_refs` for 'ls-refs'?
Thank you. I am reading through the old comments left by Taylor
at https://lore.kernel.org/git/YkOPyc9tUfe2Tozx@nand.local/
" Makes obvious sense, and this was something that jumped out to me when I
looked at the first and second versions of this patch. I'm glad that
this is getting factored out."
It seems refactoring this into a more general function is on purpose.
It is encouraged to use this general function to request capability
rather than adding a custom function.
Taylor’s comment was 2 years ago, but I think refactoring this into a
more general function to
enforce DRY still makes sense.
It would make sense then to move the existing users to also use
`write_command_and_capabilities` eventually. I guess this could be done
in a follow up series.
Then I would say `write_command_and_capabilities()` should be moved to
`transport.c`, no?
Thank you. I am not sure about this. Currently, the file dependency is
like this:
`transport.c` -> `fetch-pack.c` -> `connect.c` where "->" means "depends on".
Moving `write_command_and_capabilities()` to `transport.c` would make
circle dependency.
If we want `write_command_and_capabilities()` to be a more general
utility function,
it seems make more sense to move it to `connect.c`. I saw a bunch of
these general utility functions
in `connect.c` such as `send_capabilities()`. Some custom functions
such as `get_remote_bundle_uri()` and `get_remote_refs`also lives in
it.
Please let me know what you think. Thanks.
From: Peijian Ju <hidden> Date: 2024-07-16 02:39:22
On Wed, Jul 10, 2024 at 6:13 AM Karthik Nayak [off-list ref] wrote:
Eric Ju [off-list ref] writes:
quoted
From: Calvin Wan <redacted>
Sometimes it is useful to get information about an object without having
to download it completely. The server logic has already been implemented
as “a2ba162cda (object-info: support for retrieving object info,
Nit: s/as/in
Thank you. Fixed in V2.
quoted
2021-04-20)”.
Add client functions to communicate with the server.
The client currently supports requesting a list of object ids with
features 'size' and 'type' from a v2 server. If a server does not
But do we support type? I thought we only added support for 'size'.
Thank you. Yes, only size is supported, I will revise it.
quoted
advertise either of the requested features, then the client falls back
to making the request through 'fetch'.
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
---
fetch-pack.c | 24 +++++++++++
fetch-pack.h | 10 +++++
transport-helper.c | 8 +++-
transport.c | 102 ++++++++++++++++++++++++++++++++++++++++++---
transport.h | 11 +++++
5 files changed, 148 insertions(+), 7 deletions(-)
@@ -697,13 +697,17 @@ static int fetch_refs(struct transport *transport,/**Ifwereachhere,thentheserver,theclient,and/orthetransport-*helperdoesnotsupportprotocolv2.--negotiate-onlyrequires-*protocolv2.+*helperdoesnotsupportprotocolv2.--negotiate-onlyandcat-fileremote-object-info+*requireprotocolv2.*/if(data->transport_options.acked_commits){warning(_("--negotiate-only requires protocol v2"));return-1;}+if(transport->smart_options->object_info){+// fail the command explicitly to avoid further commands input+die(_("remote-object-info requires protocol v2"));+}if(!data->get_refs_list_called)get_refs_list_using_list(transport,0);
Perhaps `*object_info_data[i]->sizep =
strtoul(object_info_values.items[1 + size_index].string, NULL, 10);`?
So, this is allocated in 'cat-file' and set here? Wouldn't it be nicer
to also do the alloc here?
> > Perhaps `*object_info_data[i]->sizep =
> > strtoul(object_info_values.items[1 + size_index].string, NULL, 10);`?
Thank you.
Seems that `*(*object_info_data)[i].sizep` and
`object_info_data[i]->sizep` are not the same.
Given object_info_data is a pointer to a pointer to struct
object_info, what `*(*object_info_data)[i].sizep` does is
1. *object_info_data dereferences object_info_data, yielding a pointer
to the first element of the array of struct object_info.
2. (*object_info_data)[i] accesses the i-th element in the array of
struct object_info that *object_info_data points to.
4, (*object_info_data)[i].sizep accesses the sizep member of the i-th
struct object_info.
5. *(*object_info_data)[i].sizep dereferences the sizep pointer,
yielding the value it points to.
So we are interested in the array of struct object_info with its first
element at *object_info_data. A more intuitive way of thinking it is
that if we think object_info_data as a 2-D array,
*(*object_info_data)[i] is accessing the object_info_data[0][i].
For `*object_info_data[i]->sizep`:
1. object_info_data[i] accesses the i-th element in the array of
pointers to struct object_info.
2. object_info_data[i]->sizep accesses the sizep member of the i-th
struct object_info that object_info_data[i] points to.
3. *object_info_data[i]->sizep dereferences the sizep pointer,
yielding the value it points to.
*object_info_data[i]->sizep will treat object_info_data as an array of
pointers. In the mental model of 2D array, *object_info_data[i] is
like object_info_data[i][0]
Nevertheless, I do think using a pointer to a pointer is tricky and
error-prone. In V2, I am refactoring the code to use just a pointer
instead of a pointer to a pointer. For example, in transport.h
git_transport_options {
...
struct object_info *object_info_data;
...
}
> > So, this is allocated in 'cat-file' and set here? Wouldn't it be nicer
> > to also do the alloc here?
Thank you.
Yes, this makes sense, V2 is refactoring the allocation into
`fetch_object_info()` in transport.c
From: Peijian Ju <hidden> Date: 2024-07-16 02:59:44
On Wed, Jul 10, 2024 at 6:16 AM Karthik Nayak [off-list ref] wrote:
Eric Ju [off-list ref] writes:
quoted
Some code declares variable i and only uses it
in a for loop, not in any other logic outside the loop.
Change the declaration of i to be inside the for loop for readability.
If we're doing this anyways, we could replace the 'int' with 'size_t'
too.
From: Peijian Ju <hidden> Date: 2024-07-17 02:38:37
On Wed, Jul 10, 2024 at 8:08 AM Karthik Nayak [off-list ref] wrote:
Eric Ju [off-list ref] writes:
quoted
From: Calvin Wan <redacted>
Since the `info` command in cat-file --batch-command prints object info
for a given object, it is natural to add another command in cat-file
--batch-command to print object info for a given object from a remote.
Add `remote-object-info` to cat-file --batch-command.
While `info` takes object ids one at a time, this creates overhead when
making requests to a server so `remote-object-info` instead can take
multiple object ids at once.
cat-file --batch-command is generally implemented in the following
manner:
- Receive and parse input from user
So this refers input delimited by newline or '\0'.
Thank you. The input should take both newline and '\0' into
consideration. We are missing some test coverage on '\0' delimited
input though. I am adding them in V2
quoted
- Call respective function attached to command
- Set batch mode state, get object info, print object info
Doesn't the batch mode get set before the input parsing begins?
Thank you. Yes, I am also unsure what Calvin "Set batch mode state"
means here. This batch mode is determined when the cat-file command is
called. But I do see `opt->batch_mode = BATCH_MODE_INFO;` in
`parse_cmd_info and()` and
`opt->batch_mode = BATCH_MODE_CONTENTS;` in `parse_cmd_contents()` I
guess that is what Calvin refers to
Anyway, I am removing "Set batch mode state" in V2 to avoid confusion,
It seems too detailed.
quoted
In --buffer mode, this changes to:
- Receive and parse input from user
- Store respective function attached to command in a queue
- After flush, loop through commands in queue
- Call respective function attached to command
- Set batch mode state, get object info, print object info
Notice how the getting and printing of object info is accomplished one
at a time. As described above, this creates a problem for making
requests to a server. Therefore, `remote-object-info` is implemented in
the following manner:
- Receive and parse input from user
If command is `remote-object-info`:
- Get object info from remote
- Loop through object info
- Call respective function attached to `info`
- Set batch mode state, use passed in object info, print object
info
Else:
- Call respective function attached to command
- Parse input, get object info, print object info
So this is because we want 'remote-object-info' to also use
'parse_cmd_info' similar to 'info'. But I'm not understanding why,
especially since 'parse_cmd_info' calls 'batch_one_object', and we skip
most of that code for 'remote-object-info'.
Wouldn't it be cleaner to just define our own 'batch_remote_object' and
create 'parse_cmd_remote_info' ?
Thank you. That makes sense. Actually, I am pushing it a bit further in V2:
1. The interface of parse_remote_info() is changed to parse_cmd_fn_t,
and its name is changed to `parse_cmd_remote_info()`.
2. In `static const struct parse_cmd{...} commands[]`, the
"remote-object-info" is attached with parse_cmd_remote_info() directly
3. In side parse_cmd_remote_info, we don't need
`batch_remote_object()`, all we need is just `batch_object_write()` to
print the object info out. That will simply the code a lot.
We don't need to call parse_cmd_info; also, we can get rid of the name
compare logic, i.e. `if (!strcmp(cmd[i].name, "remote-object-info"))
...`
quoted
And finally for --buffer mode `remote-object-info`:
- Receive and parse input from user
- Store respective function attached to command in a queue
- After flush, loop through commands in queue:
If command is `remote-object-info`:
- Get object info from remote
- Loop through object info
- Call respective function attached to `info`
- Set batch mode state, use passed in object info, print
object info
Else:
- Call respective function attached to command
- Set batch mode state, get object info, print object info
To summarize, `remote-object-info` gets object info from the remote and
then generates multiple `info` commands with the object info passed in.
In order for remote-object-info to avoid remote communication overhead
in the non-buffer mode, the objects are passed in as such:
remote-object-info <remote> <oid> <oid> ... <oid>
rather than
remote-object-info <remote> <oid>
remote-object-info <remote> <oid>
...
remote-object-info <remote> <oid>
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
---
Documentation/git-cat-file.txt | 22 +-
builtin/cat-file.c | 231 ++++++++++----
object-file.c | 11 +
object-store-ll.h | 3 +
t/t1017-cat-file-remote-object-info.sh | 412 +++++++++++++++++++++++++
5 files changed, 620 insertions(+), 59 deletions(-)
create mode 100755 t/t1017-cat-file-remote-object-info.sh
@@ -149,6 +149,12 @@ info <object>:: Print object info for object reference `<object>`. This corresponds to the output of `--batch-check`.+remote-object-info <remote> <object>...::+ Print object info for object references `<object>` at specified <remote> without+ downloading object from remote.+ Error when no object references is provided.+ This command may be combined with `--buffer`.+ flush:: Used with `--buffer` to execute all preceding commands that were issued since the beginning or since the last flush was issued. When `--buffer`
@@ -290,7 +296,8 @@ newline. The available atoms are: The full hex representation of the object name. `objecttype`::- The type of the object (the same as `cat-file -t` reports).+ The type of the object (the same as `cat-file -t` reports). See+ `CAVEATS` below. Not supported by `remote-object-info`. `objectsize`:: The size, in bytes, of the object (the same as `cat-file -s`
@@ -298,13 +305,14 @@ newline. The available atoms are: `objectsize:disk`:: The size, in bytes, that the object takes up on disk. See the- note about on-disk sizes in the `CAVEATS` section below.+ note about on-disk sizes in the `CAVEATS` section below. Not+ supported by `remote-object-info`. `deltabase`:: If the object is stored as a delta on-disk, this expands to the full hex representation of the delta base object name. Otherwise, expands to the null OID (all zeroes). See `CAVEATS`- below.+ below. Not supported by `remote-object-info`. `rest`:: If this atom is used in the output string, input lines are split
@@ -314,7 +322,9 @@ newline. The available atoms are: line) are output in place of the `%(rest)` atom. If no format is specified, the default format is `%(objectname)-%(objecttype) %(objectsize)`.+%(objecttype) %(objectsize)`, except remote-object-info command who uses+`%(objectname) %(objectsize)` for now because "%(objecttype)" is not supported yet.+When "%(objecttype)" is supported, default format should be unified. If `--batch` is specified, or if `--batch-command` is used with the `contents` command, the object information is followed by the object contents (consisting
@@ -396,6 +406,10 @@ scripting purposes. CAVEATS -------+Note that since objecttype, objectsize:disk and deltabase are currently not supported by the+remote-object-info, git will error and exit when they are in the format string.++ Note that the sizes of objects on disk are reported accurately, but care should be taken in drawing conclusions about which refs or objects are responsible for disk usage. The size of a packed non-delta object may be
Shouldn't we call `transport_disconnect(transport);`?
Thank you. transport_disconnect(transport) is added the end of
get_remote_info() in V2.
quoted
+ /*
+ * Change the format to "%(objectname) %(objectsize)" when
+ * remote-object-info command is used. Once we start supporting objecttype
+ * the default format should change to DEFAULT_FORMAT
+ */
+ if (!opt->format) {
+ opt->format = "%(objectname) %(objectsize)";
+ }
+
+ remote = remote_get(argv[0]);
+ if (!remote)
+ die(_("must supply valid remote when using remote-object-info"));
+ oid_array_clear(&object_info_oids);
+ for (size_t i = 1; i < argc; i++) {
+ if (get_oid_hex(argv[i], &oid))
+ die(_("Not a valid object name %s"), argv[i]);
+ oid_array_append(&object_info_oids, &oid);
+ }
+
+ gtransport = transport_get(remote, NULL);
+ if (gtransport->smart_options) {
+ int include_size = 0;
+
+ CALLOC_ARRAY(remote_object_info, object_info_oids.nr);
+ gtransport->smart_options->object_info = 1;
+ gtransport->smart_options->object_info_oids = &object_info_oids;
+ /*
+ * 'size' is the only option currently supported.
+ * Other options that are passed in the format will exit with error.
+ */
+ if (strstr(opt->format, "%(objectsize)")) {
+ string_list_append(&object_info_options, "size");
+ include_size = 1;
+ }
+ if (strstr(opt->format, "%(objecttype)")) {
+ die(_("objecttype is currently not supported with remote-object-info"));
+ }
+ if (strstr(opt->format, "%(objectsize:disk)"))
+ die(_("objectsize:disk is currently not supported with remote-object-info"));
+ if (strstr(opt->format, "%(deltabase)"))
+ die(_("deltabase is currently not supported with remote-object-info"));
This whole block could be replaced by an else..
if (strstr(opt->format, "%(objectsize)")) {
string_list_append(&object_info_options, "size");
include_size = 1;
} else {
die(_("%s is currently not supported with remote-object-info", opt->format));
}
Thank you. That is tricky. Originally I took them from
t/t1006-cat-file.sh. I did some research though.
13 = <file mode> + <a_space> + <file name> + <a_null>, where
file mode is 100644, which is 6 characters;
file name is hello, which is 5 characters
a space is 1 character and a null is 1 character
For commit message, here is the raw content
tree 6241ab2a5314798183b5c4ee8a7b0ccd12c651e6
author A U Thor [off-list ref] 1112354055 +0200
committer C O Mitter [off-list ref] 1112354055 +0200
Initial commit
137 = <tree header> + <a_space> + <a newline> +
<Author line> + <a newline> +
<Committer line> + <a newline> +
<a newline> +
<commit message length>
An easier way is this by `git cat-file commit <commit hash> | wc -c`,
which gets 177, then it should be minus 40 hex away, and result in 137
I put them in the comments to avoid confusion.
quoted
+
+tag_header_without_oid="type blob
+tag hellotag
+tagger $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL>"
+tag_header_without_timestamp="object $hello_oid
+$tag_header_without_oid"
+tag_description="This is a tag"
+tag_content="$tag_header_without_timestamp 0 +0000
+
+$tag_description"
+
+tag_oid=$(echo_without_newline "$tag_content" | git hash-object -t tag --stdin -w)
+tag_size=$(strlen "$tag_content")
+
+# This section tests --batch-command with remote-object-info command
+# Since "%(objecttype)" is currently not supported by the command remote-object-info ,
+# the filters are set to "%(objectname) %(objectsize)".
+# Tests with the default filter are used to test the fallback to 'fetch' command
+
+
+# Test --batch-command remote-object-info with 'git://' transport
+
+. "$TEST_DIRECTORY"/lib-git-daemon.sh
+start_git_daemon --export-all --enable=receive-pack
+daemon_parent=$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent
+
+test_expect_success 'create repo to be served by git-daemon' '
+ git init "$daemon_parent" &&
+
+ echo_without_newline "$hello_content" > $daemon_parent/hello &&
+ git -C "$daemon_parent" update-index --add hello &&
+ git -C "$daemon_parent" config transfer.advertiseobjectinfo true
+'
+
+set_transport_variables () {
+ hello_sha1=$(echo_without_newline "$hello_content" | git hash-object --stdin)
+ tree_sha1=$(git -C "$1" write-tree)
+ commit_sha1=$(echo_without_newline "$commit_message" | git -C "$1" commit-tree $tree_sha1)
+ tag_sha1=$(echo_without_newline "$tag_content" | git -C "$1" hash-object -t tag --stdin -w)
+ tag_size=$(strlen "$tag_content")
+}
+
+
Some more tests I'd like to see
- Testing against the '-Z' option.
- Testing the fallback to fetch whole object when the server doesn't
support 'remote-object-info'.
Thank you. More tests are added in V2 to cover those scenarios.
This is a continuation of Calvin Wan's (calvinwan@google.com)
patch series [PATCH v5 0/6] cat-file: add --batch-command remote-object-info command at [1].
Sometimes it is useful to get information about an object without having to download
it completely. The server logic for retrieving size has already been implemented and merged in
"a2ba162cda (object-info: support for retrieving object info, 2021-04-20)"[2].
This patch series implement the client option for it.
This patch series add the `remote-object-info` command to `cat-file --batch-command`.
This command allows the client to make an object-info command request to a server
that supports protocol v2. If the server is v2, but does not have
object-info capability, the entire object is fetched and the
relevant object info is returned.
A few questions open for discussions please:
1. In the current implementation, if a user puts `remote-object-info` in protocol v1,
`cat-file --batch-command` will die. Which way do we prefer? "error and exit (i.e. die)"
or "warn and wait for new command".
2. Right now, only the size is supported. If the batch command format
contains objectsize:disk or deltabase, it will die. The question
is about objecttype. In the current implementation, it will die too.
But dying on objecttype breaks the default format. We have changed the
default format to %(objectname) %(objectsize) when remote-object-info is used.
Any suggestions on this approach?
[1] https://lore.kernel.org/git/20220728230210.2952731-1-calvinwan@google.com/#t
[2] https://git.kernel.org/pub/scm/git/git.git/commit/?id=a2ba162cda2acc171c3e36acbbc854792b093cb7
V1 of the patch series can be found here:
https://lore.kernel.org/git/20240628190503.67389-1-eric.peijian@gmail.com/
Changes since V1
================
- The function parse_remote_info() has been renamed to
parse_cmd_remote_object_info() and its signature has been modified to comply
with parse_cmd_fn_t. This function now serves as the mapped function for the
remote-object-info command.
This change simplifies the code by avoiding command name comparisons and
reusing logic that fits parse_cmd_fn_t.
- Added more tests to cover fallbacks. When the server does not support the
object-info capability, remote-object-info will fetch the objects locally and
print out the information.
- Fixed a logic issue that could lead to a potential heap-buffer-overflow error.
The alloc_ref function is now used to initialize a ref struct instead of xcalloc.
- Refactored some logic for improved readability, such as how to initialize the
transport->remote_refs linked list.
Thank you.
Eric Ju
Calvin Wan (5):
fetch-pack: refactor packet writing
fetch-pack: move fetch initialization
serve: advertise object-info feature
transport: add client support for object-info
cat-file: add remote-object-info to batch-command
Eric Ju (1):
cat-file: add declaration of variable i inside its for loop
Documentation/git-cat-file.txt | 23 +-
builtin/cat-file.c | 127 ++++-
fetch-pack.c | 48 +-
fetch-pack.h | 10 +
object-file.c | 11 +
object-store-ll.h | 3 +
serve.c | 4 +-
t/t1017-cat-file-remote-object-info.sh | 748 +++++++++++++++++++++++++
transport-helper.c | 8 +-
transport.c | 118 +++-
transport.h | 11 +
11 files changed, 1075 insertions(+), 36 deletions(-)
create mode 100755 t/t1017-cat-file-remote-object-info.sh
Range-diff against v1:
1: fdd44b16b5 ! 1: 97871cb75e fetch-pack: refactor packet writing
@@ Metadata
## Commit message ##
fetch-pack: refactor packet writing
- A subsequent patch need to write capabilities for another command.
- Refactor write_fetch_command_and_capabilities() to be used by both
- fetch and future command.
+ A subsequent patch needs to write capabilities for another command.
+ Refactor write_fetch_command_and_capabilities() to be a more general
+ purpose function write_command_and_capabilities(), so that it can be
+ used by both fetch and future command.
+
+ Here "command" means the "operations" supported by Git’s wire protocol
+ https://git-scm.com/docs/protocol-v2. An example would be a
+ git's subcommand, such as git-fetch(1); or an operation supported by
+ the server side such as "object-info" implemented in "a2ba162cda
+ (object-info: support for retrieving object info, 2021-04-20)".
- Signed-off-by: Calvin Wan [off-list ref]
- Signed-off-by: Eric Ju [off-list ref]
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
+ Signed-off-by: Calvin Wan [off-list ref]
+ Signed-off-by: Eric Ju [off-list ref]
## fetch-pack.c ##
@@ fetch-pack.c: static int add_haves(struct fetch_negotiator *negotiator,
2: 890219ce6a ! 2: 301047c574 fetch-pack: move fetch initialization
@@ Commit message
from the beginning of the first state to just before the execution of
the state machine.
- Signed-off-by: Calvin Wan [off-list ref]
- Signed-off-by: Eric Ju [off-list ref]
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
+ Signed-off-by: Calvin Wan [off-list ref]
+ Signed-off-by: Eric Ju [off-list ref]
## fetch-pack.c ##
@@ fetch-pack.c: static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
3: 6844095b26 ! 3: 5d83c4f5b2 serve: advertise object-info feature
@@ Commit message
client to decide whether to query the server for object-info or fetch
as a fallback.
- Signed-off-by: Calvin Wan [off-list ref]
- Signed-off-by: Eric Ju [off-list ref]
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
+ Signed-off-by: Calvin Wan [off-list ref]
+ Signed-off-by: Eric Ju [off-list ref]
## serve.c ##
@@ serve.c: static void session_id_receive(struct repository *r UNUSED,
4: c940cb1657 ! 4: a7210b7169 transport: add client support for object-info
@@ Commit message
Sometimes it is useful to get information about an object without having
to download it completely. The server logic has already been implemented
- as “a2ba162cda (object-info: support for retrieving object info,
+ in “a2ba162cda (object-info: support for retrieving object info,
2021-04-20)”.
Add client functions to communicate with the server.
The client currently supports requesting a list of object ids with
- features 'size' and 'type' from a v2 server. If a server does not
- advertise either of the requested features, then the client falls back
+ feature 'size' from a v2 server. If a server does not
+ advertise the feature, then the client falls back
to making the request through 'fetch'.
- Signed-off-by: Calvin Wan [off-list ref]
- Signed-off-by: Eric Ju [off-list ref]
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
+ Signed-off-by: Calvin Wan [off-list ref]
+ Signed-off-by: Eric Ju [off-list ref]
## fetch-pack.c ##
@@ fetch-pack.c: static void write_command_and_capabilities(struct strbuf *req_buf,
@@ fetch-pack.h: struct fetch_pack_args {
const struct string_list *deepen_not;
struct list_objects_filter_options filter_options;
const struct string_list *server_options;
-+ struct object_info **object_info_data;
++ struct object_info *object_info_data;
/*
* If not NULL, during packfile negotiation, fetch-pack will send "have"
@@ transport.c: static struct ref *handshake(struct transport *transport, int for_p
return refs;
}
-+static int fetch_object_info(struct transport *transport, struct object_info **object_info_data)
++static int fetch_object_info(struct transport *transport, struct object_info *object_info_data)
+{
+ int size_index = -1;
+ struct git_transport_data *data = transport->data;
-+ struct object_info_args args;
++ struct object_info_args args = { 0 };
+ struct packet_reader reader;
+
-+ memset(&args, 0, sizeof(args));
+ args.server_options = transport->server_options;
+ args.object_info_options = transport->smart_options->object_info_options;
+ args.oids = transport->smart_options->object_info_oids;
@@ transport.c: static struct ref *handshake(struct transport *transport, int for_p
+ return -1;
+ }
+ if (unsorted_string_list_has_string(args.object_info_options, reader.line)) {
-+ if (!strcmp(reader.line, "size"))
++ if (!strcmp(reader.line, "size")) {
+ size_index = i;
++ for (size_t j = 0; j < args.oids->nr; j++) {
++ object_info_data[j].sizep = xcalloc(1, sizeof(long));
++ }
++ }
+ continue;
+ }
+ return -1;
@@ transport.c: static struct ref *handshake(struct transport *transport, int for_p
+ if (!strcmp(object_info_values.items[1 + size_index].string, ""))
+ die("object-info: not our ref %s",
+ object_info_values.items[0].string);
-+ *(*object_info_data)[i].sizep = strtoul(object_info_values.items[1 + size_index].string, NULL, 10);
++
++ *object_info_data[i].sizep = strtoul(object_info_values.items[1 + size_index].string, NULL, 10);
++
+ }
++
++ string_list_clear(&object_info_values, 0);
+ }
+ check_stateless_delimiter(transport->stateless_rpc, &reader, "stateless delimiter expected");
+
@@ transport.c: static int fetch_refs_via_pack(struct transport *transport,
struct ref *refs = NULL;
struct fetch_pack_args args;
struct ref *refs_tmp = NULL;
-+ struct ref *object_info_refs = xcalloc(1, sizeof (struct ref));
++ struct ref *object_info_refs = NULL;
memset(&args, 0, sizeof(args));
args.uploadpack = data->options.uploadpack;
@@ transport.c: static int fetch_refs_via_pack(struct transport *transport,
args.server_options = transport->server_options;
args.negotiation_tips = data->options.negotiation_tips;
args.reject_shallow_remote = transport->smart_options->reject_shallow;
--
-- if (!data->finished_handshake) {
-- int i;
+ args.object_info = transport->smart_options->object_info;
+
-+ if (transport->smart_options && transport->smart_options->object_info) {
-+ struct ref *ref = object_info_refs;
++ if (transport->smart_options
++ && transport->smart_options->object_info
++ && transport->smart_options->object_info_oids->nr > 0) {
++ struct ref *ref_itr = object_info_refs = alloc_ref("");
+
+ if (!fetch_object_info(transport, data->options.object_info_data))
+ goto cleanup;
++
+ args.object_info_data = data->options.object_info_data;
+ args.quiet = 1;
+ args.no_progress = 1;
+ for (size_t i = 0; i < transport->smart_options->object_info_oids->nr; i++) {
-+ struct ref *temp_ref = xcalloc(1, sizeof (struct ref));
-+ temp_ref->old_oid = *(transport->smart_options->object_info_oids->oid + i);
-+ temp_ref->exact_oid = 1;
-+ ref->next = temp_ref;
-+ ref = ref->next;
++ ref_itr->old_oid = transport->smart_options->object_info_oids->oid[i];
++ ref_itr->exact_oid = 1;
++ if (i == transport->smart_options->object_info_oids->nr - 1)
++ /* last element, no need to allocat to next */
++ ref_itr -> next = NULL;
++ else
++ ref_itr->next = alloc_ref("");
+
+- if (!data->finished_handshake) {
+- int i;
++ ref_itr = ref_itr->next;
+ }
-+ transport->remote_refs = object_info_refs->next;
++
++ transport->remote_refs = object_info_refs;
++
+ } else if (!data->finished_handshake) {
int must_list_refs = 0;
- for (i = 0; i < nr_heads; i++) {
@@ transport.c: static int fetch_refs_via_pack(struct transport *transport,
data->finished_handshake = 0;
+ if (args.object_info) {
-+ struct ref *ref_cpy_reader = object_info_refs->next;
++ struct ref *ref_cpy_reader = object_info_refs;
+ for (int i = 0; ref_cpy_reader; i++) {
-+ oid_object_info_extended(the_repository, &ref_cpy_reader->old_oid, &(*args.object_info_data)[i], OBJECT_INFO_LOOKUP_REPLACE);
++ oid_object_info_extended(the_repository, &ref_cpy_reader->old_oid, &args.object_info_data[i], OBJECT_INFO_LOOKUP_REPLACE);
+ ref_cpy_reader = ref_cpy_reader->next;
+ }
+ }
++
data->options.self_contained_and_connected =
args.self_contained_and_connected;
data->options.connectivity_checked = args.connectivity_checked;
@@ transport.c: static int fetch_refs_via_pack(struct transport *transport,
ret = -1;
if (report_unmatched_refs(to_fetch, nr_heads))
ret = -1;
-@@ transport.c: static int fetch_refs_via_pack(struct transport *transport,
+
+ cleanup:
++ free_refs(object_info_refs);
+ close(data->fd[0]);
+ if (data->fd[1] >= 0)
+ close(data->fd[1]);
+ if (finish_connect(data->conn))
+ ret = -1;
+ data->conn = NULL;
+-
free_refs(refs_tmp);
free_refs(refs);
list_objects_filter_release(&args.filter_options);
-+ free_refs(object_info_refs);
- return ret;
- }
-
## transport.h ##
@@
@@ transport.h: struct git_transport_options {
struct oidset *acked_commits;
+
+ struct oid_array *object_info_oids;
-+ struct object_info **object_info_data;
++ struct object_info *object_info_data;
+ struct string_list *object_info_options;
};
5: 6526e24aa4 ! 5: 2787327782 cat-file: add declaration of variable i inside its for loop
@@ Commit message
Change the declaration of i to be inside the for loop for readability.
- Signed-off-by: Eric Ju [off-list ref]
Helped-by: Christian Couder [off-list ref]
+ Signed-off-by: Eric Ju [off-list ref]
## builtin/cat-file.c ##
@@ builtin/cat-file.c: static void dispatch_calls(struct batch_options *opt,
@@ builtin/cat-file.c: static void dispatch_calls(struct batch_options *opt,
die(_("flush is only for --buffer mode"));
- for (i = 0; i < nr; i++)
-+ for (int i = 0; i < nr; i++)
++ for (size_t i = 0; i < nr; i++)
cmd[i].fn(opt, cmd[i].line, output, data);
fflush(stdout);
@@ builtin/cat-file.c: static void batch_objects_command(struct batch_options *opt,
die(_("whitespace before command: '%s'"), input.buf);
- for (i = 0; i < ARRAY_SIZE(commands); i++) {
-+ for (int i = 0; i < ARRAY_SIZE(commands); i++) {
++ for (size_t i = 0; i < ARRAY_SIZE(commands); i++) {
if (!skip_prefix(input.buf, commands[i].name, &cmd_end))
continue;
6: 5cd1a1dbd2 < -: ---------- cat-file: add remote-object-info to batch-command
-: ---------- > 6: cb114765cf cat-file: add remote-object-info to batch-command
--
2.45.2
From: Calvin Wan <redacted>
A subsequent patch needs to write capabilities for another command.
Refactor write_fetch_command_and_capabilities() to be a more general
purpose function write_command_and_capabilities(), so that it can be
used by both fetch and future command.
Here "command" means the "operations" supported by Git’s wire protocol
https://git-scm.com/docs/protocol-v2. An example would be a
git's subcommand, such as git-fetch(1); or an operation supported by
the server side such as "object-info" implemented in "a2ba162cda
(object-info: support for retrieving object info, 2021-04-20)".
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
---
fetch-pack.c | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
From: Calvin Wan <redacted>
There are some variables initialized at the start of the
do_fetch_pack_v2() state machine. Currently, they are initialized
in FETCH_CHECK_LOCAL, which is the initial state set at the beginning
of the function.
However, a subsequent patch will allow for another initial state,
while still requiring these initialized variables.
Move the initialization to be before the state machine,
so that they are set regardless of the initial state.
Note that there is no change in behavior, because we're moving code
from the beginning of the first state to just before the execution of
the state machine.
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
---
fetch-pack.c | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
@@ -1675,18 +1675,18 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,reader.me="fetch-pack";}+/* v2 supports these by default */+allow_unadvertised_object_request|=ALLOW_REACHABLE_SHA1;+use_sideband=2;+if(args->depth>0||args->deepen_since||args->deepen_not)+args->deepen=1;+while(state!=FETCH_DONE){switch(state){caseFETCH_CHECK_LOCAL:sort_ref_list(&ref,ref_compare_name);QSORT(sought,nr_sought,cmp_ref_by_name);-/* v2 supports these by default */-allow_unadvertised_object_request|=ALLOW_REACHABLE_SHA1;-use_sideband=2;-if(args->depth>0||args->deepen_since||args->deepen_not)-args->deepen=1;-/* Filter 'ref' by 'sought' and those that aren't local */mark_complete_and_common_ref(negotiator,args,&ref);filter_refs(args,&ref,sought,nr_sought);
From: Calvin Wan <redacted>
In order for a client to know what object-info components a server can
provide, advertise supported object-info features. This will allow a
client to decide whether to query the server for object-info or fetch
as a fallback.
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
---
serve.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
From: Calvin Wan <redacted>
Sometimes it is useful to get information about an object without having
to download it completely. The server logic has already been implemented
in “a2ba162cda (object-info: support for retrieving object info,
2021-04-20)”.
Add client functions to communicate with the server.
The client currently supports requesting a list of object ids with
feature 'size' from a v2 server. If a server does not
advertise the feature, then the client falls back
to making the request through 'fetch'.
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
---
fetch-pack.c | 24 +++++++++
fetch-pack.h | 10 ++++
transport-helper.c | 8 ++-
transport.c | 118 +++++++++++++++++++++++++++++++++++++++++++--
transport.h | 11 +++++
5 files changed, 164 insertions(+), 7 deletions(-)
@@ -106,4 +114,6 @@ int report_unmatched_refs(struct ref **sought, int nr_sought);*/intfetch_pack_fsck_objects(void);+voidsend_object_info_request(intfd_out,structobject_info_args*args);+#endif
@@ -699,13 +699,17 @@ static int fetch_refs(struct transport *transport,/**Ifwereachhere,thentheserver,theclient,and/orthetransport-*helperdoesnotsupportprotocolv2.--negotiate-onlyrequires-*protocolv2.+*helperdoesnotsupportprotocolv2.--negotiate-onlyandcat-fileremote-object-info+*requireprotocolv2.*/if(data->transport_options.acked_commits){warning(_("--negotiate-only requires protocol v2"));return-1;}+if(transport->smart_options->object_info){+// fail the command explicitly to avoid further commands input+die(_("remote-object-info requires protocol v2"));+}if(!data->get_refs_list_called)get_refs_list_using_list(transport,0);
@@ -366,6 +366,80 @@ static struct ref *handshake(struct transport *transport, int for_push,returnrefs;}+staticintfetch_object_info(structtransport*transport,structobject_info*object_info_data)+{+intsize_index=-1;+structgit_transport_data*data=transport->data;+structobject_info_argsargs={0};+structpacket_readerreader;++args.server_options=transport->server_options;+args.object_info_options=transport->smart_options->object_info_options;+args.oids=transport->smart_options->object_info_oids;++connect_setup(transport,0);+packet_reader_init(&reader,data->fd[0],NULL,0,+PACKET_READ_CHOMP_NEWLINE|+PACKET_READ_GENTLE_ON_EOF|+PACKET_READ_DIE_ON_ERR_PACKET);+data->version=discover_version(&reader);++transport->hash_algo=reader.hash_algo;++switch(data->version){+caseprotocol_v2:+if(!server_supports_v2("object-info"))+return-1;+if(unsorted_string_list_has_string(args.object_info_options,"size")+&&!server_supports_feature("object-info","size",0)){+return-1;+}+send_object_info_request(data->fd[1],&args);+break;+caseprotocol_v1:+caseprotocol_v0:+die(_("wrong protocol version. expected v2"));+caseprotocol_unknown_version:+BUG("unknown protocol version");+}++for(size_ti=0;i<args.object_info_options->nr;i++){+if(packet_reader_read(&reader)!=PACKET_READ_NORMAL){+check_stateless_delimiter(transport->stateless_rpc,&reader,"stateless delimiter expected");+return-1;+}+if(unsorted_string_list_has_string(args.object_info_options,reader.line)){+if(!strcmp(reader.line,"size")){+size_index=i;+for(size_tj=0;j<args.oids->nr;j++){+object_info_data[j].sizep=xcalloc(1,sizeof(long));+}+}+continue;+}+return-1;+}++for(size_ti=0;packet_reader_read(&reader)==PACKET_READ_NORMAL&&i<args.oids->nr;i++){+structstring_listobject_info_values=STRING_LIST_INIT_DUP;++string_list_split(&object_info_values,reader.line,' ',-1);+if(0<=size_index){+if(!strcmp(object_info_values.items[1+size_index].string,""))+die("object-info: not our ref %s",+object_info_values.items[0].string);++*object_info_data[i].sizep=strtoul(object_info_values.items[1+size_index].string,NULL,10);++}++string_list_clear(&object_info_values,0);+}+check_stateless_delimiter(transport->stateless_rpc,&reader,"stateless delimiter expected");++return0;+}+staticstructref*get_refs_via_connect(structtransport*transport,intfor_push,structtransport_ls_refs_options*options){
@@ -413,6 +487,7 @@ static int fetch_refs_via_pack(struct transport *transport,structref*refs=NULL;structfetch_pack_argsargs;structref*refs_tmp=NULL;+structref*object_info_refs=NULL;memset(&args,0,sizeof(args));args.uploadpack=data->options.uploadpack;
@@ -439,11 +514,36 @@ static int fetch_refs_via_pack(struct transport *transport,args.server_options=transport->server_options;args.negotiation_tips=data->options.negotiation_tips;args.reject_shallow_remote=transport->smart_options->reject_shallow;+args.object_info=transport->smart_options->object_info;++if(transport->smart_options+&&transport->smart_options->object_info+&&transport->smart_options->object_info_oids->nr>0){+structref*ref_itr=object_info_refs=alloc_ref("");++if(!fetch_object_info(transport,data->options.object_info_data))+gotocleanup;++args.object_info_data=data->options.object_info_data;+args.quiet=1;+args.no_progress=1;+for(size_ti=0;i<transport->smart_options->object_info_oids->nr;i++){+ref_itr->old_oid=transport->smart_options->object_info_oids->oid[i];+ref_itr->exact_oid=1;+if(i==transport->smart_options->object_info_oids->nr-1)+/* last element, no need to allocat to next */+ref_itr->next=NULL;+else+ref_itr->next=alloc_ref("");-if(!data->finished_handshake){-inti;+ref_itr=ref_itr->next;+}++transport->remote_refs=object_info_refs;++}elseif(!data->finished_handshake){intmust_list_refs=0;-for(i=0;i<nr_heads;i++){+for(inti=0;i<nr_heads;i++){if(!to_fetch[i]->exact_oid){must_list_refs=1;break;
@@ -481,23 +581,31 @@ static int fetch_refs_via_pack(struct transport *transport,&transport->pack_lockfiles,data->version);data->finished_handshake=0;+if(args.object_info){+structref*ref_cpy_reader=object_info_refs;+for(inti=0;ref_cpy_reader;i++){+oid_object_info_extended(the_repository,&ref_cpy_reader->old_oid,&args.object_info_data[i],OBJECT_INFO_LOOKUP_REPLACE);+ref_cpy_reader=ref_cpy_reader->next;+}+}+data->options.self_contained_and_connected=args.self_contained_and_connected;data->options.connectivity_checked=args.connectivity_checked;-if(!refs)+if(!refs&&!args.object_info)ret=-1;if(report_unmatched_refs(to_fetch,nr_heads))ret=-1;cleanup:+free_refs(object_info_refs);close(data->fd[0]);if(data->fd[1]>=0)close(data->fd[1]);if(finish_connect(data->conn))ret=-1;data->conn=NULL;-free_refs(refs_tmp);free_refs(refs);list_objects_filter_release(&args.filter_options);
Some code declares variable i and only uses it
in a for loop, not in any other logic outside the loop.
Change the declaration of i to be inside the for loop for readability.
Helped-by: Christian Couder [off-list ref]
Signed-off-by: Eric Ju <redacted>
---
builtin/cat-file.c | 11 +++--------
1 file changed, 3 insertions(+), 8 deletions(-)
@@ -673,12 +673,10 @@ static void dispatch_calls(struct batch_options *opt,structqueued_cmd*cmd,intnr){-inti;-if(!opt->buffer_output)die(_("flush is only for --buffer mode"));-for(i=0;i<nr;i++)+for(size_ti=0;i<nr;i++)cmd[i].fn(opt,cmd[i].line,output,data);fflush(stdout);
From: Calvin Wan <redacted>
Since the `info` command in cat-file --batch-command prints object info
for a given object, it is natural to add another command in cat-file
--batch-command to print object info for a given object from a remote.
Add `remote-object-info` to cat-file --batch-command.
While `info` takes object ids one at a time, this creates overhead when
making requests to a server so `remote-object-info` instead can take
multiple object ids at once.
cat-file --batch-command is generally implemented in the following
manner:
- Receive and parse input from user
- Call respective function attached to command
- Get object info, print object info
In --buffer mode, this changes to:
- Receive and parse input from user
- Store respective function attached to command in a queue
- After flush, loop through commands in queue
- Call respective function attached to command
- Get object info, print object info
Notice how the getting and printing of object info is accomplished one
at a time. As described above, this creates a problem for making
requests to a server. Therefore, `remote-object-info` is implemented in
the following manner:
- Receive and parse input from user
If command is `remote-object-info`:
- Get object info from remote
- Loop through and print each object info
Else:
- Call respective function attached to command
- Parse input, get object info, print object info
And finally for --buffer mode `remote-object-info`:
- Receive and parse input from user
- Store respective function attached to command in a queue
- After flush, loop through commands in queue:
If command is `remote-object-info`:
- Get object info from remote
- Loop through and print each object info
Else:
- Call respective function attached to command
- Get object info, print object info
To summarize, `remote-object-info` gets object info from the remote and
then loop through the object info passed in, print the info.
In order for remote-object-info to avoid remote communication overhead
in the non-buffer mode, the objects are passed in as such:
remote-object-info <remote> <oid> <oid> ... <oid>
rather than
remote-object-info <remote> <oid>
remote-object-info <remote> <oid>
...
remote-object-info <remote> <oid>
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
---
Documentation/git-cat-file.txt | 23 +-
builtin/cat-file.c | 116 +++-
object-file.c | 11 +
object-store-ll.h | 3 +
t/t1017-cat-file-remote-object-info.sh | 748 +++++++++++++++++++++++++
5 files changed, 893 insertions(+), 8 deletions(-)
create mode 100755 t/t1017-cat-file-remote-object-info.sh
@@ -149,6 +149,13 @@ info <object>:: Print object info for object reference `<object>`. This corresponds to the output of `--batch-check`.+remote-object-info <remote> <object>...::+ Print object info for object references `<object>` at specified <remote> without+ downloading objects from remote. If the object-info capability is not+ supported by the server, the objects will be downloaded instead.+ Error when no object references is provided.+ This command may be combined with `--buffer`.+ flush:: Used with `--buffer` to execute all preceding commands that were issued since the beginning or since the last flush was issued. When `--buffer`
@@ -290,7 +297,8 @@ newline. The available atoms are: The full hex representation of the object name. `objecttype`::- The type of the object (the same as `cat-file -t` reports).+ The type of the object (the same as `cat-file -t` reports). See+ `CAVEATS` below. Not supported by `remote-object-info`. `objectsize`:: The size, in bytes, of the object (the same as `cat-file -s`
@@ -298,13 +306,14 @@ newline. The available atoms are: `objectsize:disk`:: The size, in bytes, that the object takes up on disk. See the- note about on-disk sizes in the `CAVEATS` section below.+ note about on-disk sizes in the `CAVEATS` section below. Not+ supported by `remote-object-info`. `deltabase`:: If the object is stored as a delta on-disk, this expands to the full hex representation of the delta base object name. Otherwise, expands to the null OID (all zeroes). See `CAVEATS`- below.+ below. Not supported by `remote-object-info`. `rest`:: If this atom is used in the output string, input lines are split
@@ -314,7 +323,9 @@ newline. The available atoms are: line) are output in place of the `%(rest)` atom. If no format is specified, the default format is `%(objectname)-%(objecttype) %(objectsize)`.+%(objecttype) %(objectsize)`, except remote-object-info command who uses+`%(objectname) %(objectsize)` for now because "%(objecttype)" is not supported yet.+When "%(objecttype)" is supported, default format should be unified. If `--batch` is specified, or if `--batch-command` is used with the `contents` command, the object information is followed by the object contents (consisting
@@ -396,6 +407,10 @@ scripting purposes. CAVEATS -------+Note that since objecttype, objectsize:disk and deltabase are currently not supported by the+remote-object-info, git will error and exit when they are in the format string.++ Note that the sizes of objects on disk are reported accurately, but care should be taken in drawing conclusions about which refs or objects are responsible for disk usage. The size of a packed non-delta object may be
@@ -576,6 +582,61 @@ static void batch_one_object(const char *obj_name,object_context_release(&ctx);}+staticintget_remote_info(structbatch_options*opt,intargc,constchar**argv)+{+intretval=0;+structremote*remote=NULL;+structobject_idoid;+structstring_listobject_info_options=STRING_LIST_INIT_NODUP;+staticstructtransport*gtransport;++/*+*Changetheformatto"%(objectname) %(objectsize)"when+*remote-object-infocommandisused.Oncewestartsupportingobjecttype+*thedefaultformatshouldchangetoDEFAULT_FORMAT+*/+if(!opt->format)+opt->format="%(objectname) %(objectsize)";++remote=remote_get(argv[0]);+if(!remote)+die(_("must supply valid remote when using remote-object-info"));++oid_array_clear(&object_info_oids);+for(size_ti=1;i<argc;i++){+if(get_oid_hex(argv[i],&oid))+die(_("Not a valid object name %s"),argv[i]);+oid_array_append(&object_info_oids,&oid);+}++gtransport=transport_get(remote,NULL);+if(gtransport->smart_options){+CALLOC_ARRAY(remote_object_info,object_info_oids.nr);+gtransport->smart_options->object_info=1;+gtransport->smart_options->object_info_oids=&object_info_oids;+/*+*'size'istheonlyoptioncurrentlysupported.+*Otheroptionsthatarepassedintheformatwillexitwitherror.+*/+if(strstr(opt->format,"%(objectsize)")){+string_list_append(&object_info_options,"size");+}else{+die(_("%s is currently not supported with remote-object-info"),opt->format);+}+if(object_info_options.nr>0){+gtransport->smart_options->object_info_options=&object_info_options;+gtransport->smart_options->object_info_data=remote_object_info;+retval=transport_fetch_refs(gtransport,NULL);+}+}else{+retval=-1;+}++string_list_clear(&object_info_options,0);+transport_disconnect(gtransport);+returnretval;+}+structobject_cb_data{structbatch_options*opt;structexpand_data*expand;
@@ -0,0 +1,748 @@+#!/bin/sh++test_description='git cat-file --batch-command with remote-object-info command'++GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main+exportGIT_TEST_DEFAULT_INITIAL_BRANCH_NAME++../test-lib.sh++echo_without_newline(){+printf'%s'"$*"+}++echo_without_newline_nul(){+echo_without_newline"$@"|tr'\n''\0'+}++strlen(){+echo_without_newline"$1"|wc-c|sed-e's/^ *//'+}++hello_content="Hello World"+hello_size=$(strlen"$hello_content")+hello_oid=$(echo_without_newline"$hello_content"|githash-object--stdin)++# This is how we get 13:+# 13 = <file mode> + <a_space> + <file name> + <a_null>, where+# file mode is 100644, which is 6 characters;+# file name is hello, which is 5 characters+# a space is 1 character and a null is 1 character+tree_size=$(($(test_oidrawsz)+13))++commit_message="Initial commit"++# This is how we get 137:+# 137 = <tree header> + <a_space> + <a newline> ++# <Author line> + <a newline> ++# <Committer line> + <a newline> ++# <a newline> ++# <commit message length>+# An easier way to calculate is: 1. use `git cat-file commit <commit hash> | wc -c`,+# to get 177, 2. then deduct 40 hex characters to get 137+commit_size=$(($(test_oidhexsz)+137))++tag_header_without_oid="type blob+taghellotag+tagger$GIT_COMMITTER_NAME<$GIT_COMMITTER_EMAIL>"+tag_header_without_timestamp="object $hello_oid+$tag_header_without_oid"+tag_description="This is a tag"+tag_content="$tag_header_without_timestamp 0 +0000++$tag_description"++tag_oid=$(echo_without_newline"$tag_content"|githash-object-ttag--stdin-w)+tag_size=$(strlen"$tag_content")++set_transport_variables(){+hello_sha1=$(echo_without_newline"$hello_content"|githash-object--stdin)+tree_sha1=$(git-C"$1"write-tree)+commit_sha1=$(echo_without_newline"$commit_message"|git-C"$1"commit-tree$tree_sha1)+tag_sha1=$(echo_without_newline"$tag_content"|git-C"$1"hash-object-ttag--stdin-w)+tag_size=$(strlen"$tag_content")+}++# This section tests --batch-command with remote-object-info command+# Since "%(objecttype)" is currently not supported by the command remote-object-info ,+# the filters are set to "%(objectname) %(objectsize)" in some test cases.++# Test --batch-command remote-object-info with 'git://' transport+."$TEST_DIRECTORY"/lib-git-daemon.sh+start_git_daemon--export-all--enable=receive-pack+daemon_parent=$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent++test_expect_success'create repo to be served by git-daemon''+gitinit"$daemon_parent"&&+echo_without_newline"$hello_content">$daemon_parent/hello&&+git-C"$daemon_parent"update-index--addhello&&+git-C"$daemon_parent"configtransfer.advertiseobjectinfotrue&&+gitclone"$GIT_DAEMON_URL/parent"-n"$daemon_parent/daemon_client_empty"+'++test_expect_success'batch-command remote-object-info git://''+(+set_transport_variables"$daemon_parent"&&+cd"$daemon_parent/daemon_client_empty"&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_sha1$hello_size">expect&&+echo"$tree_sha1$tree_size">>expect&&+echo"$commit_sha1$commit_size">>expect&&+echo"$tag_sha1$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_sha1 missing">>expect&&+echo"$tree_sha1 missing">>expect&&+echo"$commit_sha1 missing">>expect&&+echo"$tag_sha1 missing">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+remote-object-info"$GIT_DAEMON_URL/parent"$hello_sha1+remote-object-info"$GIT_DAEMON_URL/parent"$tree_sha1+remote-object-info"$GIT_DAEMON_URL/parent"$commit_sha1+remote-object-info"$GIT_DAEMON_URL/parent"$tag_sha1+info$hello_sha1+info$tree_sha1+info$commit_sha1+info$tag_sha1+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command remote-object-info git:// multiple sha1 per line''+(+set_transport_variables"$daemon_parent"&&+cd"$daemon_parent/daemon_client_empty"&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_sha1$hello_size">expect&&+echo"$tree_sha1$tree_size">>expect&&+echo"$commit_sha1$commit_size">>expect&&+echo"$tag_sha1$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_sha1 missing">>expect&&+echo"$tree_sha1 missing">>expect&&+echo"$commit_sha1 missing">>expect&&+echo"$tag_sha1 missing">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+remote-object-info"$GIT_DAEMON_URL/parent"$hello_sha1$tree_sha1$commit_sha1$tag_sha1+info$hello_sha1+info$tree_sha1+info$commit_sha1+info$tag_sha1+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command remote-object-info git:// default filter''+(+set_transport_variables"$daemon_parent"&&+cd"$daemon_parent/daemon_client_empty"&&++echo"$hello_sha1$hello_size">expect&&+echo"$tree_sha1$tree_size">>expect&&+echo"$commit_sha1$commit_size">>expect&&+echo"$tag_sha1$tag_size">>expect&&+GIT_TRACE_PACKET=1gitcat-file--batch-command>actual<<-EOF&&+remote-object-info"$GIT_DAEMON_URL/parent"$hello_sha1$tree_sha1+remote-object-info"$GIT_DAEMON_URL/parent"$commit_sha1$tag_sha1+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command --buffer remote-object-info git://''+(+set_transport_variables"$daemon_parent"&&+cd"$daemon_parent/daemon_client_empty"&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_sha1$hello_size">expect&&+echo"$tree_sha1$tree_size">>expect&&+echo"$commit_sha1$commit_size">>expect&&+echo"$tag_sha1$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_sha1 missing">>expect&&+echo"$tree_sha1 missing">>expect&&+echo"$commit_sha1 missing">>expect&&+echo"$tag_sha1 missing">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)"--buffer>actual<<-EOF&&+remote-object-info"$GIT_DAEMON_URL/parent"$hello_sha1$tree_sha1+remote-object-info"$GIT_DAEMON_URL/parent"$commit_sha1$tag_sha1+info$hello_sha1+info$tree_sha1+info$commit_sha1+info$tag_sha1+flush+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command -Z remote-object-info git:// default filter''+(+set_transport_variables"$daemon_parent"&&+cd"$daemon_parent/daemon_client_empty"&&++printf"%s\0""$hello_sha1$hello_size">expect&&+printf"%s\0""$tree_sha1$tree_size">>expect&&+printf"%s\0""$commit_sha1$commit_size">>expect&&+printf"%s\0""$tag_sha1$tag_size">>expect&&++printf"%s\0""$hello_sha1 missing">>expect&&+printf"%s\0""$tree_sha1 missing">>expect&&+printf"%s\0""$commit_sha1 missing">>expect&&+printf"%s\0""$tag_sha1 missing">>expect&&++batch_input="remote-object-info $GIT_DAEMON_URL/parent $hello_sha1$tree_sha1+remote-object-info$GIT_DAEMON_URL/parent$commit_sha1$tag_sha1+info$hello_sha1+info$tree_sha1+info$commit_sha1+info$tag_sha1+" &&+echo_without_newline_nul"$batch_input">commands_null_delimited&&++gitcat-file--batch-command-Z<commands_null_delimited>actual&&+test_cmpexpectactual+)+'++# Test --batch-command remote-object-info with 'git://' and+# transfer.advertiseobjectinfo set to false, i.e. server does not have object-info capability++test_expect_success'remote-object-info fallback git://: fetch objects to client''+(+git-C"$daemon_parent"configtransfer.advertiseobjectinfofalse&&+set_transport_variables"$daemon_parent"&&+cd"$daemon_parent/daemon_client_empty"&&++# Prove object is not on the client+echo"$hello_sha1 missing">expect&&+echo"$tree_sha1 missing">>expect&&+echo"$commit_sha1 missing">>expect&&+echo"$tag_sha1 missing">>expect&&++# These results prove remote-object-info can retrieve object info+echo"$hello_sha1$hello_size">>expect&&+echo"$tree_sha1$tree_size">>expect&&+echo"$commit_sha1$commit_size">>expect&&+echo"$tag_sha1$tag_size">>expect&&++# These results are for the info command+# They prove objects are downloaded+echo"$hello_sha1$hello_size">>expect&&+echo"$tree_sha1$tree_size">>expect&&+echo"$commit_sha1$commit_size">>expect&&+echo"$tag_sha1$tag_size">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+info$hello_sha1+info$tree_sha1+info$commit_sha1+info$tag_sha1+remote-object-info$GIT_DAEMON_URL/parent$hello_sha1$tree_sha1$commit_sha1$tag_sha1+info$hello_sha1+info$tree_sha1+info$commit_sha1+info$tag_sha1+EOF++# revert server state back+git-C"$daemon_parent"configtransfer.advertiseobjectinfotrue&&++test_cmpexpectactual+)+'++stop_git_daemon++# Test --batch-command remote-object-info with 'file://' transport+# shellcheck disable=SC2016+test_expect_success'create repo to be served by file:// transport''+gitinitserver&&+git-Cserverconfigprotocol.version2&&+git-Cserverconfigtransfer.advertiseobjectinfotrue&&+echo_without_newline"$hello_content">server/hello&&+git-Cserverupdate-index--addhello&&+gitclone-n"file://$(pwd)/server"file_client_empty+'++test_expect_success'batch-command remote-object-info file://''+(+set_transport_variables"server"&&+server_path="$(pwd)/server"&&+cdfile_client_empty&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_sha1$hello_size">expect&&+echo"$tree_sha1$tree_size">>expect&&+echo"$commit_sha1$commit_size">>expect&&+echo"$tag_sha1$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_sha1 missing">>expect&&+echo"$tree_sha1 missing">>expect&&+echo"$commit_sha1 missing">>expect&&+echo"$tag_sha1 missing">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+remote-object-info"file://${server_path}"$hello_sha1+remote-object-info"file://${server_path}"$tree_sha1+remote-object-info"file://${server_path}"$commit_sha1+remote-object-info"file://${server_path}"$tag_sha1+info$hello_sha1+info$tree_sha1+info$commit_sha1+info$tag_sha1+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command remote-object-info file:// multiple sha1 per line''+(+set_transport_variables"server"&&+server_path="$(pwd)/server"&&+cdfile_client_empty&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_sha1$hello_size">expect&&+echo"$tree_sha1$tree_size">>expect&&+echo"$commit_sha1$commit_size">>expect&&+echo"$tag_sha1$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_sha1 missing">>expect&&+echo"$tree_sha1 missing">>expect&&+echo"$commit_sha1 missing">>expect&&+echo"$tag_sha1 missing">>expect&&+++gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+remote-object-info"file://${server_path}"$hello_sha1$tree_sha1$commit_sha1$tag_sha1+info$hello_sha1+info$tree_sha1+info$commit_sha1+info$tag_sha1+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command --buffer remote-object-info file://''+(+set_transport_variables"server"&&+server_path="$(pwd)/server"&&+cdfile_client_empty&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_sha1$hello_size">expect&&+echo"$tree_sha1$tree_size">>expect&&+echo"$commit_sha1$commit_size">>expect&&+echo"$tag_sha1$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_sha1 missing">>expect&&+echo"$tree_sha1 missing">>expect&&+echo"$commit_sha1 missing">>expect&&+echo"$tag_sha1 missing">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)"--buffer>actual<<-EOF&&+remote-object-info"file://${server_path}"$hello_sha1$tree_sha1+remote-object-info"file://${server_path}"$commit_sha1$tag_sha1+info$hello_sha1+info$tree_sha1+info$commit_sha1+info$tag_sha1+flush+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command remote-object-info file:// default filter''+(+set_transport_variables"server"&&+server_path="$(pwd)/server"&&+cdfile_client_empty&&++echo"$hello_sha1$hello_size">expect&&+echo"$tree_sha1$tree_size">>expect&&+echo"$commit_sha1$commit_size">>expect&&+echo"$tag_sha1$tag_size">>expect&&++gitcat-file--batch-command>actual<<-EOF&&+remote-object-info"file://${server_path}"$hello_sha1$tree_sha1+remote-object-info"file://${server_path}"$commit_sha1$tag_sha1+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command -Z remote-object-info file:// default filter''+(+set_transport_variables"server"&&+server_path="$(pwd)/server"&&+cdfile_client_empty&&++printf"%s\0""$hello_sha1$hello_size">expect&&+printf"%s\0""$tree_sha1$tree_size">>expect&&+printf"%s\0""$commit_sha1$commit_size">>expect&&+printf"%s\0""$tag_sha1$tag_size">>expect&&++printf"%s\0""$hello_sha1 missing">>expect&&+printf"%s\0""$tree_sha1 missing">>expect&&+printf"%s\0""$commit_sha1 missing">>expect&&+printf"%s\0""$tag_sha1 missing">>expect&&++batch_input="remote-object-info \"file://${server_path}\" $hello_sha1$tree_sha1+remote-object-info\"file://${server_path}\"$commit_sha1$tag_sha1+info$hello_sha1+info$tree_sha1+info$commit_sha1+info$tag_sha1+" &&+echo_without_newline_nul"$batch_input">commands_null_delimited&&++gitcat-file--batch-command-Z<commands_null_delimited>actual&&+test_cmpexpectactual+)+'++# Test --batch-command remote-object-info with 'file://' and+# transfer.advertiseobjectinfo set to false, i.e. server does not have object-info capability++test_expect_success'remote-object-info fallback file://: fetch objects to client''+(+set_transport_variables"server"&&+server_path="$(pwd)/server"&&+git-C"${server_path}"configtransfer.advertiseobjectinfofalse&&+cdfile_client_empty&&++# Prove object is not on the client+echo"$hello_sha1 missing">expect&&+echo"$tree_sha1 missing">>expect&&+echo"$commit_sha1 missing">>expect&&+echo"$tag_sha1 missing">>expect&&++# These results prove remote-object-info can retrieve object info+echo"$hello_sha1$hello_size">>expect&&+echo"$tree_sha1$tree_size">>expect&&+echo"$commit_sha1$commit_size">>expect&&+echo"$tag_sha1$tag_size">>expect&&++# These results are for the info command+# They prove objects are downloaded+echo"$hello_sha1$hello_size">>expect&&+echo"$tree_sha1$tree_size">>expect&&+echo"$commit_sha1$commit_size">>expect&&+echo"$tag_sha1$tag_size">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+info$hello_sha1+info$tree_sha1+info$commit_sha1+info$tag_sha1+remote-object-info"file://${server_path}"$hello_sha1$tree_sha1$commit_sha1$tag_sha1+info$hello_sha1+info$tree_sha1+info$commit_sha1+info$tag_sha1+EOF++# revert server state back+git-C"${server_path}"configtransfer.advertiseobjectinfotrue&&+test_cmpexpectactual+)+'++# Test --batch-command remote-object-info with 'http://' transport with+# transfer.advertiseobjectinfo set to true, i.e. server has object-info capability++."$TEST_DIRECTORY"/lib-httpd.sh+start_httpd++test_expect_success'create repo to be served by http:// transport''+gitinit"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+git-C"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"confighttp.receivepacktrue&&+git-C"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"configtransfer.advertiseobjectinfotrue&&+echo_without_newline"$hello_content">$HTTPD_DOCUMENT_ROOT_PATH/http_parent/hello&&+git-C"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"update-index--addhello&&+gitclone"$HTTPD_URL/smart/http_parent"-n"$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty"+'++test_expect_success'batch-command remote-object-info http://''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty"&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_sha1$hello_size">expect&&+echo"$tree_sha1$tree_size">>expect&&+echo"$commit_sha1$commit_size">>expect&&+echo"$tag_sha1$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_sha1 missing">>expect&&+echo"$tree_sha1 missing">>expect&&+echo"$commit_sha1 missing">>expect&&+echo"$tag_sha1 missing">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_sha1+remote-object-info"$HTTPD_URL/smart/http_parent"$tree_sha1+remote-object-info"$HTTPD_URL/smart/http_parent"$commit_sha1+remote-object-info"$HTTPD_URL/smart/http_parent"$tag_sha1+info$hello_sha1+info$tree_sha1+info$commit_sha1+info$tag_sha1+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command remote-object-info http:// one line''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty"&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_sha1$hello_size">expect&&+echo"$tree_sha1$tree_size">>expect&&+echo"$commit_sha1$commit_size">>expect&&+echo"$tag_sha1$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_sha1 missing">>expect&&+echo"$tree_sha1 missing">>expect&&+echo"$commit_sha1 missing">>expect&&+echo"$tag_sha1 missing">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_sha1$tree_sha1$commit_sha1$tag_sha1+info$hello_sha1+info$tree_sha1+info$commit_sha1+info$tag_sha1+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command --buffer remote-object-info http://''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty"&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_sha1$hello_size">expect&&+echo"$tree_sha1$tree_size">>expect&&+echo"$commit_sha1$commit_size">>expect&&+echo"$tag_sha1$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_sha1 missing">>expect&&+echo"$tree_sha1 missing">>expect&&+echo"$commit_sha1 missing">>expect&&+echo"$tag_sha1 missing">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)"--buffer>actual<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_sha1$tree_sha1+remote-object-info"$HTTPD_URL/smart/http_parent"$commit_sha1$tag_sha1+info$hello_sha1+info$tree_sha1+info$commit_sha1+info$tag_sha1+flush+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command remote-object-info http:// default filter''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty"&&++echo"$hello_sha1$hello_size">expect&&+echo"$tree_sha1$tree_size">>expect&&+echo"$commit_sha1$commit_size">>expect&&+echo"$tag_sha1$tag_size">>expect&&++gitcat-file--batch-command>actual<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_sha1$tree_sha1+remote-object-info"$HTTPD_URL/smart/http_parent"$commit_sha1$tag_sha1+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command -Z remote-object-info http:// default filter''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty"&&++printf"%s\0""$hello_sha1$hello_size">expect&&+printf"%s\0""$tree_sha1$tree_size">>expect&&+printf"%s\0""$commit_sha1$commit_size">>expect&&+printf"%s\0""$tag_sha1$tag_size">>expect&&++batch_input="remote-object-info $HTTPD_URL/smart/http_parent $hello_sha1$tree_sha1+remote-object-info$HTTPD_URL/smart/http_parent$commit_sha1$tag_sha1+" &&+echo_without_newline_nul"$batch_input">commands_null_delimited&&++gitcat-file--batch-command-Z<commands_null_delimited>actual&&+test_cmpexpectactual+)+'++test_expect_success'remote-object-info fails on unspported filter option (objectsize:disk)''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&++test_must_failgitcat-file--batch-command="%(objectsize:disk)"2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_sha1+EOF+test_grep"%(objectsize:disk) is currently not supported with remote-object-info"err+)+'++test_expect_success'remote-object-info fails on unspported filter option (deltabase)''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&++test_must_failgitcat-file--batch-command="%(deltabase)"2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_sha1+EOF+test_grep"%(deltabase) is currently not supported with remote-object-info"err+)+'++test_expect_success'remote-object-info fails on server with legacy protocol''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&++test_must_failgit-cprotocol.version=0cat-file--batch-command="%(objectname) %(objectsize)"2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_sha1+EOF+test_grep"remote-object-info requires protocol v2"err+)+'++test_expect_success'remote-object-info fails on server with legacy protocol fallback''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&++test_must_failgit-cprotocol.version=0cat-file--batch-command2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_sha1+EOF+test_grep"remote-object-info requires protocol v2"err+)+'++test_expect_success'remote-object-info fails on malformed OID''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+malformed_object_id="this_id_is_not_valid"&&++test_must_failgitcat-file--batch-command="%(objectname) %(objectsize)"2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$malformed_object_id+EOF+test_grep"Not a valid object name '$malformed_object_id'"err+)+'++test_expect_success'remote-object-info fails on malformed OID fallback''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+malformed_object_id="this_id_is_not_valid"&&++test_must_failgitcat-file--batch-command2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$malformed_object_id+EOF+test_grep"Not a valid object name '$malformed_object_id'"err+)+'++test_expect_success'remote-object-info fails on missing OID''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+gitclone"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"missing_oid_repo&&+test_commit-Cmissing_oid_repomessage1c.txt&&+cdmissing_oid_repo&&++object_id=$(gitrev-parsemessage1:c.txt)&&+test_must_failgitcat-file--batch-command="%(objectname) %(objectsize)"2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$object_id+EOF+test_grep"object-info: not our ref $object_id"err+)+'++# Test --batch-command remote-object-info with 'http://' transport and+# transfer.advertiseobjectinfo set to false, i.e. server does not have object-info capability++test_expect_success'remote-object-info fallback http://: fetch objects to client''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+git-C"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"configtransfer.advertiseobjectinfofalse&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty"&&++# Prove object is not on the client+echo"$hello_sha1 missing">expect&&+echo"$tree_sha1 missing">>expect&&+echo"$commit_sha1 missing">>expect&&+echo"$tag_sha1 missing">>expect&&++# These results prove remote-object-info can retrieve object info+echo"$hello_sha1$hello_size">>expect&&+echo"$tree_sha1$tree_size">>expect&&+echo"$commit_sha1$commit_size">>expect&&+echo"$tag_sha1$tag_size">>expect&&++# These results are for the info command+# They prove objects are downloaded+echo"$hello_sha1$hello_size">>expect&&+echo"$tree_sha1$tree_size">>expect&&+echo"$commit_sha1$commit_size">>expect&&+echo"$tag_sha1$tag_size">>expect&&++gitcat-file--batch-command>actual<<-EOF&&+info$hello_sha1+info$tree_sha1+info$commit_sha1+info$tag_sha1+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_sha1$tree_sha1$commit_sha1$tag_sha1+info$hello_sha1+info$tree_sha1+info$commit_sha1+info$tag_sha1+EOF++# revert server state back+git-C"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"configtransfer.advertiseobjectinfotrue&&++test_cmpexpectactual+)+'++# DO NOT add non-httpd-specific tests here, because the last part of this+# test script is only executed when httpd is available and enabled.++test_done
From: Peijian Ju <hidden> Date: 2024-08-22 21:24:46
Dear Reviewers,
Thank you for your thorough review of v1. I have addressed all the
issues identified in that version and have now prepared v2.
Could you please take another look and provide your acknowledgment?
Thank you very much for your time and effort.
Best regards,
Peijian
On Fri, Jun 28, 2024 at 3:05 PM Eric Ju [off-list ref] wrote:
This is a continuation of Calvin Wan's (calvinwan@google.com)
patch series [PATCH v5 0/6] cat-file: add --batch-command remote-object-info command at [1].
Sometimes it is useful to get information about an object without having to download
it completely. The server logic for retrieving size has already been implemented and merged in
"a2ba162cda (object-info: support for retrieving object info, 2021-04-20)"[2].
This patch series implement the client option for it.
This patch series add the `remote-object-info` command to `cat-file --batch-command`. This command
allows the client to make an object-info command request to a server
that supports protocol v2. If the server is v2, but does not have
object-info capability, the entire object is fetched and the
relevant object info is returned.
A few questions open for discussions please:
1. In the current implementation, if a user puts `remote-object-info` in protocol v1,
`cat-file --batch-command` will die. Which way do we prefer? "error and exit (i.e. die)"
or "warn and wait for new command".
2. Right now, only the size is supported. If the batch command format
contains objectsize:disk or deltabase, it will die. The question
is about objecttype. In the current implementation, it will die too.
But dying on objecttype breaks the default format. We have changed the
default format to %(objectname) %(objectsize) when remote-object-info is used.
Any suggestions on this approach?
[1] https://lore.kernel.org/git/20220728230210.2952731-1-calvinwan@google.com/#t
[2] https://git.kernel.org/pub/scm/git/git.git/commit/?id=a2ba162cda2acc171c3e36acbbc854792b093cb7
Calvin Wan (5):
fetch-pack: refactor packet writing
fetch-pack: move fetch initialization
serve: advertise object-info feature
transport: add client support for object-info
cat-file: add remote-object-info to batch-command
Eric Ju (1):
cat-file: add declaration of variable i inside its for loop
Documentation/git-cat-file.txt | 22 +-
builtin/cat-file.c | 240 ++++++++++----
fetch-pack.c | 48 ++-
fetch-pack.h | 10 +
object-file.c | 11 +
object-store-ll.h | 3 +
serve.c | 4 +-
t/t1017-cat-file-remote-object-info.sh | 412 +++++++++++++++++++++++++
transport-helper.c | 8 +-
transport.c | 102 +++++-
transport.h | 11 +
11 files changed, 785 insertions(+), 86 deletions(-)
create mode 100755 t/t1017-cat-file-remote-object-info.sh
--
2.45.2
@@ -0,0 +1,748 @@+#!/bin/sh++test_description='git cat-file --batch-command with remote-object-info command'++GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main+exportGIT_TEST_DEFAULT_INITIAL_BRANCH_NAME++../test-lib.sh++echo_without_newline(){+printf'%s'"$*"+}++echo_without_newline_nul(){+echo_without_newline"$@"|tr'\n''\0'+}++strlen(){+echo_without_newline"$1"|wc-c|sed-e's/^ *//'+}++hello_content="Hello World"+hello_size=$(strlen"$hello_content")+hello_oid=$(echo_without_newline"$hello_content"|githash-object--stdin)++# This is how we get 13:+# 13 = <file mode> + <a_space> + <file name> + <a_null>, where+# file mode is 100644, which is 6 characters;+# file name is hello, which is 5 characters+# a space is 1 character and a null is 1 character+tree_size=$(($(test_oidrawsz)+13))++commit_message="Initial commit"++# This is how we get 137:+# 137 = <tree header> + <a_space> + <a newline> ++# <Author line> + <a newline> ++# <Committer line> + <a newline> ++# <a newline> ++# <commit message length>+# An easier way to calculate is: 1. use `git cat-file commit <commit hash> | wc -c`,+# to get 177, 2. then deduct 40 hex characters to get 137+commit_size=$(($(test_oidhexsz)+137))++tag_header_without_oid="type blob+taghellotag+tagger$GIT_COMMITTER_NAME<$GIT_COMMITTER_EMAIL>"+tag_header_without_timestamp="object $hello_oid+$tag_header_without_oid"+tag_description="This is a tag"+tag_content="$tag_header_without_timestamp 0 +0000++$tag_description"++tag_oid=$(echo_without_newline"$tag_content"|githash-object-ttag--stdin-w)+tag_size=$(strlen"$tag_content")++set_transport_variables(){+hello_sha1=$(echo_without_newline"$hello_content"|githash-object--stdin)+tree_sha1=$(git-C"$1"write-tree)+commit_sha1=$(echo_without_newline"$commit_message"|git-C"$1"commit-tree$tree_sha1)+tag_sha1=$(echo_without_newline"$tag_content"|git-C"$1"hash-object-ttag--stdin-w)
I see here and various other places in this file names with "_sha1". I
think it makes more sense to name them "_oid" because these works also
fine with GIT_TEST_DEFAULT_HASH=sha256.
Other than that I don't have any comments about this patch series.
--
Toon
From: Christian Couder <hidden> Date: 2024-09-24 11:45:45
On Sat, Jul 20, 2024 at 5:43 AM Eric Ju [off-list ref] wrote:
From: Calvin Wan <redacted>
A subsequent patch needs to write capabilities for another command.
Refactor write_fetch_command_and_capabilities() to be a more general
purpose function write_command_and_capabilities(), so that it can be
used by both fetch and future command.
Here "command" means the "operations" supported by Git’s wire protocol
https://git-scm.com/docs/protocol-v2. An example would be a
git's subcommand, such as git-fetch(1); or an operation supported by
the server side such as "object-info" implemented in "a2ba162cda
(object-info: support for retrieving object info, 2021-04-20)".
I agree that reusing or refactoring the new
write_command_and_capabilities() function for more commands can be
done in a separate series that could perhaps also move the new
function to connect.c. Maybe this could be added to the commit message
though.
[...]
In https://lore.kernel.org/git/xmqqfsn0qsi4.fsf@gitster.g/ Junio
suggested swaping the "command" and "server_options" arguments as well
as sticking the "*" to "command" instead of "char", so:
static void write_command_and_capabilities(struct strbuf *req_buf,
const char *command,
const struct string_list *server_options)
The rest of the patch looks good.
Karthik suggested adding tests at this stage, but I see no tests here.
Maybe the tests are added later, but I agree with Karthik that it
would be nice to add them early if possible.
@@ -699,13 +699,17 @@ static int fetch_refs(struct transport *transport,/**Ifwereachhere,thentheserver,theclient,and/orthetransport-*helperdoesnotsupportprotocolv2.--negotiate-onlyrequires-*protocolv2.+*helperdoesnotsupportprotocolv2.--negotiate-onlyandcat-fileremote-object-info+*requireprotocolv2.*/if(data->transport_options.acked_commits){warning(_("--negotiate-only requires protocol v2"));return-1;}+if(transport->smart_options->object_info){+// fail the command explicitly to avoid further commands input
We use "/* stuff */" for one line comments instead of "// stuff". Also
the comment could go before the if (...) above and the "{" and "}"
could be dropped.
From: Christian Couder <hidden> Date: 2024-09-24 12:13:14
On Sat, Jul 20, 2024 at 5:44 AM Eric Ju [off-list ref] wrote:
+remote-object-info <remote> <object>...::
+ Print object info for object references `<object>` at specified <remote> without
+ downloading objects from remote. If the object-info capability is not
+ supported by the server, the objects will be downloaded instead.
+ Error when no object references is provided.
Maybe s/is provided/are provided/
+ This command may be combined with `--buffer`.
quoted hunk
`deltabase`::
If the object is stored as a delta on-disk, this expands to the
full hex representation of the delta base object name.
Otherwise, expands to the null OID (all zeroes). See `CAVEATS`
- below.
+ below. Not supported by `remote-object-info`.
`rest`::
If this atom is used in the output string, input lines are split
@@ -314,7 +323,9 @@ newline. The available atoms are: line) are output in place of the `%(rest)` atom. If no format is specified, the default format is `%(objectname)-%(objecttype) %(objectsize)`.+%(objecttype) %(objectsize)`, except remote-object-info command who uses
s/except remote-object-info command who uses/except for
`remote-object-info` commands which use/
quoted hunk
+`%(objectname) %(objectsize)` for now because "%(objecttype)" is not supported yet.
+When "%(objecttype)" is supported, default format should be unified.
If `--batch` is specified, or if `--batch-command` is used with the `contents`
command, the object information is followed by the object contents (consisting
@@ -396,6 +407,10 @@ scripting purposes. CAVEATS -------+Note that since objecttype, objectsize:disk and deltabase are currently not supported by the
s/objecttype, objectsize:disk and deltabase/%(objecttype),
%(objectsize:disk) and %(deltabase)/
+remote-object-info, git will error and exit when they are in the format string.
Note that the sizes of objects on disk are reported accurately, but care
should be taken in drawing conclusions about which refs or objects are
responsible for disk usage. The size of a packed non-delta object may be
[...]
+ gtransport = transport_get(remote, NULL);
+ if (gtransport->smart_options) {
+ CALLOC_ARRAY(remote_object_info, object_info_oids.nr);
+ gtransport->smart_options->object_info = 1;
+ gtransport->smart_options->object_info_oids = &object_info_oids;
+ /*
+ * 'size' is the only option currently supported.
+ * Other options that are passed in the format will exit with error.
+ */
+ if (strstr(opt->format, "%(objectsize)")) {
+ string_list_append(&object_info_options, "size");
+ } else {
+ die(_("%s is currently not supported with remote-object-info"), opt->format);
+ }
Something like the following might be a bit shorter and simpler:
/* 'objectsize' is the only option currently supported */
if (!strstr(opt->format, "%(objectsize)"))
die(_("%s is currently not supported with
remote-object-info"), opt->format);
string_list_append(&object_info_options, "size");
From: Peijian Ju <hidden> Date: 2024-09-25 18:12:59
On Tue, Sep 24, 2024 at 8:13 AM Christian Couder
[off-list ref] wrote:
On Sat, Jul 20, 2024 at 5:44 AM Eric Ju [off-list ref] wrote:
quoted
+remote-object-info <remote> <object>...::
+ Print object info for object references `<object>` at specified <remote> without
+ downloading objects from remote. If the object-info capability is not
+ supported by the server, the objects will be downloaded instead.
+ Error when no object references is provided.
Maybe s/is provided/are provided/
Thank you. Fixed in V3.
quoted
+ This command may be combined with `--buffer`.
quoted
`deltabase`::
If the object is stored as a delta on-disk, this expands to the
full hex representation of the delta base object name.
Otherwise, expands to the null OID (all zeroes). See `CAVEATS`
- below.
+ below. Not supported by `remote-object-info`.
`rest`::
If this atom is used in the output string, input lines are split
@@ -314,7 +323,9 @@ newline. The available atoms are: line) are output in place of the `%(rest)` atom. If no format is specified, the default format is `%(objectname)-%(objecttype) %(objectsize)`.+%(objecttype) %(objectsize)`, except remote-object-info command who uses
s/except remote-object-info command who uses/except for
`remote-object-info` commands which use/
Thank you. Fixed in V3.
quoted
+`%(objectname) %(objectsize)` for now because "%(objecttype)" is not supported yet.
+When "%(objecttype)" is supported, default format should be unified.
If `--batch` is specified, or if `--batch-command` is used with the `contents`
command, the object information is followed by the object contents (consisting
@@ -396,6 +407,10 @@ scripting purposes. CAVEATS -------+Note that since objecttype, objectsize:disk and deltabase are currently not supported by the
s/objecttype, objectsize:disk and deltabase/%(objecttype),
%(objectsize:disk) and %(deltabase)/
Thank you. Fixed in V3.
quoted
+remote-object-info, git will error and exit when they are in the format string.
Note that the sizes of objects on disk are reported accurately, but care
should be taken in drawing conclusions about which refs or objects are
responsible for disk usage. The size of a packed non-delta object may be
[...]
quoted
+ gtransport = transport_get(remote, NULL);
+ if (gtransport->smart_options) {
+ CALLOC_ARRAY(remote_object_info, object_info_oids.nr);
+ gtransport->smart_options->object_info = 1;
+ gtransport->smart_options->object_info_oids = &object_info_oids;
+ /*
+ * 'size' is the only option currently supported.
+ * Other options that are passed in the format will exit with error.
+ */
+ if (strstr(opt->format, "%(objectsize)")) {
+ string_list_append(&object_info_options, "size");
+ } else {
+ die(_("%s is currently not supported with remote-object-info"), opt->format);
+ }
Something like the following might be a bit shorter and simpler:
/* 'objectsize' is the only option currently supported */
if (!strstr(opt->format, "%(objectsize)"))
die(_("%s is currently not supported with
remote-object-info"), opt->format);
string_list_append(&object_info_options, "size");
@@ -0,0 +1,748 @@+#!/bin/sh++test_description='git cat-file --batch-command with remote-object-info command'++GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main+exportGIT_TEST_DEFAULT_INITIAL_BRANCH_NAME++../test-lib.sh++echo_without_newline(){+printf'%s'"$*"+}++echo_without_newline_nul(){+echo_without_newline"$@"|tr'\n''\0'+}++strlen(){+echo_without_newline"$1"|wc-c|sed-e's/^ *//'+}++hello_content="Hello World"+hello_size=$(strlen"$hello_content")+hello_oid=$(echo_without_newline"$hello_content"|githash-object--stdin)++# This is how we get 13:+# 13 = <file mode> + <a_space> + <file name> + <a_null>, where+# file mode is 100644, which is 6 characters;+# file name is hello, which is 5 characters+# a space is 1 character and a null is 1 character+tree_size=$(($(test_oidrawsz)+13))++commit_message="Initial commit"++# This is how we get 137:+# 137 = <tree header> + <a_space> + <a newline> ++# <Author line> + <a newline> ++# <Committer line> + <a newline> ++# <a newline> ++# <commit message length>+# An easier way to calculate is: 1. use `git cat-file commit <commit hash> | wc -c`,+# to get 177, 2. then deduct 40 hex characters to get 137+commit_size=$(($(test_oidhexsz)+137))++tag_header_without_oid="type blob+taghellotag+tagger$GIT_COMMITTER_NAME<$GIT_COMMITTER_EMAIL>"+tag_header_without_timestamp="object $hello_oid+$tag_header_without_oid"+tag_description="This is a tag"+tag_content="$tag_header_without_timestamp 0 +0000++$tag_description"++tag_oid=$(echo_without_newline"$tag_content"|githash-object-ttag--stdin-w)+tag_size=$(strlen"$tag_content")++set_transport_variables(){+hello_sha1=$(echo_without_newline"$hello_content"|githash-object--stdin)+tree_sha1=$(git-C"$1"write-tree)+commit_sha1=$(echo_without_newline"$commit_message"|git-C"$1"commit-tree$tree_sha1)+tag_sha1=$(echo_without_newline"$tag_content"|git-C"$1"hash-object-ttag--stdin-w)
I see here and various other places in this file names with "_sha1". I
think it makes more sense to name them "_oid" because these works also
fine with GIT_TEST_DEFAULT_HASH=sha256.
Other than that I don't have any comments about this patch series.
--
Toon
Thank you. In V3, all the variables end with "_sha1" are changed to
end with "_oid"
Karthik suggested adding tests at this stage, but I see no tests here.
Maybe the tests are added later, but I agree with Karthik that it
would be nice to add them early if possible.
Thank you. I’m not sure if there’s an easy way to directly add unit
tests for the changes in fetch-pack.c, transport-helper.c, and
transport.c, as the relevant functions are deeply nested within the
client’s call stack. Therefore, I’m attempting to test them indirectly
through git cat-file in t/t1017-cat-file-remote-object-info.sh in the
next commit.
Specifically, in t/t1017-cat-file-remote-object-info.sh:
- The code in transport-helper.c is tested in the cases
“remote-object-info fails on server with legacy protocol” and
“remote-object-info fails on server with legacy protocol fallback”.
- The code in transport.c and fetch-pack.c is tested in the cases
where transfer.advertiseobjectinfo is set to true, such as in
“batch-command remote-object-info http://”, “batch-command
remote-object-info file://”, and “batch-command remote-object-info
git://”. In these tests, we verify that remote-object-info
successfully retrieves the size from the remote without downloading
the objects locally.
@@ -699,13 +699,17 @@ static int fetch_refs(struct transport *transport,/**Ifwereachhere,thentheserver,theclient,and/orthetransport-*helperdoesnotsupportprotocolv2.--negotiate-onlyrequires-*protocolv2.+*helperdoesnotsupportprotocolv2.--negotiate-onlyandcat-fileremote-object-info+*requireprotocolv2.*/if(data->transport_options.acked_commits){warning(_("--negotiate-only requires protocol v2"));return-1;}+if(transport->smart_options->object_info){+// fail the command explicitly to avoid further commands input
We use "/* stuff */" for one line comments instead of "// stuff". Also
the comment could go before the if (...) above and the "{" and "}"
could be dropped.
From: Peijian Ju <hidden> Date: 2024-09-25 20:43:09
On Tue, Sep 24, 2024 at 7:45 AM Christian Couder
[off-list ref] wrote:
On Sat, Jul 20, 2024 at 5:43 AM Eric Ju [off-list ref] wrote:
quoted
From: Calvin Wan <redacted>
A subsequent patch needs to write capabilities for another command.
Refactor write_fetch_command_and_capabilities() to be a more general
purpose function write_command_and_capabilities(), so that it can be
used by both fetch and future command.
Here "command" means the "operations" supported by Git’s wire protocol
https://git-scm.com/docs/protocol-v2. An example would be a
git's subcommand, such as git-fetch(1); or an operation supported by
the server side such as "object-info" implemented in "a2ba162cda
(object-info: support for retrieving object info, 2021-04-20)".
I agree that reusing or refactoring the new
write_command_and_capabilities() function for more commands can be
done in a separate series that could perhaps also move the new
function to Maybe this could be added to the commit message
though.
Thank you, I am adding this to the commit message,
"In a future separate series, we can move
write_command_and_capabilities() to a higher-level file, such as
connect.c, so that it becomes accessible to other commands."
In https://lore.kernel.org/git/xmqqfsn0qsi4.fsf@gitster.g/ Junio
suggested swaping the "command" and "server_options" arguments as well
as sticking the "*" to "command" instead of "char", so:
static void write_command_and_capabilities(struct strbuf *req_buf,
const char *command,
const struct string_list *server_options)
The rest of the patch looks good.
From: Calvin Wan <redacted>
Refactor write_fetch_command_and_capabilities() to be a more general
purpose function write_command_and_capabilities(), so that it can be
used by both fetch and future command.
Here "command" means the "operations" supported by Git’s wire protocol
https://git-scm.com/docs/protocol-v2. An example would be a
git's subcommand, such as git-fetch(1); or an operation supported by
the server side such as "object-info" implemented in "a2ba162cda
(object-info: support for retrieving object info, 2021-04-20)".
In a future separate series, we can move
write_command_and_capabilities() to a higher-level file, such as
connect.c, so that it becomes accessible to other commands.
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
---
fetch-pack.c | 35 +++++++++++++++++++++++++++++------
1 file changed, 29 insertions(+), 6 deletions(-)
From: Calvin Wan <redacted>
There are some variables initialized at the start of the
do_fetch_pack_v2() state machine. Currently, they are initialized
in FETCH_CHECK_LOCAL, which is the initial state set at the beginning
of the function.
However, a subsequent patch will allow for another initial state,
while still requiring these initialized variables.
Move the initialization to be before the state machine,
so that they are set regardless of the initial state.
Note that there is no change in behavior, because we're moving code
from the beginning of the first state to just before the execution of
the state machine.
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
---
fetch-pack.c | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
@@ -1700,18 +1700,18 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,reader.me="fetch-pack";}+/* v2 supports these by default */+allow_unadvertised_object_request|=ALLOW_REACHABLE_SHA1;+use_sideband=2;+if(args->depth>0||args->deepen_since||args->deepen_not)+args->deepen=1;+while(state!=FETCH_DONE){switch(state){caseFETCH_CHECK_LOCAL:sort_ref_list(&ref,ref_compare_name);QSORT(sought,nr_sought,cmp_ref_by_name);-/* v2 supports these by default */-allow_unadvertised_object_request|=ALLOW_REACHABLE_SHA1;-use_sideband=2;-if(args->depth>0||args->deepen_since||args->deepen_not)-args->deepen=1;-/* Filter 'ref' by 'sought' and those that aren't local */mark_complete_and_common_ref(negotiator,args,&ref);filter_refs(args,&ref,sought,nr_sought);
From: Calvin Wan <redacted>
In order for a client to know what object-info components a server can
provide, advertise supported object-info features. This will allow a
client to decide whether to query the server for object-info or fetch
as a fallback.
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
---
serve.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
From: Calvin Wan <redacted>
Sometimes it is useful to get information about an object without having
to download it completely. The server logic has already been implemented
in “a2ba162cda (object-info: support for retrieving object info,
2021-04-20)”.
Add client functions to communicate with the server.
The client currently supports requesting a list of object ids with
feature 'size' from a v2 server. If a server does not
advertise the feature, then the client falls back
to making the request through 'fetch'.
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
---
fetch-pack.c | 4 +-
fetch-pack.h | 10 ++++
transport-helper.c | 8 +++-
transport.c | 116 +++++++++++++++++++++++++++++++++++++++++++--
transport.h | 11 +++++
5 files changed, 141 insertions(+), 8 deletions(-)
@@ -106,4 +114,6 @@ int report_unmatched_refs(struct ref **sought, int nr_sought);*/intfetch_pack_fsck_objects(void);+voidsend_object_info_request(intfd_out,structobject_info_args*args);+#endif
@@ -709,14 +709,18 @@ static int fetch_refs(struct transport *transport,/**Ifwereachhere,thentheserver,theclient,and/orthetransport-*helperdoesnotsupportprotocolv2.--negotiate-onlyrequires-*protocolv2.+*helperdoesnotsupportprotocolv2.--negotiate-onlyandcat-fileremote-object-info+*requireprotocolv2.*/if(data->transport_options.acked_commits){warning(_("--negotiate-only requires protocol v2"));return-1;}+/* fail the command explicitly to avoid further commands input. */+if(transport->smart_options->object_info)+die(_("remote-object-info requires protocol v2"));+if(!data->get_refs_list_called)get_refs_list_using_list(transport,0);
@@ -368,6 +368,77 @@ static struct ref *handshake(struct transport *transport, int for_push,returnrefs;}+staticintfetch_object_info(structtransport*transport,structobject_info*object_info_data)+{+intsize_index=-1;+structgit_transport_data*data=transport->data;+structobject_info_argsargs={0};+structpacket_readerreader;++args.server_options=transport->server_options;+args.object_info_options=transport->smart_options->object_info_options;+args.oids=transport->smart_options->object_info_oids;++connect_setup(transport,0);+packet_reader_init(&reader,data->fd[0],NULL,0,+PACKET_READ_CHOMP_NEWLINE|+PACKET_READ_GENTLE_ON_EOF|+PACKET_READ_DIE_ON_ERR_PACKET);+data->version=discover_version(&reader);++transport->hash_algo=reader.hash_algo;++switch(data->version){+caseprotocol_v2:+if(!server_supports_v2("object-info"))+return-1;+if(unsorted_string_list_has_string(args.object_info_options,"size")+&&!server_supports_feature("object-info","size",0))+return-1;+send_object_info_request(data->fd[1],&args);+break;+caseprotocol_v1:+caseprotocol_v0:+die(_("wrong protocol version. expected v2"));+caseprotocol_unknown_version:+BUG("unknown protocol version");+}++for(size_ti=0;i<args.object_info_options->nr;i++){+if(packet_reader_read(&reader)!=PACKET_READ_NORMAL){+check_stateless_delimiter(transport->stateless_rpc,&reader,"stateless delimiter expected");+return-1;+}+if(unsorted_string_list_has_string(args.object_info_options,reader.line)){+if(!strcmp(reader.line,"size")){+size_index=i;+for(size_tj=0;j<args.oids->nr;j++)+object_info_data[j].sizep=xcalloc(1,sizeof(long));+}+continue;+}+return-1;+}++for(size_ti=0;packet_reader_read(&reader)==PACKET_READ_NORMAL&&i<args.oids->nr;i++){+structstring_listobject_info_values=STRING_LIST_INIT_DUP;++string_list_split(&object_info_values,reader.line,' ',-1);+if(0<=size_index){+if(!strcmp(object_info_values.items[1+size_index].string,""))+die("object-info: not our ref %s",+object_info_values.items[0].string);++*object_info_data[i].sizep=strtoul(object_info_values.items[1+size_index].string,NULL,10);+}++string_list_clear(&object_info_values,0);+}+check_stateless_delimiter(transport->stateless_rpc,&reader,"stateless delimiter expected");++return0;+}+staticstructref*get_refs_via_connect(structtransport*transport,intfor_push,structtransport_ls_refs_options*options){
@@ -415,6 +486,7 @@ static int fetch_refs_via_pack(struct transport *transport,structref*refs=NULL;structfetch_pack_argsargs;structref*refs_tmp=NULL;+structref*object_info_refs=NULL;memset(&args,0,sizeof(args));args.uploadpack=data->options.uploadpack;
@@ -441,11 +513,36 @@ static int fetch_refs_via_pack(struct transport *transport,args.server_options=transport->server_options;args.negotiation_tips=data->options.negotiation_tips;args.reject_shallow_remote=transport->smart_options->reject_shallow;+args.object_info=transport->smart_options->object_info;++if(transport->smart_options+&&transport->smart_options->object_info+&&transport->smart_options->object_info_oids->nr>0){+structref*ref_itr=object_info_refs=alloc_ref("");++if(!fetch_object_info(transport,data->options.object_info_data))+gotocleanup;++args.object_info_data=data->options.object_info_data;+args.quiet=1;+args.no_progress=1;+for(size_ti=0;i<transport->smart_options->object_info_oids->nr;i++){+ref_itr->old_oid=transport->smart_options->object_info_oids->oid[i];+ref_itr->exact_oid=1;+if(i==transport->smart_options->object_info_oids->nr-1)+/* last element, no need to allocate to next */+ref_itr->next=NULL;+else+ref_itr->next=alloc_ref("");-if(!data->finished_handshake){-inti;+ref_itr=ref_itr->next;+}++transport->remote_refs=object_info_refs;++}elseif(!data->finished_handshake){intmust_list_refs=0;-for(i=0;i<nr_heads;i++){+for(inti=0;i<nr_heads;i++){if(!to_fetch[i]->exact_oid){must_list_refs=1;break;
@@ -483,23 +580,32 @@ static int fetch_refs_via_pack(struct transport *transport,&transport->pack_lockfiles,data->version);data->finished_handshake=0;+if(args.object_info){+structref*ref_cpy_reader=object_info_refs;+for(inti=0;ref_cpy_reader;i++){+oid_object_info_extended(the_repository,&ref_cpy_reader->old_oid,+&args.object_info_data[i],OBJECT_INFO_LOOKUP_REPLACE);+ref_cpy_reader=ref_cpy_reader->next;+}+}+data->options.self_contained_and_connected=args.self_contained_and_connected;data->options.connectivity_checked=args.connectivity_checked;-if(!refs)+if(!refs&&!args.object_info)ret=-1;if(report_unmatched_refs(to_fetch,nr_heads))ret=-1;cleanup:+free_refs(object_info_refs);close(data->fd[0]);if(data->fd[1]>=0)close(data->fd[1]);if(finish_connect(data->conn))ret=-1;data->conn=NULL;-free_refs(refs_tmp);free_refs(refs);list_objects_filter_release(&args.filter_options);
Some code declares variable i and only uses it
in a for loop, not in any other logic outside the loop.
Change the declaration of i to be inside the for loop for readability.
Helped-by: Christian Couder [off-list ref]
Signed-off-by: Eric Ju <redacted>
---
builtin/cat-file.c | 11 +++--------
1 file changed, 3 insertions(+), 8 deletions(-)
@@ -673,12 +673,10 @@ static void dispatch_calls(struct batch_options *opt,structqueued_cmd*cmd,intnr){-inti;-if(!opt->buffer_output)die(_("flush is only for --buffer mode"));-for(i=0;i<nr;i++)+for(size_ti=0;i<nr;i++)cmd[i].fn(opt,cmd[i].line,output,data);fflush(stdout);
From: Calvin Wan <redacted>
Since the `info` command in cat-file --batch-command prints object info
for a given object, it is natural to add another command in cat-file
--batch-command to print object info for a given object from a remote.
Add `remote-object-info` to cat-file --batch-command.
While `info` takes object ids one at a time, this creates overhead when
making requests to a server so `remote-object-info` instead can take
multiple object ids at once.
cat-file --batch-command is generally implemented in the following
manner:
- Receive and parse input from user
- Call respective function attached to command
- Get object info, print object info
In --buffer mode, this changes to:
- Receive and parse input from user
- Store respective function attached to command in a queue
- After flush, loop through commands in queue
- Call respective function attached to command
- Get object info, print object info
Notice how the getting and printing of object info is accomplished one
at a time. As described above, this creates a problem for making
requests to a server. Therefore, `remote-object-info` is implemented in
the following manner:
- Receive and parse input from user
If command is `remote-object-info`:
- Get object info from remote
- Loop through and print each object info
Else:
- Call respective function attached to command
- Parse input, get object info, print object info
And finally for --buffer mode `remote-object-info`:
- Receive and parse input from user
- Store respective function attached to command in a queue
- After flush, loop through commands in queue:
If command is `remote-object-info`:
- Get object info from remote
- Loop through and print each object info
Else:
- Call respective function attached to command
- Get object info, print object info
To summarize, `remote-object-info` gets object info from the remote and
then loop through the object info passed in, print the info.
In order for remote-object-info to avoid remote communication overhead
in the non-buffer mode, the objects are passed in as such:
remote-object-info <remote> <oid> <oid> ... <oid>
rather than
remote-object-info <remote> <oid>
remote-object-info <remote> <oid>
...
remote-object-info <remote> <oid>
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
---
Documentation/git-cat-file.txt | 22 +-
builtin/cat-file.c | 108 +++-
object-file.c | 11 +
object-store-ll.h | 3 +
t/t1017-cat-file-remote-object-info.sh | 750 +++++++++++++++++++++++++
5 files changed, 889 insertions(+), 5 deletions(-)
create mode 100755 t/t1017-cat-file-remote-object-info.sh
@@ -149,6 +149,13 @@ info <object>:: Print object info for object reference `<object>`. This corresponds to the output of `--batch-check`.+remote-object-info <remote> <object>...::+ Print object info for object references `<object>` at specified <remote> without+ downloading objects from remote. If the object-info capability is not+ supported by the server, the objects will be downloaded instead.+ Error when no object references are provided.+ This command may be combined with `--buffer`.+ flush:: Used with `--buffer` to execute all preceding commands that were issued since the beginning or since the last flush was issued. When `--buffer`
@@ -290,7 +297,8 @@ newline. The available atoms are: The full hex representation of the object name. `objecttype`::- The type of the object (the same as `cat-file -t` reports).+ The type of the object (the same as `cat-file -t` reports). See+ `CAVEATS` below. Not supported by `remote-object-info`. `objectsize`:: The size, in bytes, of the object (the same as `cat-file -s`
@@ -298,13 +306,14 @@ newline. The available atoms are: `objectsize:disk`:: The size, in bytes, that the object takes up on disk. See the- note about on-disk sizes in the `CAVEATS` section below.+ note about on-disk sizes in the `CAVEATS` section below. Not+ supported by `remote-object-info`. `deltabase`:: If the object is stored as a delta on-disk, this expands to the full hex representation of the delta base object name. Otherwise, expands to the null OID (all zeroes). See `CAVEATS`- below.+ below. Not supported by `remote-object-info`. `rest`:: If this atom is used in the output string, input lines are split
@@ -314,7 +323,9 @@ newline. The available atoms are: line) are output in place of the `%(rest)` atom. If no format is specified, the default format is `%(objectname)-%(objecttype) %(objectsize)`.+%(objecttype) %(objectsize)`, except for `remote-object-info` commands which use+`%(objectname) %(objectsize)` for now because "%(objecttype)" is not supported yet.+When "%(objecttype)" is supported, default format should be unified. If `--batch` is specified, or if `--batch-command` is used with the `contents` command, the object information is followed by the object contents (consisting
@@ -396,6 +407,9 @@ scripting purposes. CAVEATS -------+Note that since %(objecttype), %(objectsize:disk) and %(deltabase) are currently not supported by the+`remote-object-info` command, we will error and exit when they are in the format string.+ Note that the sizes of objects on disk are reported accurately, but care should be taken in drawing conclusions about which refs or objects are responsible for disk usage. The size of a packed non-delta object may be
@@ -576,6 +582,59 @@ static void batch_one_object(const char *obj_name,object_context_release(&ctx);}+staticintget_remote_info(structbatch_options*opt,intargc,constchar**argv)+{+intretval=0;+structremote*remote=NULL;+structobject_idoid;+structstring_listobject_info_options=STRING_LIST_INIT_NODUP;+staticstructtransport*gtransport;++/*+*Changetheformatto"%(objectname) %(objectsize)"when+*remote-object-infocommandisused.Oncewestartsupportingobjecttype+*thedefaultformatshouldchangetoDEFAULT_FORMAT+*/+if(!opt->format)+opt->format="%(objectname) %(objectsize)";++remote=remote_get(argv[0]);+if(!remote)+die(_("must supply valid remote when using remote-object-info"));++oid_array_clear(&object_info_oids);+for(size_ti=1;i<argc;i++){+if(get_oid_hex(argv[i],&oid))+die(_("Not a valid object name %s"),argv[i]);+oid_array_append(&object_info_oids,&oid);+}++gtransport=transport_get(remote,NULL);+if(gtransport->smart_options){+CALLOC_ARRAY(remote_object_info,object_info_oids.nr);+gtransport->smart_options->object_info=1;+gtransport->smart_options->object_info_oids=&object_info_oids;++/* 'objectsize' is the only option currently supported */+if(!strstr(opt->format,"%(objectsize)"))+die(_("%s is currently not supported with remote-object-info"),opt->format);++string_list_append(&object_info_options,"size");++if(object_info_options.nr>0){+gtransport->smart_options->object_info_options=&object_info_options;+gtransport->smart_options->object_info_data=remote_object_info;+retval=transport_fetch_refs(gtransport,NULL);+}+}else{+retval=-1;+}++string_list_clear(&object_info_options,0);+transport_disconnect(gtransport);+returnretval;+}+structobject_cb_data{structbatch_options*opt;structexpand_data*expand;
@@ -0,0 +1,750 @@+#!/bin/sh++test_description='git cat-file --batch-command with remote-object-info command'++GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main+exportGIT_TEST_DEFAULT_INITIAL_BRANCH_NAME++../test-lib.sh++echo_without_newline(){+printf'%s'"$*"+}++echo_without_newline_nul(){+echo_without_newline"$@"|tr'\n''\0'+}++strlen(){+echo_without_newline"$1"|wc-c|sed-e's/^ *//'+}++hello_content="Hello World"+hello_size=$(strlen"$hello_content")+hello_oid=$(echo_without_newline"$hello_content"|githash-object--stdin)++# This is how we get 13:+# 13 = <file mode> + <a_space> + <file name> + <a_null>, where+# file mode is 100644, which is 6 characters;+# file name is hello, which is 5 characters+# a space is 1 character and a null is 1 character+tree_size=$(($(test_oidrawsz)+13))++commit_message="Initial commit"++# This is how we get 137:+# 137 = <tree header> + <a_space> + <a newline> ++# <Author line> + <a newline> ++# <Committer line> + <a newline> ++# <a newline> ++# <commit message length>+# An easier way to calculate is: 1. use `git cat-file commit <commit hash> | wc -c`,+# to get 177, 2. then deduct 40 hex characters to get 137+commit_size=$(($(test_oidhexsz)+137))++tag_header_without_oid="type blob+taghellotag+tagger$GIT_COMMITTER_NAME<$GIT_COMMITTER_EMAIL>"+tag_header_without_timestamp="object $hello_oid+$tag_header_without_oid"+tag_description="This is a tag"+tag_content="$tag_header_without_timestamp 0 +0000++$tag_description"++tag_oid=$(echo_without_newline"$tag_content"|githash-object-ttag--stdin-w)+tag_size=$(strlen"$tag_content")++set_transport_variables(){+hello_oid=$(echo_without_newline"$hello_content"|githash-object--stdin)+tree_oid=$(git-C"$1"write-tree)+commit_oid=$(echo_without_newline"$commit_message"|git-C"$1"commit-tree$tree_oid)+tag_oid=$(echo_without_newline"$tag_content"|git-C"$1"hash-object-ttag--stdin-w)+tag_size=$(strlen"$tag_content")+}++# This section tests --batch-command with remote-object-info command+# Since "%(objecttype)" is currently not supported by the command remote-object-info ,+# the filters are set to "%(objectname) %(objectsize)" in some test cases.++# Test --batch-command remote-object-info with 'git://' transport with+# transfer.advertiseobjectinfo set to true, i.e. server has object-info capability+."$TEST_DIRECTORY"/lib-git-daemon.sh+start_git_daemon--export-all--enable=receive-pack+daemon_parent=$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent++test_expect_success'create repo to be served by git-daemon''+gitinit"$daemon_parent"&&+echo_without_newline"$hello_content">$daemon_parent/hello&&+git-C"$daemon_parent"update-index--addhello&&+git-C"$daemon_parent"configtransfer.advertiseobjectinfotrue&&+gitclone"$GIT_DAEMON_URL/parent"-n"$daemon_parent/daemon_client_empty"+'++test_expect_success'batch-command remote-object-info git://''+(+set_transport_variables"$daemon_parent"&&+cd"$daemon_parent/daemon_client_empty"&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_oid missing">>expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+remote-object-info"$GIT_DAEMON_URL/parent"$hello_oid+remote-object-info"$GIT_DAEMON_URL/parent"$tree_oid+remote-object-info"$GIT_DAEMON_URL/parent"$commit_oid+remote-object-info"$GIT_DAEMON_URL/parent"$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command remote-object-info git:// multiple sha1 per line''+(+set_transport_variables"$daemon_parent"&&+cd"$daemon_parent/daemon_client_empty"&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_oid missing">>expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+remote-object-info"$GIT_DAEMON_URL/parent"$hello_oid$tree_oid$commit_oid$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command remote-object-info git:// default filter''+(+set_transport_variables"$daemon_parent"&&+cd"$daemon_parent/daemon_client_empty"&&++echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&+GIT_TRACE_PACKET=1gitcat-file--batch-command>actual<<-EOF&&+remote-object-info"$GIT_DAEMON_URL/parent"$hello_oid$tree_oid+remote-object-info"$GIT_DAEMON_URL/parent"$commit_oid$tag_oid+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command --buffer remote-object-info git://''+(+set_transport_variables"$daemon_parent"&&+cd"$daemon_parent/daemon_client_empty"&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_oid missing">>expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)"--buffer>actual<<-EOF&&+remote-object-info"$GIT_DAEMON_URL/parent"$hello_oid$tree_oid+remote-object-info"$GIT_DAEMON_URL/parent"$commit_oid$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+flush+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command -Z remote-object-info git:// default filter''+(+set_transport_variables"$daemon_parent"&&+cd"$daemon_parent/daemon_client_empty"&&++printf"%s\0""$hello_oid$hello_size">expect&&+printf"%s\0""$tree_oid$tree_size">>expect&&+printf"%s\0""$commit_oid$commit_size">>expect&&+printf"%s\0""$tag_oid$tag_size">>expect&&++printf"%s\0""$hello_oid missing">>expect&&+printf"%s\0""$tree_oid missing">>expect&&+printf"%s\0""$commit_oid missing">>expect&&+printf"%s\0""$tag_oid missing">>expect&&++batch_input="remote-object-info $GIT_DAEMON_URL/parent $hello_oid$tree_oid+remote-object-info$GIT_DAEMON_URL/parent$commit_oid$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+" &&+echo_without_newline_nul"$batch_input">commands_null_delimited&&++gitcat-file--batch-command-Z<commands_null_delimited>actual&&+test_cmpexpectactual+)+'++# Test --batch-command remote-object-info with 'git://' and+# transfer.advertiseobjectinfo set to false, i.e. server does not have object-info capability++test_expect_success'remote-object-info fallback git://: fetch objects to client''+(+git-C"$daemon_parent"configtransfer.advertiseobjectinfofalse&&+set_transport_variables"$daemon_parent"&&+cd"$daemon_parent/daemon_client_empty"&&++# Prove object is not on the client+echo"$hello_oid missing">expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&++# These results prove remote-object-info can retrieve object info+echo"$hello_oid$hello_size">>expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results are for the info command+# They prove objects are downloaded+echo"$hello_oid$hello_size">>expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+remote-object-info$GIT_DAEMON_URL/parent$hello_oid$tree_oid$commit_oid$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+EOF++# revert server state back+git-C"$daemon_parent"configtransfer.advertiseobjectinfotrue&&++test_cmpexpectactual+)+'++stop_git_daemon++# Test --batch-command remote-object-info with 'file://' transport with+# transfer.advertiseobjectinfo set to true, i.e. server has object-info capability+# shellcheck disable=SC2016+test_expect_success'create repo to be served by file:// transport''+gitinitserver&&+git-Cserverconfigprotocol.version2&&+git-Cserverconfigtransfer.advertiseobjectinfotrue&&+echo_without_newline"$hello_content">server/hello&&+git-Cserverupdate-index--addhello&&+gitclone-n"file://$(pwd)/server"file_client_empty+'++test_expect_success'batch-command remote-object-info file://''+(+set_transport_variables"server"&&+server_path="$(pwd)/server"&&+cdfile_client_empty&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_oid missing">>expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+remote-object-info"file://${server_path}"$hello_oid+remote-object-info"file://${server_path}"$tree_oid+remote-object-info"file://${server_path}"$commit_oid+remote-object-info"file://${server_path}"$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command remote-object-info file:// multiple sha1 per line''+(+set_transport_variables"server"&&+server_path="$(pwd)/server"&&+cdfile_client_empty&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_oid missing">>expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&+++gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+remote-object-info"file://${server_path}"$hello_oid$tree_oid$commit_oid$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command --buffer remote-object-info file://''+(+set_transport_variables"server"&&+server_path="$(pwd)/server"&&+cdfile_client_empty&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_oid missing">>expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)"--buffer>actual<<-EOF&&+remote-object-info"file://${server_path}"$hello_oid$tree_oid+remote-object-info"file://${server_path}"$commit_oid$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+flush+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command remote-object-info file:// default filter''+(+set_transport_variables"server"&&+server_path="$(pwd)/server"&&+cdfile_client_empty&&++echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++gitcat-file--batch-command>actual<<-EOF&&+remote-object-info"file://${server_path}"$hello_oid$tree_oid+remote-object-info"file://${server_path}"$commit_oid$tag_oid+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command -Z remote-object-info file:// default filter''+(+set_transport_variables"server"&&+server_path="$(pwd)/server"&&+cdfile_client_empty&&++printf"%s\0""$hello_oid$hello_size">expect&&+printf"%s\0""$tree_oid$tree_size">>expect&&+printf"%s\0""$commit_oid$commit_size">>expect&&+printf"%s\0""$tag_oid$tag_size">>expect&&++printf"%s\0""$hello_oid missing">>expect&&+printf"%s\0""$tree_oid missing">>expect&&+printf"%s\0""$commit_oid missing">>expect&&+printf"%s\0""$tag_oid missing">>expect&&++batch_input="remote-object-info \"file://${server_path}\" $hello_oid$tree_oid+remote-object-info\"file://${server_path}\"$commit_oid$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+" &&+echo_without_newline_nul"$batch_input">commands_null_delimited&&++gitcat-file--batch-command-Z<commands_null_delimited>actual&&+test_cmpexpectactual+)+'++# Test --batch-command remote-object-info with 'file://' and+# transfer.advertiseobjectinfo set to false, i.e. server does not have object-info capability++test_expect_success'remote-object-info fallback file://: fetch objects to client''+(+set_transport_variables"server"&&+server_path="$(pwd)/server"&&+git-C"${server_path}"configtransfer.advertiseobjectinfofalse&&+cdfile_client_empty&&++# Prove object is not on the client+echo"$hello_oid missing">expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&++# These results prove remote-object-info can retrieve object info+echo"$hello_oid$hello_size">>expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results are for the info command+# They prove objects are downloaded+echo"$hello_oid$hello_size">>expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+remote-object-info"file://${server_path}"$hello_oid$tree_oid$commit_oid$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+EOF++# revert server state back+git-C"${server_path}"configtransfer.advertiseobjectinfotrue&&+test_cmpexpectactual+)+'++# Test --batch-command remote-object-info with 'http://' transport with+# transfer.advertiseobjectinfo set to true, i.e. server has object-info capability++."$TEST_DIRECTORY"/lib-httpd.sh+start_httpd++test_expect_success'create repo to be served by http:// transport''+gitinit"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+git-C"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"confighttp.receivepacktrue&&+git-C"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"configtransfer.advertiseobjectinfotrue&&+echo_without_newline"$hello_content">$HTTPD_DOCUMENT_ROOT_PATH/http_parent/hello&&+git-C"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"update-index--addhello&&+gitclone"$HTTPD_URL/smart/http_parent"-n"$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty"+'++test_expect_success'batch-command remote-object-info http://''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty"&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_oid missing">>expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_oid+remote-object-info"$HTTPD_URL/smart/http_parent"$tree_oid+remote-object-info"$HTTPD_URL/smart/http_parent"$commit_oid+remote-object-info"$HTTPD_URL/smart/http_parent"$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command remote-object-info http:// one line''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty"&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_oid missing">>expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_oid$tree_oid$commit_oid$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command --buffer remote-object-info http://''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty"&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_oid missing">>expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)"--buffer>actual<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_oid$tree_oid+remote-object-info"$HTTPD_URL/smart/http_parent"$commit_oid$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+flush+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command remote-object-info http:// default filter''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty"&&++echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++gitcat-file--batch-command>actual<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_oid$tree_oid+remote-object-info"$HTTPD_URL/smart/http_parent"$commit_oid$tag_oid+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command -Z remote-object-info http:// default filter''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty"&&++printf"%s\0""$hello_oid$hello_size">expect&&+printf"%s\0""$tree_oid$tree_size">>expect&&+printf"%s\0""$commit_oid$commit_size">>expect&&+printf"%s\0""$tag_oid$tag_size">>expect&&++batch_input="remote-object-info $HTTPD_URL/smart/http_parent $hello_oid$tree_oid+remote-object-info$HTTPD_URL/smart/http_parent$commit_oid$tag_oid+" &&+echo_without_newline_nul"$batch_input">commands_null_delimited&&++gitcat-file--batch-command-Z<commands_null_delimited>actual&&+test_cmpexpectactual+)+'++test_expect_success'remote-object-info fails on unspported filter option (objectsize:disk)''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&++test_must_failgitcat-file--batch-command="%(objectsize:disk)"2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_oid+EOF+test_grep"%(objectsize:disk) is currently not supported with remote-object-info"err+)+'++test_expect_success'remote-object-info fails on unspported filter option (deltabase)''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&++test_must_failgitcat-file--batch-command="%(deltabase)"2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_oid+EOF+test_grep"%(deltabase) is currently not supported with remote-object-info"err+)+'++test_expect_success'remote-object-info fails on server with legacy protocol''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&++test_must_failgit-cprotocol.version=0cat-file--batch-command="%(objectname) %(objectsize)"2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_oid+EOF+test_grep"remote-object-info requires protocol v2"err+)+'++test_expect_success'remote-object-info fails on server with legacy protocol fallback''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&++test_must_failgit-cprotocol.version=0cat-file--batch-command2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_oid+EOF+test_grep"remote-object-info requires protocol v2"err+)+'++test_expect_success'remote-object-info fails on malformed OID''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+malformed_object_id="this_id_is_not_valid"&&++test_must_failgitcat-file--batch-command="%(objectname) %(objectsize)"2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$malformed_object_id+EOF+test_grep"Not a valid object name '$malformed_object_id'"err+)+'++test_expect_success'remote-object-info fails on malformed OID fallback''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+malformed_object_id="this_id_is_not_valid"&&++test_must_failgitcat-file--batch-command2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$malformed_object_id+EOF+test_grep"Not a valid object name '$malformed_object_id'"err+)+'++test_expect_success'remote-object-info fails on missing OID''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+gitclone"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"missing_oid_repo&&+test_commit-Cmissing_oid_repomessage1c.txt&&+cdmissing_oid_repo&&++object_id=$(gitrev-parsemessage1:c.txt)&&+test_must_failgitcat-file--batch-command="%(objectname) %(objectsize)"2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$object_id+EOF+test_grep"object-info: not our ref $object_id"err+)+'++# Test --batch-command remote-object-info with 'http://' transport and+# transfer.advertiseobjectinfo set to false, i.e. server does not have object-info capability++test_expect_success'remote-object-info fallback http://: fetch objects to client''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+git-C"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"configtransfer.advertiseobjectinfofalse&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty"&&++# Prove object is not on the client+echo"$hello_oid missing">expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&++# These results prove remote-object-info can retrieve object info+echo"$hello_oid$hello_size">>expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results are for the info command+# They prove objects are downloaded+echo"$hello_oid$hello_size">>expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++gitcat-file--batch-command>actual<<-EOF&&+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_oid$tree_oid$commit_oid$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+EOF++# revert server state back+git-C"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"configtransfer.advertiseobjectinfotrue&&++test_cmpexpectactual+)+'++# DO NOT add non-httpd-specific tests here, because the last part of this+# test script is only executed when httpd is available and enabled.++test_done
From: Christian Couder <hidden> Date: 2024-10-23 09:49:57
On Thu, Sep 26, 2024 at 3:39 AM Eric Ju [off-list ref] wrote:
And finally for --buffer mode `remote-object-info`:
- Receive and parse input from user
- Store respective function attached to command in a queue
- After flush, loop through commands in queue:
If command is `remote-object-info`:
- Get object info from remote
- Loop through and print each object info
Else:
- Call respective function attached to command
- Get object info, print object info
To summarize, `remote-object-info` gets object info from the remote and
then loop through the object info passed in, print the info.
Maybe: s/print the info/printing the info/
In order for remote-object-info to avoid remote communication overhead
in the non-buffer mode, the objects are passed in as such:
remote-object-info <remote> <oid> <oid> ... <oid>
rather than
remote-object-info <remote> <oid>
remote-object-info <remote> <oid>
...
remote-object-info <remote> <oid>
[...]
If no format is specified, the default format is `%(objectname)
-%(objecttype) %(objectsize)`.
+%(objecttype) %(objectsize)`, except for `remote-object-info` commands which use
+`%(objectname) %(objectsize)` for now because "%(objecttype)" is not supported yet.
+When "%(objecttype)" is supported, default format should be unified.
I think we should warn more clearly and strongly that users should
take into account that the default format will change. So they should
better not rely on the current format in their code.
Maybe something like:
`%(objectname) %(objectsize)` for now because "%(objecttype)" is not
supported yet.
WARNING: When "%(objecttype)" is supported, default format WILL be unified, so
DO NOT RELY on the current default format to stay the same!!!
If `--batch` is specified, or if `--batch-command` is used with the `contents`
command, the object information is followed by the object contents (consisting
@@ -0,0 +1,750 @@+#!/bin/sh++test_description='git cat-file --batch-command with remote-object-info command'++GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main+exportGIT_TEST_DEFAULT_INITIAL_BRANCH_NAME++../test-lib.sh++echo_without_newline(){+printf'%s'"$*"+}++echo_without_newline_nul(){+echo_without_newline"$@"|tr'\n''\0'+}++strlen(){+echo_without_newline"$1"|wc-c|sed-e's/^ *//'+}
The above functions have been copied verbatim from t1006-cat-file.sh.
I think this is worth a comment or a TODO before these functions
saying that common code might want to be unified in the future.
Maybe something like:
# TODO: refactor these functions which were copied from
t1006-cat-file.sh into a new common file, maybe "lib-cat-file.sh"
Except the above nits and another one I found in patch 4/6, the rest
of this patch series looks good to me.
Thanks!
From: Taylor Blau <hidden> Date: 2024-10-23 20:25:24
On Wed, Oct 23, 2024 at 11:49:44AM +0200, Christian Couder wrote:
Except the above nits and another one I found in patch 4/6, the rest
of this patch series looks good to me.
Thanks for reviewing. Sounds like we are expecting another round here In
the meantime, do other reviewers have any feedback on this series?
Thanks,
Taylor
From: Peijian Ju <hidden> Date: 2024-10-24 20:28:13
On Wed, Oct 23, 2024 at 5:49 AM Christian Couder
[off-list ref] wrote:
On Thu, Sep 26, 2024 at 3:39 AM Eric Ju [off-list ref] wrote:
quoted
And finally for --buffer mode `remote-object-info`:
- Receive and parse input from user
- Store respective function attached to command in a queue
- After flush, loop through commands in queue:
If command is `remote-object-info`:
- Get object info from remote
- Loop through and print each object info
Else:
- Call respective function attached to command
- Get object info, print object info
To summarize, `remote-object-info` gets object info from the remote and
then loop through the object info passed in, print the info.
Maybe: s/print the info/printing the info/
Thank you. Fixed in V4.
quoted
In order for remote-object-info to avoid remote communication overhead
in the non-buffer mode, the objects are passed in as such:
remote-object-info <remote> <oid> <oid> ... <oid>
rather than
remote-object-info <remote> <oid>
remote-object-info <remote> <oid>
...
remote-object-info <remote> <oid>
[...]
quoted
If no format is specified, the default format is `%(objectname)
-%(objecttype) %(objectsize)`.
+%(objecttype) %(objectsize)`, except for `remote-object-info` commands which use
+`%(objectname) %(objectsize)` for now because "%(objecttype)" is not supported yet.
+When "%(objecttype)" is supported, default format should be unified.
I think we should warn more clearly and strongly that users should
take into account that the default format will change. So they should
better not rely on the current format in their code.
Maybe something like:
`%(objectname) %(objectsize)` for now because "%(objecttype)" is not
supported yet.
WARNING: When "%(objecttype)" is supported, default format WILL be unified, so
DO NOT RELY on the current default format to stay the same!!!
Thank you. The warning is added to v4.
quoted
If `--batch` is specified, or if `--batch-command` is used with the `contents`
command, the object information is followed by the object contents (consisting
@@ -0,0 +1,750 @@+#!/bin/sh++test_description='git cat-file --batch-command with remote-object-info command'++GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main+exportGIT_TEST_DEFAULT_INITIAL_BRANCH_NAME++../test-lib.sh++echo_without_newline(){+printf'%s'"$*"+}++echo_without_newline_nul(){+echo_without_newline"$@"|tr'\n''\0'+}++strlen(){+echo_without_newline"$1"|wc-c|sed-e's/^ *//'+}
The above functions have been copied verbatim from t1006-cat-file.sh.
I think this is worth a comment or a TODO before these functions
saying that common code might want to be unified in the future.
Maybe something like:
# TODO: refactor these functions which were copied from
t1006-cat-file.sh into a new common file, maybe "lib-cat-file.sh"
Thank you. I added "lib-cat-file.sh" in v4, and let both tests
t1006-cat-file.sh
and t1017-cat-file-remote-object-info.sh refer to it.
Except the above nits and another one I found in patch 4/6, the rest
of this patch series looks good to me.
Thanks!
From: Peijian Ju <hidden> Date: 2024-10-24 20:28:19
On Wed, Oct 23, 2024 at 4:25 PM Taylor Blau [off-list ref] wrote:
On Wed, Oct 23, 2024 at 11:49:44AM +0200, Christian Couder wrote:
quoted
Except the above nits and another one I found in patch 4/6, the rest
of this patch series looks good to me.
Thanks for reviewing. Sounds like we are expecting another round here In
the meantime, do other reviewers have any feedback on this series?
Thanks,
Taylor
Thank you Tayor, no other reviewers' feedback yet.
This is a continuation of Calvin Wan's (calvinwan@google.com)
patch series [PATCH v5 0/6] cat-file: add --batch-command remote-object-info command at [1].
Sometimes it is useful to get information about an object without having to download
it completely. The server logic for retrieving size has already been implemented and merged in
"a2ba162cda (object-info: support for retrieving object info, 2021-04-20)"[2].
This patch series implement the client option for it.
This patch series add the `remote-object-info` command to `cat-file --batch-command`.
This command allows the client to make an object-info command request to a server
that supports protocol v2. If the server is v2, but does not have
object-info capability, the entire object is fetched and the
relevant object info is returned.
A few questions open for discussions please:
1. In the current implementation, if a user puts `remote-object-info` in protocol v1,
`cat-file --batch-command` will die. Which way do we prefer? "error and exit (i.e. die)"
or "warn and wait for new command".
2. Right now, only the size is supported. If the batch command format
contains objectsize:disk or deltabase, it will die. The question
is about objecttype. In the current implementation, it will die too.
But dying on objecttype breaks the default format. We have changed the
default format to %(objectname) %(objectsize) when remote-object-info is used.
Any suggestions on this approach?
[1] https://lore.kernel.org/git/20220728230210.2952731-1-calvinwan@google.com/#t
[2] https://git.kernel.org/pub/scm/git/git.git/commit/?id=a2ba162cda2acc171c3e36acbbc854792b093cb7
V1 of the patch series can be found here:
https://lore.kernel.org/git/20240628190503.67389-1-eric.peijian@gmail.com/
v2 of the patch series can be found here:
https://lore.kernel.org/git/20240720034337.57125-1-eric.peijian@gmail.com/
Changes since V3
================
- Fix typos and formatting errors
- Add warning in the git-cat-file doc about the default format
- Add a new test lib file, lib-cat-file.sh. And put the shared code of
t1017-cat-file-remote-object-info.sh and t1006-cat-file.sh in it.
Thank you.
Eric Ju
Calvin Wan (5):
fetch-pack: refactor packet writing
fetch-pack: move fetch initialization
serve: advertise object-info feature
transport: add client support for object-info
cat-file: add remote-object-info to batch-command
Eric Ju (1):
cat-file: add declaration of variable i inside its for loop
Documentation/git-cat-file.txt | 24 +-
builtin/cat-file.c | 119 +++-
fetch-pack.c | 49 +-
fetch-pack.h | 10 +
object-file.c | 11 +
object-store-ll.h | 3 +
serve.c | 4 +-
t/lib-cat-file.sh | 16 +
t/t1006-cat-file.sh | 13 +-
t/t1017-cat-file-remote-object-info.sh | 739 +++++++++++++++++++++++++
transport-helper.c | 11 +-
transport.c | 115 +++-
transport.h | 11 +
13 files changed, 1081 insertions(+), 44 deletions(-)
create mode 100644 t/lib-cat-file.sh
create mode 100755 t/t1017-cat-file-remote-object-info.sh
Range-diff against v3:
1: b570dee186 = 1: 41898fe23e fetch-pack: refactor packet writing
2: e8777e8776 = 2: b3a1bee551 fetch-pack: move fetch initialization
3: d00d19cf2c = 3: d363b0f768 serve: advertise object-info feature
4: 3e1773910c ! 4: 3118061b21 transport: add client support for object-info
@@ transport-helper.c: static int fetch_refs(struct transport *transport,
*/
if (data->transport_options.acked_commits) {
warning(_("--negotiate-only requires protocol v2"));
- return -1;
+@@ transport-helper.c: static int fetch_refs(struct transport *transport,
+ free_refs(dummy);
}
+ /* fail the command explicitly to avoid further commands input. */
+ if (transport->smart_options->object_info)
+ die(_("remote-object-info requires protocol v2"));
+
- if (!data->get_refs_list_called)
- get_refs_list_using_list(transport, 0);
-
++ if (!data->get_refs_list_called)
++ get_refs_list_using_list(transport, 0);
++
+ count = 0;
+ for (i = 0; i < nr_heads; i++)
+ if (!(to_fetch[i]->status & REF_STATUS_UPTODATE))
## transport.c ##
@@ transport.c: static struct ref *handshake(struct transport *transport, int for_push,
@@ transport.c: static struct ref *handshake(struct transport *transport, int for_p
@@ transport.c: static int fetch_refs_via_pack(struct transport *transport,
struct ref *refs = NULL;
struct fetch_pack_args args;
- struct ref *refs_tmp = NULL;
+ struct ref *refs_tmp = NULL, **to_fetch_dup = NULL;
+ struct ref *object_info_refs = NULL;
memset(&args, 0, sizeof(args));
@@ transport.c: static int fetch_refs_via_pack(struct transport *transport,
+ ref_itr->exact_oid = 1;
+ if (i == transport->smart_options->object_info_oids->nr - 1)
+ /* last element, no need to allocate to next */
-+ ref_itr -> next = NULL;
++ ref_itr->next = NULL;
+ else
+ ref_itr->next = alloc_ref("");
@@ transport.c: static int fetch_refs_via_pack(struct transport *transport,
close(data->fd[0]);
if (data->fd[1] >= 0)
close(data->fd[1]);
- if (finish_connect(data->conn))
- ret = -1;
- data->conn = NULL;
--
- free_refs(refs_tmp);
- free_refs(refs);
- list_objects_filter_release(&args.filter_options);
## transport.h ##
@@
5: bb110fbc93 = 5: 2ae81acf2a cat-file: add declaration of variable i inside its for loop
6: 6dd143c164 ! 6: b5aa6c1888 cat-file: add remote-object-info to batch-command
@@ Commit message
- Get object info, print object info
To summarize, `remote-object-info` gets object info from the remote and
- then loop through the object info passed in, print the info.
+ then loop through the object info passed in, printing the info.
In order for remote-object-info to avoid remote communication overhead
in the non-buffer mode, the objects are passed in as such:
@@ Documentation/git-cat-file.txt: newline. The available atoms are:
-%(objecttype) %(objectsize)`.
+%(objecttype) %(objectsize)`, except for `remote-object-info` commands which use
+`%(objectname) %(objectsize)` for now because "%(objecttype)" is not supported yet.
-+When "%(objecttype)" is supported, default format should be unified.
++WARNING: When "%(objecttype)" is supported, the default format WILL be unified, so
++DO NOT RELY on the current the default format to stay the same!!!
If `--batch` is specified, or if `--batch-command` is used with the `contents`
command, the object information is followed by the object contents (consisting
@@ Documentation/git-cat-file.txt: scripting purposes.
CAVEATS
-------
-+Note that since %(objecttype), %(objectsize:disk) and %(deltabase) are currently not supported by the
-+`remote-object-info` command, we will error and exit when they are in the format string.
++Note that since %(objecttype), %(objectsize:disk) and %(deltabase) are
++currently not supported by the `remote-object-info` command, we will error
++and exit when they are in the format string.
+
Note that the sizes of objects on disk are reported accurately, but care
should be taken in drawing conclusions about which refs or objects are
@@ object-store-ll.h: int for_each_object_in_pack(struct packed_git *p,
+
#endif /* OBJECT_STORE_LL_H */
- ## t/t1017-cat-file-remote-object-info.sh (new) ##
+ ## t/lib-cat-file.sh (new) ##
@@
-+#!/bin/sh
-+
-+test_description='git cat-file --batch-command with remote-object-info command'
-+
-+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
-+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
-+
-+. ./test-lib.sh
++# Library of git-cat-file related functions.
+
++# Print a string without a trailing newline
+echo_without_newline () {
-+ printf '%s' "$*"
++ printf '%s' "$*"
+}
+
++# Print a string without newlines and replaces them with a NULL character (\0).
+echo_without_newline_nul () {
-+ echo_without_newline "$@" | tr '\n' '\0'
++ echo_without_newline "$@" | tr '\n' '\0'
+}
+
++# Calculate the length of a string removing any leading spaces.
+strlen () {
-+ echo_without_newline "$1" | wc -c | sed -e 's/^ *//'
++ echo_without_newline "$1" | wc -c | sed -e 's/^ *//'
+}
+
+ ## t/t1006-cat-file.sh ##
+@@ t/t1006-cat-file.sh: test_description='git cat-file'
+
+ TEST_PASSES_SANITIZE_LEAK=true
+ . ./test-lib.sh
++. "$TEST_DIRECTORY"/lib-cat-file.sh
+
+ test_cmdmode_usage () {
+ test_expect_code 129 "$@" 2>err &&
+@@ t/t1006-cat-file.sh: do
+ '
+ done
+
+-echo_without_newline () {
+- printf '%s' "$*"
+-}
+-
+-echo_without_newline_nul () {
+- echo_without_newline "$@" | tr '\n' '\0'
+-}
+-
+-strlen () {
+- echo_without_newline "$1" | wc -c | sed -e 's/^ *//'
+-}
+-
+ run_tests () {
+ type=$1
+ oid=$2
+
+ ## t/t1017-cat-file-remote-object-info.sh (new) ##
+@@
++#!/bin/sh
++
++test_description='git cat-file --batch-command with remote-object-info command'
++
++GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
++export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
++
++. ./test-lib.sh
++. "$TEST_DIRECTORY"/lib-cat-file.sh
+
+hello_content="Hello World"
+hello_size=$(strlen "$hello_content")
--
2.47.0
From: Calvin Wan <redacted>
Refactor write_fetch_command_and_capabilities() to be a more general
purpose function write_command_and_capabilities(), so that it can be
used by both fetch and future command.
Here "command" means the "operations" supported by Git’s wire protocol
https://git-scm.com/docs/protocol-v2. An example would be a
git's subcommand, such as git-fetch(1); or an operation supported by
the server side such as "object-info" implemented in "a2ba162cda
(object-info: support for retrieving object info, 2021-04-20)".
In a future separate series, we can move
write_command_and_capabilities() to a higher-level file, such as
connect.c, so that it becomes accessible to other commands.
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
---
fetch-pack.c | 35 +++++++++++++++++++++++++++++------
1 file changed, 29 insertions(+), 6 deletions(-)
From: Calvin Wan <redacted>
There are some variables initialized at the start of the
do_fetch_pack_v2() state machine. Currently, they are initialized
in FETCH_CHECK_LOCAL, which is the initial state set at the beginning
of the function.
However, a subsequent patch will allow for another initial state,
while still requiring these initialized variables.
Move the initialization to be before the state machine,
so that they are set regardless of the initial state.
Note that there is no change in behavior, because we're moving code
from the beginning of the first state to just before the execution of
the state machine.
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
---
fetch-pack.c | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
@@ -1700,18 +1700,18 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,reader.me="fetch-pack";}+/* v2 supports these by default */+allow_unadvertised_object_request|=ALLOW_REACHABLE_SHA1;+use_sideband=2;+if(args->depth>0||args->deepen_since||args->deepen_not)+args->deepen=1;+while(state!=FETCH_DONE){switch(state){caseFETCH_CHECK_LOCAL:sort_ref_list(&ref,ref_compare_name);QSORT(sought,nr_sought,cmp_ref_by_name);-/* v2 supports these by default */-allow_unadvertised_object_request|=ALLOW_REACHABLE_SHA1;-use_sideband=2;-if(args->depth>0||args->deepen_since||args->deepen_not)-args->deepen=1;-/* Filter 'ref' by 'sought' and those that aren't local */mark_complete_and_common_ref(negotiator,args,&ref);filter_refs(args,&ref,sought,nr_sought);
From: Calvin Wan <redacted>
In order for a client to know what object-info components a server can
provide, advertise supported object-info features. This will allow a
client to decide whether to query the server for object-info or fetch
as a fallback.
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
---
serve.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
From: Calvin Wan <redacted>
Sometimes it is useful to get information about an object without having
to download it completely. The server logic has already been implemented
in “a2ba162cda (object-info: support for retrieving object info,
2021-04-20)”.
Add client functions to communicate with the server.
The client currently supports requesting a list of object ids with
feature 'size' from a v2 server. If a server does not
advertise the feature, then the client falls back
to making the request through 'fetch'.
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
---
fetch-pack.c | 4 +-
fetch-pack.h | 10 ++++
transport-helper.c | 11 ++++-
transport.c | 115 +++++++++++++++++++++++++++++++++++++++++++--
transport.h | 11 +++++
5 files changed, 144 insertions(+), 7 deletions(-)
@@ -106,4 +114,6 @@ int report_unmatched_refs(struct ref **sought, int nr_sought);*/intfetch_pack_fsck_objects(void);+voidsend_object_info_request(intfd_out,structobject_info_args*args);+#endif
@@ -709,8 +709,8 @@ static int fetch_refs(struct transport *transport,/**Ifwereachhere,thentheserver,theclient,and/orthetransport-*helperdoesnotsupportprotocolv2.--negotiate-onlyrequires-*protocolv2.+*helperdoesnotsupportprotocolv2.--negotiate-onlyandcat-fileremote-object-info+*requireprotocolv2.*/if(data->transport_options.acked_commits){warning(_("--negotiate-only requires protocol v2"));
@@ -726,6 +726,13 @@ static int fetch_refs(struct transport *transport,free_refs(dummy);}+/* fail the command explicitly to avoid further commands input. */+if(transport->smart_options->object_info)+die(_("remote-object-info requires protocol v2"));++if(!data->get_refs_list_called)+get_refs_list_using_list(transport,0);+count=0;for(i=0;i<nr_heads;i++)if(!(to_fetch[i]->status&REF_STATUS_UPTODATE))
@@ -371,6 +371,77 @@ static struct ref *handshake(struct transport *transport, int for_push,returnrefs;}+staticintfetch_object_info(structtransport*transport,structobject_info*object_info_data)+{+intsize_index=-1;+structgit_transport_data*data=transport->data;+structobject_info_argsargs={0};+structpacket_readerreader;++args.server_options=transport->server_options;+args.object_info_options=transport->smart_options->object_info_options;+args.oids=transport->smart_options->object_info_oids;++connect_setup(transport,0);+packet_reader_init(&reader,data->fd[0],NULL,0,+PACKET_READ_CHOMP_NEWLINE|+PACKET_READ_GENTLE_ON_EOF|+PACKET_READ_DIE_ON_ERR_PACKET);+data->version=discover_version(&reader);++transport->hash_algo=reader.hash_algo;++switch(data->version){+caseprotocol_v2:+if(!server_supports_v2("object-info"))+return-1;+if(unsorted_string_list_has_string(args.object_info_options,"size")+&&!server_supports_feature("object-info","size",0))+return-1;+send_object_info_request(data->fd[1],&args);+break;+caseprotocol_v1:+caseprotocol_v0:+die(_("wrong protocol version. expected v2"));+caseprotocol_unknown_version:+BUG("unknown protocol version");+}++for(size_ti=0;i<args.object_info_options->nr;i++){+if(packet_reader_read(&reader)!=PACKET_READ_NORMAL){+check_stateless_delimiter(transport->stateless_rpc,&reader,"stateless delimiter expected");+return-1;+}+if(unsorted_string_list_has_string(args.object_info_options,reader.line)){+if(!strcmp(reader.line,"size")){+size_index=i;+for(size_tj=0;j<args.oids->nr;j++)+object_info_data[j].sizep=xcalloc(1,sizeof(long));+}+continue;+}+return-1;+}++for(size_ti=0;packet_reader_read(&reader)==PACKET_READ_NORMAL&&i<args.oids->nr;i++){+structstring_listobject_info_values=STRING_LIST_INIT_DUP;++string_list_split(&object_info_values,reader.line,' ',-1);+if(0<=size_index){+if(!strcmp(object_info_values.items[1+size_index].string,""))+die("object-info: not our ref %s",+object_info_values.items[0].string);++*object_info_data[i].sizep=strtoul(object_info_values.items[1+size_index].string,NULL,10);+}++string_list_clear(&object_info_values,0);+}+check_stateless_delimiter(transport->stateless_rpc,&reader,"stateless delimiter expected");++return0;+}+staticstructref*get_refs_via_connect(structtransport*transport,intfor_push,structtransport_ls_refs_options*options){
@@ -418,6 +489,7 @@ static int fetch_refs_via_pack(struct transport *transport,structref*refs=NULL;structfetch_pack_argsargs;structref*refs_tmp=NULL,**to_fetch_dup=NULL;+structref*object_info_refs=NULL;memset(&args,0,sizeof(args));args.uploadpack=data->options.uploadpack;
@@ -444,11 +516,36 @@ static int fetch_refs_via_pack(struct transport *transport,args.server_options=transport->server_options;args.negotiation_tips=data->options.negotiation_tips;args.reject_shallow_remote=transport->smart_options->reject_shallow;+args.object_info=transport->smart_options->object_info;++if(transport->smart_options+&&transport->smart_options->object_info+&&transport->smart_options->object_info_oids->nr>0){+structref*ref_itr=object_info_refs=alloc_ref("");++if(!fetch_object_info(transport,data->options.object_info_data))+gotocleanup;++args.object_info_data=data->options.object_info_data;+args.quiet=1;+args.no_progress=1;+for(size_ti=0;i<transport->smart_options->object_info_oids->nr;i++){+ref_itr->old_oid=transport->smart_options->object_info_oids->oid[i];+ref_itr->exact_oid=1;+if(i==transport->smart_options->object_info_oids->nr-1)+/* last element, no need to allocate to next */+ref_itr->next=NULL;+else+ref_itr->next=alloc_ref("");-if(!data->finished_handshake){-inti;+ref_itr=ref_itr->next;+}++transport->remote_refs=object_info_refs;++}elseif(!data->finished_handshake){intmust_list_refs=0;-for(i=0;i<nr_heads;i++){+for(inti=0;i<nr_heads;i++){if(!to_fetch[i]->exact_oid){must_list_refs=1;break;
@@ -494,16 +591,26 @@ static int fetch_refs_via_pack(struct transport *transport,&transport->pack_lockfiles,data->version);data->finished_handshake=0;+if(args.object_info){+structref*ref_cpy_reader=object_info_refs;+for(inti=0;ref_cpy_reader;i++){+oid_object_info_extended(the_repository,&ref_cpy_reader->old_oid,+&args.object_info_data[i],OBJECT_INFO_LOOKUP_REPLACE);+ref_cpy_reader=ref_cpy_reader->next;+}+}+data->options.self_contained_and_connected=args.self_contained_and_connected;data->options.connectivity_checked=args.connectivity_checked;-if(!refs)+if(!refs&&!args.object_info)ret=-1;if(report_unmatched_refs(to_fetch,nr_heads))ret=-1;cleanup:+free_refs(object_info_refs);close(data->fd[0]);if(data->fd[1]>=0)close(data->fd[1]);
Some code declares variable i and only uses it
in a for loop, not in any other logic outside the loop.
Change the declaration of i to be inside the for loop for readability.
Helped-by: Christian Couder [off-list ref]
Signed-off-by: Eric Ju <redacted>
---
builtin/cat-file.c | 11 +++--------
1 file changed, 3 insertions(+), 8 deletions(-)
@@ -673,12 +673,10 @@ static void dispatch_calls(struct batch_options *opt,structqueued_cmd*cmd,intnr){-inti;-if(!opt->buffer_output)die(_("flush is only for --buffer mode"));-for(i=0;i<nr;i++)+for(size_ti=0;i<nr;i++)cmd[i].fn(opt,cmd[i].line,output,data);fflush(stdout);
From: Calvin Wan <redacted>
Since the `info` command in cat-file --batch-command prints object info
for a given object, it is natural to add another command in cat-file
--batch-command to print object info for a given object from a remote.
Add `remote-object-info` to cat-file --batch-command.
While `info` takes object ids one at a time, this creates overhead when
making requests to a server so `remote-object-info` instead can take
multiple object ids at once.
cat-file --batch-command is generally implemented in the following
manner:
- Receive and parse input from user
- Call respective function attached to command
- Get object info, print object info
In --buffer mode, this changes to:
- Receive and parse input from user
- Store respective function attached to command in a queue
- After flush, loop through commands in queue
- Call respective function attached to command
- Get object info, print object info
Notice how the getting and printing of object info is accomplished one
at a time. As described above, this creates a problem for making
requests to a server. Therefore, `remote-object-info` is implemented in
the following manner:
- Receive and parse input from user
If command is `remote-object-info`:
- Get object info from remote
- Loop through and print each object info
Else:
- Call respective function attached to command
- Parse input, get object info, print object info
And finally for --buffer mode `remote-object-info`:
- Receive and parse input from user
- Store respective function attached to command in a queue
- After flush, loop through commands in queue:
If command is `remote-object-info`:
- Get object info from remote
- Loop through and print each object info
Else:
- Call respective function attached to command
- Get object info, print object info
To summarize, `remote-object-info` gets object info from the remote and
then loop through the object info passed in, printing the info.
In order for remote-object-info to avoid remote communication overhead
in the non-buffer mode, the objects are passed in as such:
remote-object-info <remote> <oid> <oid> ... <oid>
rather than
remote-object-info <remote> <oid>
remote-object-info <remote> <oid>
...
remote-object-info <remote> <oid>
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
---
Documentation/git-cat-file.txt | 24 +-
builtin/cat-file.c | 108 +++-
object-file.c | 11 +
object-store-ll.h | 3 +
t/lib-cat-file.sh | 16 +
t/t1006-cat-file.sh | 13 +-
t/t1017-cat-file-remote-object-info.sh | 739 +++++++++++++++++++++++++
7 files changed, 897 insertions(+), 17 deletions(-)
create mode 100644 t/lib-cat-file.sh
create mode 100755 t/t1017-cat-file-remote-object-info.sh
@@ -149,6 +149,13 @@ info <object>:: Print object info for object reference `<object>`. This corresponds to the output of `--batch-check`.+remote-object-info <remote> <object>...::+ Print object info for object references `<object>` at specified <remote> without+ downloading objects from remote. If the object-info capability is not+ supported by the server, the objects will be downloaded instead.+ Error when no object references are provided.+ This command may be combined with `--buffer`.+ flush:: Used with `--buffer` to execute all preceding commands that were issued since the beginning or since the last flush was issued. When `--buffer`
@@ -290,7 +297,8 @@ newline. The available atoms are: The full hex representation of the object name. `objecttype`::- The type of the object (the same as `cat-file -t` reports).+ The type of the object (the same as `cat-file -t` reports). See+ `CAVEATS` below. Not supported by `remote-object-info`. `objectsize`:: The size, in bytes, of the object (the same as `cat-file -s`
@@ -298,13 +306,14 @@ newline. The available atoms are: `objectsize:disk`:: The size, in bytes, that the object takes up on disk. See the- note about on-disk sizes in the `CAVEATS` section below.+ note about on-disk sizes in the `CAVEATS` section below. Not+ supported by `remote-object-info`. `deltabase`:: If the object is stored as a delta on-disk, this expands to the full hex representation of the delta base object name. Otherwise, expands to the null OID (all zeroes). See `CAVEATS`- below.+ below. Not supported by `remote-object-info`. `rest`:: If this atom is used in the output string, input lines are split
@@ -314,7 +323,10 @@ newline. The available atoms are: line) are output in place of the `%(rest)` atom. If no format is specified, the default format is `%(objectname)-%(objecttype) %(objectsize)`.+%(objecttype) %(objectsize)`, except for `remote-object-info` commands which use+`%(objectname) %(objectsize)` for now because "%(objecttype)" is not supported yet.+WARNING: When "%(objecttype)" is supported, the default format WILL be unified, so+DO NOT RELY on the current the default format to stay the same!!! If `--batch` is specified, or if `--batch-command` is used with the `contents` command, the object information is followed by the object contents (consisting
@@ -396,6 +408,10 @@ scripting purposes. CAVEATS -------+Note that since %(objecttype), %(objectsize:disk) and %(deltabase) are+currently not supported by the `remote-object-info` command, we will error+and exit when they are in the format string.+ Note that the sizes of objects on disk are reported accurately, but care should be taken in drawing conclusions about which refs or objects are responsible for disk usage. The size of a packed non-delta object may be
@@ -576,6 +582,59 @@ static void batch_one_object(const char *obj_name,object_context_release(&ctx);}+staticintget_remote_info(structbatch_options*opt,intargc,constchar**argv)+{+intretval=0;+structremote*remote=NULL;+structobject_idoid;+structstring_listobject_info_options=STRING_LIST_INIT_NODUP;+staticstructtransport*gtransport;++/*+*Changetheformatto"%(objectname) %(objectsize)"when+*remote-object-infocommandisused.Oncewestartsupportingobjecttype+*thedefaultformatshouldchangetoDEFAULT_FORMAT+*/+if(!opt->format)+opt->format="%(objectname) %(objectsize)";++remote=remote_get(argv[0]);+if(!remote)+die(_("must supply valid remote when using remote-object-info"));++oid_array_clear(&object_info_oids);+for(size_ti=1;i<argc;i++){+if(get_oid_hex(argv[i],&oid))+die(_("Not a valid object name %s"),argv[i]);+oid_array_append(&object_info_oids,&oid);+}++gtransport=transport_get(remote,NULL);+if(gtransport->smart_options){+CALLOC_ARRAY(remote_object_info,object_info_oids.nr);+gtransport->smart_options->object_info=1;+gtransport->smart_options->object_info_oids=&object_info_oids;++/* 'objectsize' is the only option currently supported */+if(!strstr(opt->format,"%(objectsize)"))+die(_("%s is currently not supported with remote-object-info"),opt->format);++string_list_append(&object_info_options,"size");++if(object_info_options.nr>0){+gtransport->smart_options->object_info_options=&object_info_options;+gtransport->smart_options->object_info_data=remote_object_info;+retval=transport_fetch_refs(gtransport,NULL);+}+}else{+retval=-1;+}++string_list_clear(&object_info_options,0);+transport_disconnect(gtransport);+returnretval;+}+structobject_cb_data{structbatch_options*opt;structexpand_data*expand;
@@ -0,0 +1,16 @@+# Library of git-cat-file related functions.++# Print a string without a trailing newline+echo_without_newline(){+printf'%s'"$*"+}++# Print a string without newlines and replaces them with a NULL character (\0).+echo_without_newline_nul(){+echo_without_newline"$@"|tr'\n''\0'+}++# Calculate the length of a string removing any leading spaces.+strlen(){+echo_without_newline"$1"|wc-c|sed-e's/^ *//'+}
@@ -0,0 +1,739 @@+#!/bin/sh++test_description='git cat-file --batch-command with remote-object-info command'++GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main+exportGIT_TEST_DEFAULT_INITIAL_BRANCH_NAME++../test-lib.sh+."$TEST_DIRECTORY"/lib-cat-file.sh++hello_content="Hello World"+hello_size=$(strlen"$hello_content")+hello_oid=$(echo_without_newline"$hello_content"|githash-object--stdin)++# This is how we get 13:+# 13 = <file mode> + <a_space> + <file name> + <a_null>, where+# file mode is 100644, which is 6 characters;+# file name is hello, which is 5 characters+# a space is 1 character and a null is 1 character+tree_size=$(($(test_oidrawsz)+13))++commit_message="Initial commit"++# This is how we get 137:+# 137 = <tree header> + <a_space> + <a newline> ++# <Author line> + <a newline> ++# <Committer line> + <a newline> ++# <a newline> ++# <commit message length>+# An easier way to calculate is: 1. use `git cat-file commit <commit hash> | wc -c`,+# to get 177, 2. then deduct 40 hex characters to get 137+commit_size=$(($(test_oidhexsz)+137))++tag_header_without_oid="type blob+taghellotag+tagger$GIT_COMMITTER_NAME<$GIT_COMMITTER_EMAIL>"+tag_header_without_timestamp="object $hello_oid+$tag_header_without_oid"+tag_description="This is a tag"+tag_content="$tag_header_without_timestamp 0 +0000++$tag_description"++tag_oid=$(echo_without_newline"$tag_content"|githash-object-ttag--stdin-w)+tag_size=$(strlen"$tag_content")++set_transport_variables(){+hello_oid=$(echo_without_newline"$hello_content"|githash-object--stdin)+tree_oid=$(git-C"$1"write-tree)+commit_oid=$(echo_without_newline"$commit_message"|git-C"$1"commit-tree$tree_oid)+tag_oid=$(echo_without_newline"$tag_content"|git-C"$1"hash-object-ttag--stdin-w)+tag_size=$(strlen"$tag_content")+}++# This section tests --batch-command with remote-object-info command+# Since "%(objecttype)" is currently not supported by the command remote-object-info ,+# the filters are set to "%(objectname) %(objectsize)" in some test cases.++# Test --batch-command remote-object-info with 'git://' transport with+# transfer.advertiseobjectinfo set to true, i.e. server has object-info capability+."$TEST_DIRECTORY"/lib-git-daemon.sh+start_git_daemon--export-all--enable=receive-pack+daemon_parent=$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent++test_expect_success'create repo to be served by git-daemon''+gitinit"$daemon_parent"&&+echo_without_newline"$hello_content">$daemon_parent/hello&&+git-C"$daemon_parent"update-index--addhello&&+git-C"$daemon_parent"configtransfer.advertiseobjectinfotrue&&+gitclone"$GIT_DAEMON_URL/parent"-n"$daemon_parent/daemon_client_empty"+'++test_expect_success'batch-command remote-object-info git://''+(+set_transport_variables"$daemon_parent"&&+cd"$daemon_parent/daemon_client_empty"&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_oid missing">>expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+remote-object-info"$GIT_DAEMON_URL/parent"$hello_oid+remote-object-info"$GIT_DAEMON_URL/parent"$tree_oid+remote-object-info"$GIT_DAEMON_URL/parent"$commit_oid+remote-object-info"$GIT_DAEMON_URL/parent"$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command remote-object-info git:// multiple sha1 per line''+(+set_transport_variables"$daemon_parent"&&+cd"$daemon_parent/daemon_client_empty"&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_oid missing">>expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+remote-object-info"$GIT_DAEMON_URL/parent"$hello_oid$tree_oid$commit_oid$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command remote-object-info git:// default filter''+(+set_transport_variables"$daemon_parent"&&+cd"$daemon_parent/daemon_client_empty"&&++echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&+GIT_TRACE_PACKET=1gitcat-file--batch-command>actual<<-EOF&&+remote-object-info"$GIT_DAEMON_URL/parent"$hello_oid$tree_oid+remote-object-info"$GIT_DAEMON_URL/parent"$commit_oid$tag_oid+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command --buffer remote-object-info git://''+(+set_transport_variables"$daemon_parent"&&+cd"$daemon_parent/daemon_client_empty"&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_oid missing">>expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)"--buffer>actual<<-EOF&&+remote-object-info"$GIT_DAEMON_URL/parent"$hello_oid$tree_oid+remote-object-info"$GIT_DAEMON_URL/parent"$commit_oid$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+flush+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command -Z remote-object-info git:// default filter''+(+set_transport_variables"$daemon_parent"&&+cd"$daemon_parent/daemon_client_empty"&&++printf"%s\0""$hello_oid$hello_size">expect&&+printf"%s\0""$tree_oid$tree_size">>expect&&+printf"%s\0""$commit_oid$commit_size">>expect&&+printf"%s\0""$tag_oid$tag_size">>expect&&++printf"%s\0""$hello_oid missing">>expect&&+printf"%s\0""$tree_oid missing">>expect&&+printf"%s\0""$commit_oid missing">>expect&&+printf"%s\0""$tag_oid missing">>expect&&++batch_input="remote-object-info $GIT_DAEMON_URL/parent $hello_oid$tree_oid+remote-object-info$GIT_DAEMON_URL/parent$commit_oid$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+" &&+echo_without_newline_nul"$batch_input">commands_null_delimited&&++gitcat-file--batch-command-Z<commands_null_delimited>actual&&+test_cmpexpectactual+)+'++# Test --batch-command remote-object-info with 'git://' and+# transfer.advertiseobjectinfo set to false, i.e. server does not have object-info capability++test_expect_success'remote-object-info fallback git://: fetch objects to client''+(+git-C"$daemon_parent"configtransfer.advertiseobjectinfofalse&&+set_transport_variables"$daemon_parent"&&+cd"$daemon_parent/daemon_client_empty"&&++# Prove object is not on the client+echo"$hello_oid missing">expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&++# These results prove remote-object-info can retrieve object info+echo"$hello_oid$hello_size">>expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results are for the info command+# They prove objects are downloaded+echo"$hello_oid$hello_size">>expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+remote-object-info$GIT_DAEMON_URL/parent$hello_oid$tree_oid$commit_oid$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+EOF++# revert server state back+git-C"$daemon_parent"configtransfer.advertiseobjectinfotrue&&++test_cmpexpectactual+)+'++stop_git_daemon++# Test --batch-command remote-object-info with 'file://' transport with+# transfer.advertiseobjectinfo set to true, i.e. server has object-info capability+# shellcheck disable=SC2016+test_expect_success'create repo to be served by file:// transport''+gitinitserver&&+git-Cserverconfigprotocol.version2&&+git-Cserverconfigtransfer.advertiseobjectinfotrue&&+echo_without_newline"$hello_content">server/hello&&+git-Cserverupdate-index--addhello&&+gitclone-n"file://$(pwd)/server"file_client_empty+'++test_expect_success'batch-command remote-object-info file://''+(+set_transport_variables"server"&&+server_path="$(pwd)/server"&&+cdfile_client_empty&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_oid missing">>expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+remote-object-info"file://${server_path}"$hello_oid+remote-object-info"file://${server_path}"$tree_oid+remote-object-info"file://${server_path}"$commit_oid+remote-object-info"file://${server_path}"$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command remote-object-info file:// multiple sha1 per line''+(+set_transport_variables"server"&&+server_path="$(pwd)/server"&&+cdfile_client_empty&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_oid missing">>expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&+++gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+remote-object-info"file://${server_path}"$hello_oid$tree_oid$commit_oid$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command --buffer remote-object-info file://''+(+set_transport_variables"server"&&+server_path="$(pwd)/server"&&+cdfile_client_empty&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_oid missing">>expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)"--buffer>actual<<-EOF&&+remote-object-info"file://${server_path}"$hello_oid$tree_oid+remote-object-info"file://${server_path}"$commit_oid$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+flush+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command remote-object-info file:// default filter''+(+set_transport_variables"server"&&+server_path="$(pwd)/server"&&+cdfile_client_empty&&++echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++gitcat-file--batch-command>actual<<-EOF&&+remote-object-info"file://${server_path}"$hello_oid$tree_oid+remote-object-info"file://${server_path}"$commit_oid$tag_oid+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command -Z remote-object-info file:// default filter''+(+set_transport_variables"server"&&+server_path="$(pwd)/server"&&+cdfile_client_empty&&++printf"%s\0""$hello_oid$hello_size">expect&&+printf"%s\0""$tree_oid$tree_size">>expect&&+printf"%s\0""$commit_oid$commit_size">>expect&&+printf"%s\0""$tag_oid$tag_size">>expect&&++printf"%s\0""$hello_oid missing">>expect&&+printf"%s\0""$tree_oid missing">>expect&&+printf"%s\0""$commit_oid missing">>expect&&+printf"%s\0""$tag_oid missing">>expect&&++batch_input="remote-object-info \"file://${server_path}\" $hello_oid$tree_oid+remote-object-info\"file://${server_path}\"$commit_oid$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+" &&+echo_without_newline_nul"$batch_input">commands_null_delimited&&++gitcat-file--batch-command-Z<commands_null_delimited>actual&&+test_cmpexpectactual+)+'++# Test --batch-command remote-object-info with 'file://' and+# transfer.advertiseobjectinfo set to false, i.e. server does not have object-info capability++test_expect_success'remote-object-info fallback file://: fetch objects to client''+(+set_transport_variables"server"&&+server_path="$(pwd)/server"&&+git-C"${server_path}"configtransfer.advertiseobjectinfofalse&&+cdfile_client_empty&&++# Prove object is not on the client+echo"$hello_oid missing">expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&++# These results prove remote-object-info can retrieve object info+echo"$hello_oid$hello_size">>expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results are for the info command+# They prove objects are downloaded+echo"$hello_oid$hello_size">>expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+remote-object-info"file://${server_path}"$hello_oid$tree_oid$commit_oid$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+EOF++# revert server state back+git-C"${server_path}"configtransfer.advertiseobjectinfotrue&&+test_cmpexpectactual+)+'++# Test --batch-command remote-object-info with 'http://' transport with+# transfer.advertiseobjectinfo set to true, i.e. server has object-info capability++."$TEST_DIRECTORY"/lib-httpd.sh+start_httpd++test_expect_success'create repo to be served by http:// transport''+gitinit"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+git-C"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"confighttp.receivepacktrue&&+git-C"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"configtransfer.advertiseobjectinfotrue&&+echo_without_newline"$hello_content">$HTTPD_DOCUMENT_ROOT_PATH/http_parent/hello&&+git-C"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"update-index--addhello&&+gitclone"$HTTPD_URL/smart/http_parent"-n"$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty"+'++test_expect_success'batch-command remote-object-info http://''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty"&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_oid missing">>expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_oid+remote-object-info"$HTTPD_URL/smart/http_parent"$tree_oid+remote-object-info"$HTTPD_URL/smart/http_parent"$commit_oid+remote-object-info"$HTTPD_URL/smart/http_parent"$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command remote-object-info http:// one line''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty"&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_oid missing">>expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_oid$tree_oid$commit_oid$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command --buffer remote-object-info http://''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty"&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_oid missing">>expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)"--buffer>actual<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_oid$tree_oid+remote-object-info"$HTTPD_URL/smart/http_parent"$commit_oid$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+flush+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command remote-object-info http:// default filter''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty"&&++echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++gitcat-file--batch-command>actual<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_oid$tree_oid+remote-object-info"$HTTPD_URL/smart/http_parent"$commit_oid$tag_oid+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command -Z remote-object-info http:// default filter''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty"&&++printf"%s\0""$hello_oid$hello_size">expect&&+printf"%s\0""$tree_oid$tree_size">>expect&&+printf"%s\0""$commit_oid$commit_size">>expect&&+printf"%s\0""$tag_oid$tag_size">>expect&&++batch_input="remote-object-info $HTTPD_URL/smart/http_parent $hello_oid$tree_oid+remote-object-info$HTTPD_URL/smart/http_parent$commit_oid$tag_oid+" &&+echo_without_newline_nul"$batch_input">commands_null_delimited&&++gitcat-file--batch-command-Z<commands_null_delimited>actual&&+test_cmpexpectactual+)+'++test_expect_success'remote-object-info fails on unspported filter option (objectsize:disk)''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&++test_must_failgitcat-file--batch-command="%(objectsize:disk)"2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_oid+EOF+test_grep"%(objectsize:disk) is currently not supported with remote-object-info"err+)+'++test_expect_success'remote-object-info fails on unspported filter option (deltabase)''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&++test_must_failgitcat-file--batch-command="%(deltabase)"2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_oid+EOF+test_grep"%(deltabase) is currently not supported with remote-object-info"err+)+'++test_expect_success'remote-object-info fails on server with legacy protocol''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&++test_must_failgit-cprotocol.version=0cat-file--batch-command="%(objectname) %(objectsize)"2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_oid+EOF+test_grep"remote-object-info requires protocol v2"err+)+'++test_expect_success'remote-object-info fails on server with legacy protocol fallback''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&++test_must_failgit-cprotocol.version=0cat-file--batch-command2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_oid+EOF+test_grep"remote-object-info requires protocol v2"err+)+'++test_expect_success'remote-object-info fails on malformed OID''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+malformed_object_id="this_id_is_not_valid"&&++test_must_failgitcat-file--batch-command="%(objectname) %(objectsize)"2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$malformed_object_id+EOF+test_grep"Not a valid object name '$malformed_object_id'"err+)+'++test_expect_success'remote-object-info fails on malformed OID fallback''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+malformed_object_id="this_id_is_not_valid"&&++test_must_failgitcat-file--batch-command2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$malformed_object_id+EOF+test_grep"Not a valid object name '$malformed_object_id'"err+)+'++test_expect_success'remote-object-info fails on missing OID''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+gitclone"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"missing_oid_repo&&+test_commit-Cmissing_oid_repomessage1c.txt&&+cdmissing_oid_repo&&++object_id=$(gitrev-parsemessage1:c.txt)&&+test_must_failgitcat-file--batch-command="%(objectname) %(objectsize)"2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$object_id+EOF+test_grep"object-info: not our ref $object_id"err+)+'++# Test --batch-command remote-object-info with 'http://' transport and+# transfer.advertiseobjectinfo set to false, i.e. server does not have object-info capability++test_expect_success'remote-object-info fallback http://: fetch objects to client''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+git-C"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"configtransfer.advertiseobjectinfofalse&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty"&&++# Prove object is not on the client+echo"$hello_oid missing">expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&++# These results prove remote-object-info can retrieve object info+echo"$hello_oid$hello_size">>expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results are for the info command+# They prove objects are downloaded+echo"$hello_oid$hello_size">>expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++gitcat-file--batch-command>actual<<-EOF&&+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_oid$tree_oid$commit_oid$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+EOF++# revert server state back+git-C"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"configtransfer.advertiseobjectinfotrue&&++test_cmpexpectactual+)+'++# DO NOT add non-httpd-specific tests here, because the last part of this+# test script is only executed when httpd is available and enabled.++test_done
@@ -709,8 +709,8 @@ static int fetch_refs(struct transport *transport,/**Ifwereachhere,thentheserver,theclient,and/orthetransport-*helperdoesnotsupportprotocolv2.--negotiate-onlyrequires-*protocolv2.+*helperdoesnotsupportprotocolv2.--negotiate-onlyandcat-fileremote-object-info
+ for (size_t i = 0; i < transport->smart_options->object_info_oids->nr; i++) {
+ ref_itr->old_oid = transport->smart_options->object_info_oids->oid[i];
+ ref_itr->exact_oid = 1;
+ if (i == transport->smart_options->object_info_oids->nr - 1)
+ /* last element, no need to allocate to next */
+ ref_itr->next = NULL;
+ else
+ ref_itr->next = alloc_ref("");
- if (!data->finished_handshake) {
- int i;
+ ref_itr = ref_itr->next;
+ }
+
+ transport->remote_refs = object_info_refs;
+
+ } else if (!data->finished_handshake) {
int must_list_refs = 0;
- for (i = 0; i < nr_heads; i++) {
+ for (int i = 0; i < nr_heads; i++) {
if (!to_fetch[i]->exact_oid) {
must_list_refs = 1;
break;
@@ -494,16 +591,26 @@ static int fetch_refs_via_pack(struct transport *transport, &transport->pack_lockfiles, data->version); data->finished_handshake = 0;+ if (args.object_info) {+ struct ref *ref_cpy_reader = object_info_refs;+ for (int i = 0; ref_cpy_reader; i++) {+ oid_object_info_extended(the_repository, &ref_cpy_reader->old_oid,+ &args.object_info_data[i], OBJECT_INFO_LOOKUP_REPLACE);+ ref_cpy_reader = ref_cpy_reader->next;+ }+ }+ data->options.self_contained_and_connected = args.self_contained_and_connected; data->options.connectivity_checked = args.connectivity_checked;- if (!refs)+ if (!refs && !args.object_info) ret = -1;
This is because, now we don't necessary always fetch the refs, since
sometimes we're just happy fetching the object info. Would be nice to
have a comment here.
if (report_unmatched_refs(to_fetch, nr_heads))
ret = -1;
cleanup:
+ free_refs(object_info_refs);
close(data->fd[0]);
if (data->fd[1] >= 0)
close(data->fd[1]);
@@ -314,7 +323,10 @@ newline. The available atoms are: line) are output in place of the `%(rest)` atom. If no format is specified, the default format is `%(objectname)-%(objecttype) %(objectsize)`.+%(objecttype) %(objectsize)`, except for `remote-object-info` commands which use+`%(objectname) %(objectsize)` for now because "%(objecttype)" is not supported yet.+WARNING: When "%(objecttype)" is supported, the default format WILL be unified, so+DO NOT RELY on the current the default format to stay the same!!!
This seems like a planned breakage, wouldn't it make more sense to
implement %(objecttype) first?
From: Christian Couder <hidden> Date: 2024-10-25 13:55:45
Hi Karthik,
On Fri, Oct 25, 2024 at 12:53 PM karthik nayak [off-list ref] wrote:
Eric Ju [off-list ref] writes:
[snip]
quoted
@@ -314,7 +323,10 @@ newline. The available atoms are: line) are output in place of the `%(rest)` atom. If no format is specified, the default format is `%(objectname)-%(objecttype) %(objectsize)`.+%(objecttype) %(objectsize)`, except for `remote-object-info` commands which use+`%(objectname) %(objectsize)` for now because "%(objecttype)" is not supported yet.+WARNING: When "%(objecttype)" is supported, the default format WILL be unified, so+DO NOT RELY on the current the default format to stay the same!!!
This seems like a planned breakage, wouldn't it make more sense to
implement %(objecttype) first?
I don't think it's fair to say it's a planned breakage. For example if
a default specifies what is displayed on the command line and if
what's displayed has an informational purpose there, then changing the
default to add more information is not really a breakage. Here we make
it clear that a possible breakage (in case the feature wasn't used for
informational purposes only for example) could easily be avoided by
not relying on the default, but instead specifying exactly the desired
output format.
And yeah ideally both %(objecttype) and %(objectsize) should be
implemented first, but on the other hand sending short patch series
and growing some features step by step is a nicer approach in some
ways than sending big patch series that are hard to review. So I think
it's fair here to start with just %(objectsize) and leave
%(objecttype) for a future patch series.
Except for this I agree with the comments in your review of the v4.
Thanks!
From: Peijian Ju <hidden> Date: 2024-10-25 16:06:54
On Fri, Oct 25, 2024 at 5:52 AM karthik nayak [off-list ref] wrote:
Eric Ju [off-list ref] writes:
[snip]
quoted
+
+void send_object_info_request(int fd_out, struct object_info_args *args)
+{
+ struct strbuf req_buf = STRBUF_INIT;
+
+ write_command_and_capabilities(&req_buf, "object-info", args->server_options);
+
+ if (unsorted_string_list_has_string(args->object_info_options, "size"))
+ packet_buf_write(&req_buf, "size");
+
+ if (args->oids) {
+ for (size_t i = 0; i < args->oids->nr; i++)
+ packet_buf_write(&req_buf, "oid %s", oid_to_hex(&args->oids->oid[i]));
+ }
+
+ packet_buf_flush(&req_buf);
+ if (write_in_full(fd_out, req_buf.buf, req_buf.len) < 0)
+ die_errno(_("unable to write request to remote"));
+
+ strbuf_release(&req_buf);
+}
+
Was this function meant to be added here? I mean, there is no reference
to it in the commit message or anywhere else.
Thank you.
The `send_object_info_request` function is used in `transport.c`
`fetch_object_info()` in patch 4/6. Its functionality is similar to
`send_fetch_request()`: sending the object-info command along with
sub-command (e.g. size) and arguments (e.g. oids) to the remote.
I guess Clavin put it here because
1. it has similar functionality as `send_fetch_request()`
2. `write_command_and_capabilities()` is only visible within `fecth-pack.c`.
However, I believe your comment is valid. Adding everything to
`fetch-pack.c` makes the file overly bloated with functionality
unrelated to fetch-pack. For v5, I plan to address this by:
I will:
1. move `write_command_and_capabilities()` a level up to connect.c
2. add a new file f`ecth-object-info.c` at the same level of
`fetch-pack.c`. This new file contains the logic related to
object-info command, i.e. `send_object_info_request()` and
`fetch_object_info()`
3. move `fetch_object_info()` away from `transport.c`
The dependency WAS like this, as I pointed out in a previous reply at
https://lore.kernel.org/git/CAN2LT1AM5rYpwjZ+rhYerxDkL6mbxr7iDc=wvuhvNKS8VVXQ8w@mail.gmail.com/#t
`transport.c` -> `fetch-pack.c` -> `connect.c`, where "->" means
"depends on".
In v5, it would be like this:
`transport.c` -> `fetch-pack.c` -> `connect.c`
| /
-> fecth-object-info.c ->
Let me know if that makes sense.
From: Taylor Blau <hidden> Date: 2024-10-25 20:56:36
On Thu, Oct 24, 2024 at 04:53:53PM -0400, Eric Ju wrote:
Calvin Wan (5):
fetch-pack: refactor packet writing
fetch-pack: move fetch initialization
serve: advertise object-info feature
transport: add client support for object-info
cat-file: add remote-object-info to batch-command
Eric Ju (1):
cat-file: add declaration of variable i inside its for loop
Thanks. I just want to make sure that I have the right base here... this
was previously based on 3857aae53f (Git 2.47-rc0, 2024-09-25), but
applying the new round did not cleanly apply on top of that commit as
its merge base.
I applied the new round on top of the current tip of 'master', which is
6a11438f43 (The fifth batch, 2024-10-25) at the time of writing.
Please let me know if that was the right choice to make ;-).
Thanks,
Taylor
From: Peijian Ju <hidden> Date: 2024-10-27 03:54:15
On Fri, Oct 25, 2024 at 4:56 PM Taylor Blau [off-list ref] wrote:
On Thu, Oct 24, 2024 at 04:53:53PM -0400, Eric Ju wrote:
quoted
Calvin Wan (5):
fetch-pack: refactor packet writing
fetch-pack: move fetch initialization
serve: advertise object-info feature
transport: add client support for object-info
cat-file: add remote-object-info to batch-command
Eric Ju (1):
cat-file: add declaration of variable i inside its for loop
Thanks. I just want to make sure that I have the right base here... this
was previously based on 3857aae53f (Git 2.47-rc0, 2024-09-25), but
applying the new round did not cleanly apply on top of that commit as
its merge base.
I applied the new round on top of the current tip of 'master', which is
6a11438f43 (The fifth batch, 2024-10-25) at the time of writing.
Please let me know if that was the right choice to make ;-).
Thanks,
Taylor
Hi Taylor,
I probably rebase on the wrong master tip. I am working on a new v5 now.
Would you like to resend v4 or can we skip v4 and use v5?
Thank you.
Eric
From: Taylor Blau <hidden> Date: 2024-10-28 00:01:12
On Sat, Oct 26, 2024 at 11:54:03PM -0400, Peijian Ju wrote:
I probably rebase on the wrong master tip. I am working on a new v5 now.
Would you like to resend v4 or can we skip v4 and use v5?
Let's skip resending this round since I found a suitable base for it.
Once you send v5, I can apply it onto any base that you specify as
appropriate.
Thanks,
Taylor
@@ -709,8 +709,8 @@ static int fetch_refs(struct transport *transport,/**Ifwereachhere,thentheserver,theclient,and/orthetransport-*helperdoesnotsupportprotocolv2.--negotiate-onlyrequires-*protocolv2.+*helperdoesnotsupportprotocolv2.--negotiate-onlyandcat-fileremote-object-info
Thank you. If the code reaches here, it means we fall back to
downloading the pack file with fetch_pack().
Setting quiet and no_progress just wants to make fetch_pack less
verbose and do its job quietly in the background.
It is like calling `git fetch-pack -q ...`. I see setting quiet and
no_progress is necessary here because:
1. If the call git fetch-pack is from an internal command, we would
better keep the call lean and efficient.
2. If the user wants to do a verbose call, they have the choice to
call git fetch-pack directly from the client.
I add a comment in v5 to explain it, like this:
" we can't retrieve object info in packets, so we will fall back to
downland pack files. We set quiet and no_progress to be true, so that
the internal call of fetch-pack is less verbose."
quoted
+ for (size_t i = 0; i < transport->smart_options->object_info_oids->nr; i++) {
+ ref_itr->old_oid = transport->smart_options->object_info_oids->oid[i];
+ ref_itr->exact_oid = 1;
+ if (i == transport->smart_options->object_info_oids->nr - 1)
+ /* last element, no need to allocate to next */
+ ref_itr->next = NULL;
+ else
+ ref_itr->next = alloc_ref("");
- if (!data->finished_handshake) {
- int i;
+ ref_itr = ref_itr->next;
+ }
+
+ transport->remote_refs = object_info_refs;
+
+ } else if (!data->finished_handshake) {
int must_list_refs = 0;
- for (i = 0; i < nr_heads; i++) {
+ for (int i = 0; i < nr_heads; i++) {
if (!to_fetch[i]->exact_oid) {
must_list_refs = 1;
break;
@@ -494,16 +591,26 @@ static int fetch_refs_via_pack(struct transport *transport, &transport->pack_lockfiles, data->version); data->finished_handshake = 0;+ if (args.object_info) {+ struct ref *ref_cpy_reader = object_info_refs;+ for (int i = 0; ref_cpy_reader; i++) {+ oid_object_info_extended(the_repository, &ref_cpy_reader->old_oid,+ &args.object_info_data[i], OBJECT_INFO_LOOKUP_REPLACE);+ ref_cpy_reader = ref_cpy_reader->next;+ }+ }+ data->options.self_contained_and_connected = args.self_contained_and_connected; data->options.connectivity_checked = args.connectivity_checked;- if (!refs)+ if (!refs && !args.object_info) ret = -1;
This is because, now we don't necessary always fetch the refs, since
sometimes we're just happy fetching the object info. Would be nice to
have a comment here.
Thank you. Acutally, if the code reaches here, it means we fall back
to downloading the pack file.
I would expect there is no difference from the old logic, so
`!args.object_info` might not be needed here.
I am removing `!args.object_info` in v5.
quoted
if (report_unmatched_refs(to_fetch, nr_heads))
ret = -1;
cleanup:
+ free_refs(object_info_refs);
close(data->fd[0]);
if (data->fd[1] >= 0)
close(data->fd[1]);
From: Calvin Wan <redacted>
Refactor write_fetch_command_and_capabilities() to be a more general
purpose function write_command_and_capabilities(), so that it can be
used by both fetch and future commands.
Here "command" means the "operations" supported by Git’s wire protocol
https://git-scm.com/docs/protocol-v2. An example would be a
git's subcommand, such as git-fetch(1); or an operation supported by
the server side such as "object-info" implemented in "a2ba162cda
(object-info: support for retrieving object info, 2021-04-20)".
The new write_command_and_capabilities() function is also moved to
connect.c, so that it becomes accessible to other commands.
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
---
connect.c | 34 ++++++++++++++++++++++++++++++++++
connect.h | 4 ++++
fetch-pack.c | 36 ++----------------------------------
3 files changed, 40 insertions(+), 34 deletions(-)
@@ -688,6 +688,40 @@ int server_supports(const char *feature)return!!server_feature_value(feature,NULL);}+voidwrite_command_and_capabilities(structstrbuf*req_buf,constchar*command,+conststructstring_list*server_options)+{+constchar*hash_name;+intadvertise_sid;++git_config_get_bool("transfer.advertisesid",&advertise_sid);++ensure_server_supports_v2(command);+packet_buf_write(req_buf,"command=%s",command);+if(server_supports_v2("agent"))+packet_buf_write(req_buf,"agent=%s",git_user_agent_sanitized());+if(advertise_sid&&server_supports_v2("session-id"))+packet_buf_write(req_buf,"session-id=%s",trace2_session_id());+if(server_options&&server_options->nr){+ensure_server_supports_v2("server-option");+for(inti=0;i<server_options->nr;i++)+packet_buf_write(req_buf,"server-option=%s",+server_options->items[i].string);+}++if(server_feature_v2("object-format",&hash_name)){+constinthash_algo=hash_algo_by_name(hash_name);+if(hash_algo_by_ptr(the_hash_algo)!=hash_algo)+die(_("mismatched algorithms: client %s; server %s"),+the_hash_algo->name,hash_name);+packet_buf_write(req_buf,"object-format=%s",the_hash_algo->name);+}elseif(hash_algo_by_ptr(the_hash_algo)!=GIT_HASH_SHA1){+die(_("the server does not support algorithm '%s'"),+the_hash_algo->name);+}+packet_buf_delim(req_buf);+}+enumprotocol{PROTO_LOCAL=1,PROTO_FILE,
@@ -1314,38 +1314,6 @@ static int add_haves(struct fetch_negotiator *negotiator,returnhaves_added;}-staticvoidwrite_fetch_command_and_capabilities(structstrbuf*req_buf,-conststructstring_list*server_options)-{-constchar*hash_name;--ensure_server_supports_v2("fetch");-packet_buf_write(req_buf,"command=fetch");-if(server_supports_v2("agent"))-packet_buf_write(req_buf,"agent=%s",git_user_agent_sanitized());-if(advertise_sid&&server_supports_v2("session-id"))-packet_buf_write(req_buf,"session-id=%s",trace2_session_id());-if(server_options&&server_options->nr){-inti;-ensure_server_supports_v2("server-option");-for(i=0;i<server_options->nr;i++)-packet_buf_write(req_buf,"server-option=%s",-server_options->items[i].string);-}--if(server_feature_v2("object-format",&hash_name)){-inthash_algo=hash_algo_by_name(hash_name);-if(hash_algo_by_ptr(the_hash_algo)!=hash_algo)-die(_("mismatched algorithms: client %s; server %s"),-the_hash_algo->name,hash_name);-packet_buf_write(req_buf,"object-format=%s",the_hash_algo->name);-}elseif(hash_algo_by_ptr(the_hash_algo)!=GIT_HASH_SHA1){-die(_("the server does not support algorithm '%s'"),-the_hash_algo->name);-}-packet_buf_delim(req_buf);-}-staticintsend_fetch_request(structfetch_negotiator*negotiator,intfd_out,structfetch_pack_args*args,conststructref*wants,structoidset*common,
@@ -1356,7 +1324,7 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,intdone_sent=0;structstrbufreq_buf=STRBUF_INIT;-write_fetch_command_and_capabilities(&req_buf,args->server_options);+write_command_and_capabilities(&req_buf,"fetch",args->server_options);if(args->use_thin_pack)packet_buf_write(&req_buf,"thin-pack");
From: Calvin Wan <redacted>
There are some variables initialized at the start of the
do_fetch_pack_v2() state machine. Currently, they are initialized
in FETCH_CHECK_LOCAL, which is the initial state set at the beginning
of the function.
However, a subsequent patch will allow for another initial state,
while still requiring these initialized variables.
Move the initialization to be before the state machine,
so that they are set regardless of the initial state.
Note that there is no change in behavior, because we're moving code
from the beginning of the first state to just before the execution of
the state machine.
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
---
fetch-pack.c | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
@@ -1645,18 +1645,18 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,reader.me="fetch-pack";}+/* v2 supports these by default */+allow_unadvertised_object_request|=ALLOW_REACHABLE_SHA1;+use_sideband=2;+if(args->depth>0||args->deepen_since||args->deepen_not)+args->deepen=1;+while(state!=FETCH_DONE){switch(state){caseFETCH_CHECK_LOCAL:sort_ref_list(&ref,ref_compare_name);QSORT(sought,nr_sought,cmp_ref_by_name);-/* v2 supports these by default */-allow_unadvertised_object_request|=ALLOW_REACHABLE_SHA1;-use_sideband=2;-if(args->depth>0||args->deepen_since||args->deepen_not)-args->deepen=1;-/* Filter 'ref' by 'sought' and those that aren't local */mark_complete_and_common_ref(negotiator,args,&ref);filter_refs(args,&ref,sought,nr_sought);
From: Calvin Wan <redacted>
In order for a client to know what object-info components a server can
provide, advertise supported object-info features. This will allow a
client to decide whether to query the server for object-info or fetch
as a fallback.
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
---
serve.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
From: Calvin Wan <redacted>
Sometimes it is useful to get information about an object without having
to download it completely. The server logic has already been implemented
in “a2ba162cda (object-info: support for retrieving object info,
2021-04-20)”.
Add client functions to communicate with the server.
The client currently supports requesting a list of object ids with
feature 'size' from a v2 server. If a server does not
advertise the feature, then the client falls back
to making the request through 'fetch'.
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
---
Makefile | 1 +
fetch-object-info.c | 95 +++++++++++++++++++++++++++++++++++++++++++++
fetch-object-info.h | 18 +++++++++
fetch-pack.c | 3 ++
fetch-pack.h | 2 +
transport-helper.c | 11 +++++-
transport.c | 80 ++++++++++++++++++++++++++++++++++++--
transport.h | 11 ++++++
8 files changed, 216 insertions(+), 5 deletions(-)
create mode 100644 fetch-object-info.c
create mode 100644 fetch-object-info.h
@@ -709,8 +709,8 @@ static int fetch_refs(struct transport *transport,/**Ifwereachhere,thentheserver,theclient,and/orthetransport-*helperdoesnotsupportprotocolv2.--negotiate-onlyrequires-*protocolv2.+*helperdoesnotsupportprotocolv2.--negotiate-onlyandcat-file+*remote-object-inforequireprotocolv2.*/if(data->transport_options.acked_commits){warning(_("--negotiate-only requires protocol v2"));
@@ -726,6 +726,13 @@ static int fetch_refs(struct transport *transport,free_refs(dummy);}+/* fail the command explicitly to avoid further commands input. */+if(transport->smart_options->object_info)+die(_("remote-object-info requires protocol v2"));++if(!data->get_refs_list_called)+get_refs_list_using_list(transport,0);+count=0;for(i=0;i<nr_heads;i++)if(!(to_fetch[i]->status&REF_STATUS_UPTODATE))
@@ -418,6 +419,7 @@ static int fetch_refs_via_pack(struct transport *transport,structref*refs=NULL;structfetch_pack_argsargs;structref*refs_tmp=NULL,**to_fetch_dup=NULL;+structref*object_info_refs=NULL;memset(&args,0,sizeof(args));args.uploadpack=data->options.uploadpack;
@@ -444,11 +446,71 @@ static int fetch_refs_via_pack(struct transport *transport,args.server_options=transport->server_options;args.negotiation_tips=data->options.negotiation_tips;args.reject_shallow_remote=transport->smart_options->reject_shallow;+args.object_info=transport->smart_options->object_info;++if(transport->smart_options+&&transport->smart_options->object_info+&&transport->smart_options->object_info_oids->nr>0){+structref*ref_itr=object_info_refs=alloc_ref("");+structpacket_readerreader;+structobject_info_argsobj_info_args={0};++obj_info_args.server_options=transport->server_options;+obj_info_args.object_info_options=transport->smart_options->object_info_options;+obj_info_args.oids=transport->smart_options->object_info_oids;++connect_setup(transport,0);+packet_reader_init(&reader,data->fd[0],NULL,0,+PACKET_READ_CHOMP_NEWLINE|+PACKET_READ_GENTLE_ON_EOF|+PACKET_READ_DIE_ON_ERR_PACKET);++data->version=discover_version(&reader);+transport->hash_algo=reader.hash_algo;++if(!fetch_object_info(data->version,&obj_info_args,&reader,+data->options.object_info_data,transport->stateless_rpc,+data->fd[1])){+/*+*Ifthecodereacheshere,fetch_object_infoissuccessfuland+*remoteobjectinfoareretrievedfrompackets(i.e.without+*downloadingtheobjects).+*/+gotocleanup;+}-if(!data->finished_handshake){-inti;+/*+*Ifthecodereacheshere,itmeanswecan'tretrieveobjectinfofrom+*packets,andwewillfallbacktodownlandthepackfiles.+*Wesetquietandno_progresstobetrue,sothattheinternalcallto+*fetch-packislessverbose.+*/+args.object_info_data=data->options.object_info_data;+args.quiet=1;+args.no_progress=1;++/*+*Allocatememoryforobjectinfodataaccordingtooids.+*Theactualresultswillberetrievedlaterfromthedownloaded+*packfiles.+*/+for(size_ti=0;i<transport->smart_options->object_info_oids->nr;i++){+ref_itr->old_oid=transport->smart_options->object_info_oids->oid[i];+ref_itr->exact_oid=1;+if(i==transport->smart_options->object_info_oids->nr-1)+/* last element, no need to allocate to next */+ref_itr->next=NULL;+else+ref_itr->next=alloc_ref("");++ref_itr=ref_itr->next;+}++transport->remote_refs=object_info_refs;++}elseif(!data->finished_handshake){intmust_list_refs=0;-for(i=0;i<nr_heads;i++){+for(inti=0;i<nr_heads;i++){if(!to_fetch[i]->exact_oid){must_list_refs=1;break;
@@ -494,6 +556,17 @@ static int fetch_refs_via_pack(struct transport *transport,&transport->pack_lockfiles,data->version);data->finished_handshake=0;++/* Retrieve object info data from the downloaded pack files */+if(args.object_info){+structref*ref_cpy_reader=object_info_refs;+for(inti=0;ref_cpy_reader;i++){+oid_object_info_extended(the_repository,&ref_cpy_reader->old_oid,+&args.object_info_data[i],OBJECT_INFO_LOOKUP_REPLACE);+ref_cpy_reader=ref_cpy_reader->next;+}+}+data->options.self_contained_and_connected=args.self_contained_and_connected;data->options.connectivity_checked=args.connectivity_checked;
@@ -504,6 +577,7 @@ static int fetch_refs_via_pack(struct transport *transport,ret=-1;cleanup:+free_refs(object_info_refs);close(data->fd[0]);if(data->fd[1]>=0)close(data->fd[1]);
Some code declares variable i and only uses it
in a for loop, not in any other logic outside the loop.
Change the declaration of i to be inside the for loop for readability.
Helped-by: Christian Couder [off-list ref]
Signed-off-by: Eric Ju <redacted>
---
builtin/cat-file.c | 11 +++--------
1 file changed, 3 insertions(+), 8 deletions(-)
@@ -673,12 +673,10 @@ static void dispatch_calls(struct batch_options *opt,structqueued_cmd*cmd,intnr){-inti;-if(!opt->buffer_output)die(_("flush is only for --buffer mode"));-for(i=0;i<nr;i++)+for(size_ti=0;i<nr;i++)cmd[i].fn(opt,cmd[i].line,output,data);fflush(stdout);
Since the `info` command in cat-file --batch-command prints object info
for a given object, it is natural to add another command in cat-file
--batch-command to print object info for a given object from a remote.
Add `remote-object-info` to cat-file --batch-command.
While `info` takes object ids one at a time, this creates overhead when
making requests to a server so `remote-object-info` instead can take
multiple object ids at once.
cat-file --batch-command is generally implemented in the following
manner:
- Receive and parse input from user
- Call respective function attached to command
- Get object info, print object info
In --buffer mode, this changes to:
- Receive and parse input from user
- Store respective function attached to command in a queue
- After flush, loop through commands in queue
- Call respective function attached to command
- Get object info, print object info
Notice how the getting and printing of object info is accomplished one
at a time. As described above, this creates a problem for making
requests to a server. Therefore, `remote-object-info` is implemented in
the following manner:
- Receive and parse input from user
If command is `remote-object-info`:
- Get object info from remote
- Loop through and print each object info
Else:
- Call respective function attached to command
- Parse input, get object info, print object info
And finally for --buffer mode `remote-object-info`:
- Receive and parse input from user
- Store respective function attached to command in a queue
- After flush, loop through commands in queue:
If command is `remote-object-info`:
- Get object info from remote
- Loop through and print each object info
Else:
- Call respective function attached to command
- Get object info, print object info
To summarize, `remote-object-info` gets object info from the remote and
then loop through the object info passed in, printing the info.
In order for remote-object-info to avoid remote communication overhead
in the non-buffer mode, the objects are passed in as such:
remote-object-info <remote> <oid> <oid> ... <oid>
rather than
remote-object-info <remote> <oid>
remote-object-info <remote> <oid>
...
remote-object-info <remote> <oid>
Helped-by: Jonathan Tan [off-list ref]
Helped-by: Christian Couder [off-list ref]
Signed-off-by: Calvin Wan <redacted>
Signed-off-by: Eric Ju <redacted>
---
Documentation/git-cat-file.txt | 24 +-
builtin/cat-file.c | 108 +++-
object-file.c | 11 +
object-store-ll.h | 3 +
t/lib-cat-file.sh | 16 +
t/t1006-cat-file.sh | 13 +-
t/t1017-cat-file-remote-object-info.sh | 739 +++++++++++++++++++++++++
7 files changed, 897 insertions(+), 17 deletions(-)
create mode 100644 t/lib-cat-file.sh
create mode 100755 t/t1017-cat-file-remote-object-info.sh
@@ -149,6 +149,13 @@ info <object>:: Print object info for object reference `<object>`. This corresponds to the output of `--batch-check`.+remote-object-info <remote> <object>...::+ Print object info for object references `<object>` at specified <remote> without+ downloading objects from remote. If the object-info capability is not+ supported by the server, the objects will be downloaded instead.+ Error when no object references are provided.+ This command may be combined with `--buffer`.+ flush:: Used with `--buffer` to execute all preceding commands that were issued since the beginning or since the last flush was issued. When `--buffer`
@@ -290,7 +297,8 @@ newline. The available atoms are: The full hex representation of the object name. `objecttype`::- The type of the object (the same as `cat-file -t` reports).+ The type of the object (the same as `cat-file -t` reports). See+ `CAVEATS` below. Not supported by `remote-object-info`. `objectsize`:: The size, in bytes, of the object (the same as `cat-file -s`
@@ -298,13 +306,14 @@ newline. The available atoms are: `objectsize:disk`:: The size, in bytes, that the object takes up on disk. See the- note about on-disk sizes in the `CAVEATS` section below.+ note about on-disk sizes in the `CAVEATS` section below. Not+ supported by `remote-object-info`. `deltabase`:: If the object is stored as a delta on-disk, this expands to the full hex representation of the delta base object name. Otherwise, expands to the null OID (all zeroes). See `CAVEATS`- below.+ below. Not supported by `remote-object-info`. `rest`:: If this atom is used in the output string, input lines are split
@@ -314,7 +323,10 @@ newline. The available atoms are: line) are output in place of the `%(rest)` atom. If no format is specified, the default format is `%(objectname)-%(objecttype) %(objectsize)`.+%(objecttype) %(objectsize)`, except for `remote-object-info` commands which use+`%(objectname) %(objectsize)` for now because "%(objecttype)" is not supported yet.+WARNING: When "%(objecttype)" is supported, the default format WILL be unified, so+DO NOT RELY on the current the default format to stay the same!!! If `--batch` is specified, or if `--batch-command` is used with the `contents` command, the object information is followed by the object contents (consisting
@@ -396,6 +408,10 @@ scripting purposes. CAVEATS -------+Note that since %(objecttype), %(objectsize:disk) and %(deltabase) are+currently not supported by the `remote-object-info` command, we will error+and exit when they are in the format string.+ Note that the sizes of objects on disk are reported accurately, but care should be taken in drawing conclusions about which refs or objects are responsible for disk usage. The size of a packed non-delta object may be
@@ -576,6 +582,59 @@ static void batch_one_object(const char *obj_name,object_context_release(&ctx);}+staticintget_remote_info(structbatch_options*opt,intargc,constchar**argv)+{+intretval=0;+structremote*remote=NULL;+structobject_idoid;+structstring_listobject_info_options=STRING_LIST_INIT_NODUP;+staticstructtransport*gtransport;++/*+*Changetheformatto"%(objectname) %(objectsize)"when+*remote-object-infocommandisused.Oncewestartsupportingobjecttype+*thedefaultformatshouldchangetoDEFAULT_FORMAT+*/+if(!opt->format)+opt->format="%(objectname) %(objectsize)";++remote=remote_get(argv[0]);+if(!remote)+die(_("must supply valid remote when using remote-object-info"));++oid_array_clear(&object_info_oids);+for(size_ti=1;i<argc;i++){+if(get_oid_hex(argv[i],&oid))+die(_("Not a valid object name %s"),argv[i]);+oid_array_append(&object_info_oids,&oid);+}++gtransport=transport_get(remote,NULL);+if(gtransport->smart_options){+CALLOC_ARRAY(remote_object_info,object_info_oids.nr);+gtransport->smart_options->object_info=1;+gtransport->smart_options->object_info_oids=&object_info_oids;++/* 'objectsize' is the only option currently supported */+if(!strstr(opt->format,"%(objectsize)"))+die(_("%s is currently not supported with remote-object-info"),opt->format);++string_list_append(&object_info_options,"size");++if(object_info_options.nr>0){+gtransport->smart_options->object_info_options=&object_info_options;+gtransport->smart_options->object_info_data=remote_object_info;+retval=transport_fetch_refs(gtransport,NULL);+}+}else{+retval=-1;+}++string_list_clear(&object_info_options,0);+transport_disconnect(gtransport);+returnretval;+}+structobject_cb_data{structbatch_options*opt;structexpand_data*expand;
@@ -0,0 +1,16 @@+# Library of git-cat-file related tests.++# Print a string without a trailing newline+echo_without_newline(){+printf'%s'"$*"+}++# Print a string without newlines and replaces them with a NULL character (\0).+echo_without_newline_nul(){+echo_without_newline"$@"|tr'\n''\0'+}++# Calculate the length of a string removing any leading spaces.+strlen(){+echo_without_newline"$1"|wc-c|sed-e's/^ *//'+}
@@ -0,0 +1,739 @@+#!/bin/sh++test_description='git cat-file --batch-command with remote-object-info command'++GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main+exportGIT_TEST_DEFAULT_INITIAL_BRANCH_NAME++../test-lib.sh+."$TEST_DIRECTORY"/lib-cat-file.sh++hello_content="Hello World"+hello_size=$(strlen"$hello_content")+hello_oid=$(echo_without_newline"$hello_content"|githash-object--stdin)++# This is how we get 13:+# 13 = <file mode> + <a_space> + <file name> + <a_null>, where+# file mode is 100644, which is 6 characters;+# file name is hello, which is 5 characters+# a space is 1 character and a null is 1 character+tree_size=$(($(test_oidrawsz)+13))++commit_message="Initial commit"++# This is how we get 137:+# 137 = <tree header> + <a_space> + <a newline> ++# <Author line> + <a newline> ++# <Committer line> + <a newline> ++# <a newline> ++# <commit message length>+# An easier way to calculate is: 1. use `git cat-file commit <commit hash> | wc -c`,+# to get 177, 2. then deduct 40 hex characters to get 137+commit_size=$(($(test_oidhexsz)+137))++tag_header_without_oid="type blob+taghellotag+tagger$GIT_COMMITTER_NAME<$GIT_COMMITTER_EMAIL>"+tag_header_without_timestamp="object $hello_oid+$tag_header_without_oid"+tag_description="This is a tag"+tag_content="$tag_header_without_timestamp 0 +0000++$tag_description"++tag_oid=$(echo_without_newline"$tag_content"|githash-object-ttag--stdin-w)+tag_size=$(strlen"$tag_content")++set_transport_variables(){+hello_oid=$(echo_without_newline"$hello_content"|githash-object--stdin)+tree_oid=$(git-C"$1"write-tree)+commit_oid=$(echo_without_newline"$commit_message"|git-C"$1"commit-tree$tree_oid)+tag_oid=$(echo_without_newline"$tag_content"|git-C"$1"hash-object-ttag--stdin-w)+tag_size=$(strlen"$tag_content")+}++# This section tests --batch-command with remote-object-info command+# Since "%(objecttype)" is currently not supported by the command remote-object-info ,+# the filters are set to "%(objectname) %(objectsize)" in some test cases.++# Test --batch-command remote-object-info with 'git://' transport with+# transfer.advertiseobjectinfo set to true, i.e. server has object-info capability+."$TEST_DIRECTORY"/lib-git-daemon.sh+start_git_daemon--export-all--enable=receive-pack+daemon_parent=$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent++test_expect_success'create repo to be served by git-daemon''+gitinit"$daemon_parent"&&+echo_without_newline"$hello_content">$daemon_parent/hello&&+git-C"$daemon_parent"update-index--addhello&&+git-C"$daemon_parent"configtransfer.advertiseobjectinfotrue&&+gitclone"$GIT_DAEMON_URL/parent"-n"$daemon_parent/daemon_client_empty"+'++test_expect_success'batch-command remote-object-info git://''+(+set_transport_variables"$daemon_parent"&&+cd"$daemon_parent/daemon_client_empty"&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_oid missing">>expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+remote-object-info"$GIT_DAEMON_URL/parent"$hello_oid+remote-object-info"$GIT_DAEMON_URL/parent"$tree_oid+remote-object-info"$GIT_DAEMON_URL/parent"$commit_oid+remote-object-info"$GIT_DAEMON_URL/parent"$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command remote-object-info git:// multiple sha1 per line''+(+set_transport_variables"$daemon_parent"&&+cd"$daemon_parent/daemon_client_empty"&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_oid missing">>expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+remote-object-info"$GIT_DAEMON_URL/parent"$hello_oid$tree_oid$commit_oid$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command remote-object-info git:// default filter''+(+set_transport_variables"$daemon_parent"&&+cd"$daemon_parent/daemon_client_empty"&&++echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&+GIT_TRACE_PACKET=1gitcat-file--batch-command>actual<<-EOF&&+remote-object-info"$GIT_DAEMON_URL/parent"$hello_oid$tree_oid+remote-object-info"$GIT_DAEMON_URL/parent"$commit_oid$tag_oid+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command --buffer remote-object-info git://''+(+set_transport_variables"$daemon_parent"&&+cd"$daemon_parent/daemon_client_empty"&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_oid missing">>expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)"--buffer>actual<<-EOF&&+remote-object-info"$GIT_DAEMON_URL/parent"$hello_oid$tree_oid+remote-object-info"$GIT_DAEMON_URL/parent"$commit_oid$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+flush+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command -Z remote-object-info git:// default filter''+(+set_transport_variables"$daemon_parent"&&+cd"$daemon_parent/daemon_client_empty"&&++printf"%s\0""$hello_oid$hello_size">expect&&+printf"%s\0""$tree_oid$tree_size">>expect&&+printf"%s\0""$commit_oid$commit_size">>expect&&+printf"%s\0""$tag_oid$tag_size">>expect&&++printf"%s\0""$hello_oid missing">>expect&&+printf"%s\0""$tree_oid missing">>expect&&+printf"%s\0""$commit_oid missing">>expect&&+printf"%s\0""$tag_oid missing">>expect&&++batch_input="remote-object-info $GIT_DAEMON_URL/parent $hello_oid$tree_oid+remote-object-info$GIT_DAEMON_URL/parent$commit_oid$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+" &&+echo_without_newline_nul"$batch_input">commands_null_delimited&&++gitcat-file--batch-command-Z<commands_null_delimited>actual&&+test_cmpexpectactual+)+'++# Test --batch-command remote-object-info with 'git://' and+# transfer.advertiseobjectinfo set to false, i.e. server does not have object-info capability++test_expect_success'remote-object-info fallback git://: fetch objects to client''+(+git-C"$daemon_parent"configtransfer.advertiseobjectinfofalse&&+set_transport_variables"$daemon_parent"&&+cd"$daemon_parent/daemon_client_empty"&&++# Prove object is not on the client+echo"$hello_oid missing">expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&++# These results prove remote-object-info can retrieve object info+echo"$hello_oid$hello_size">>expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results are for the info command+# They prove objects are downloaded+echo"$hello_oid$hello_size">>expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+remote-object-info$GIT_DAEMON_URL/parent$hello_oid$tree_oid$commit_oid$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+EOF++# revert server state back+git-C"$daemon_parent"configtransfer.advertiseobjectinfotrue&&++test_cmpexpectactual+)+'++stop_git_daemon++# Test --batch-command remote-object-info with 'file://' transport with+# transfer.advertiseobjectinfo set to true, i.e. server has object-info capability+# shellcheck disable=SC2016+test_expect_success'create repo to be served by file:// transport''+gitinitserver&&+git-Cserverconfigprotocol.version2&&+git-Cserverconfigtransfer.advertiseobjectinfotrue&&+echo_without_newline"$hello_content">server/hello&&+git-Cserverupdate-index--addhello&&+gitclone-n"file://$(pwd)/server"file_client_empty+'++test_expect_success'batch-command remote-object-info file://''+(+set_transport_variables"server"&&+server_path="$(pwd)/server"&&+cdfile_client_empty&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_oid missing">>expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+remote-object-info"file://${server_path}"$hello_oid+remote-object-info"file://${server_path}"$tree_oid+remote-object-info"file://${server_path}"$commit_oid+remote-object-info"file://${server_path}"$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command remote-object-info file:// multiple sha1 per line''+(+set_transport_variables"server"&&+server_path="$(pwd)/server"&&+cdfile_client_empty&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_oid missing">>expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&+++gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+remote-object-info"file://${server_path}"$hello_oid$tree_oid$commit_oid$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command --buffer remote-object-info file://''+(+set_transport_variables"server"&&+server_path="$(pwd)/server"&&+cdfile_client_empty&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_oid missing">>expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)"--buffer>actual<<-EOF&&+remote-object-info"file://${server_path}"$hello_oid$tree_oid+remote-object-info"file://${server_path}"$commit_oid$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+flush+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command remote-object-info file:// default filter''+(+set_transport_variables"server"&&+server_path="$(pwd)/server"&&+cdfile_client_empty&&++echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++gitcat-file--batch-command>actual<<-EOF&&+remote-object-info"file://${server_path}"$hello_oid$tree_oid+remote-object-info"file://${server_path}"$commit_oid$tag_oid+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command -Z remote-object-info file:// default filter''+(+set_transport_variables"server"&&+server_path="$(pwd)/server"&&+cdfile_client_empty&&++printf"%s\0""$hello_oid$hello_size">expect&&+printf"%s\0""$tree_oid$tree_size">>expect&&+printf"%s\0""$commit_oid$commit_size">>expect&&+printf"%s\0""$tag_oid$tag_size">>expect&&++printf"%s\0""$hello_oid missing">>expect&&+printf"%s\0""$tree_oid missing">>expect&&+printf"%s\0""$commit_oid missing">>expect&&+printf"%s\0""$tag_oid missing">>expect&&++batch_input="remote-object-info \"file://${server_path}\" $hello_oid$tree_oid+remote-object-info\"file://${server_path}\"$commit_oid$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+" &&+echo_without_newline_nul"$batch_input">commands_null_delimited&&++gitcat-file--batch-command-Z<commands_null_delimited>actual&&+test_cmpexpectactual+)+'++# Test --batch-command remote-object-info with 'file://' and+# transfer.advertiseobjectinfo set to false, i.e. server does not have object-info capability++test_expect_success'remote-object-info fallback file://: fetch objects to client''+(+set_transport_variables"server"&&+server_path="$(pwd)/server"&&+git-C"${server_path}"configtransfer.advertiseobjectinfofalse&&+cdfile_client_empty&&++# Prove object is not on the client+echo"$hello_oid missing">expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&++# These results prove remote-object-info can retrieve object info+echo"$hello_oid$hello_size">>expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results are for the info command+# They prove objects are downloaded+echo"$hello_oid$hello_size">>expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+remote-object-info"file://${server_path}"$hello_oid$tree_oid$commit_oid$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+EOF++# revert server state back+git-C"${server_path}"configtransfer.advertiseobjectinfotrue&&+test_cmpexpectactual+)+'++# Test --batch-command remote-object-info with 'http://' transport with+# transfer.advertiseobjectinfo set to true, i.e. server has object-info capability++."$TEST_DIRECTORY"/lib-httpd.sh+start_httpd++test_expect_success'create repo to be served by http:// transport''+gitinit"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+git-C"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"confighttp.receivepacktrue&&+git-C"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"configtransfer.advertiseobjectinfotrue&&+echo_without_newline"$hello_content">$HTTPD_DOCUMENT_ROOT_PATH/http_parent/hello&&+git-C"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"update-index--addhello&&+gitclone"$HTTPD_URL/smart/http_parent"-n"$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty"+'++test_expect_success'batch-command remote-object-info http://''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty"&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_oid missing">>expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_oid+remote-object-info"$HTTPD_URL/smart/http_parent"$tree_oid+remote-object-info"$HTTPD_URL/smart/http_parent"$commit_oid+remote-object-info"$HTTPD_URL/smart/http_parent"$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command remote-object-info http:// one line''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty"&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_oid missing">>expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)">actual<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_oid$tree_oid$commit_oid$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command --buffer remote-object-info http://''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty"&&++# These results prove remote-object-info can get object info from the remote+echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results prove remote-object-info did not download objects from the remote+echo"$hello_oid missing">>expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&++gitcat-file--batch-command="%(objectname) %(objectsize)"--buffer>actual<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_oid$tree_oid+remote-object-info"$HTTPD_URL/smart/http_parent"$commit_oid$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+flush+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command remote-object-info http:// default filter''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty"&&++echo"$hello_oid$hello_size">expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++gitcat-file--batch-command>actual<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_oid$tree_oid+remote-object-info"$HTTPD_URL/smart/http_parent"$commit_oid$tag_oid+EOF+test_cmpexpectactual+)+'++test_expect_success'batch-command -Z remote-object-info http:// default filter''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty"&&++printf"%s\0""$hello_oid$hello_size">expect&&+printf"%s\0""$tree_oid$tree_size">>expect&&+printf"%s\0""$commit_oid$commit_size">>expect&&+printf"%s\0""$tag_oid$tag_size">>expect&&++batch_input="remote-object-info $HTTPD_URL/smart/http_parent $hello_oid$tree_oid+remote-object-info$HTTPD_URL/smart/http_parent$commit_oid$tag_oid+" &&+echo_without_newline_nul"$batch_input">commands_null_delimited&&++gitcat-file--batch-command-Z<commands_null_delimited>actual&&+test_cmpexpectactual+)+'++test_expect_success'remote-object-info fails on unspported filter option (objectsize:disk)''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&++test_must_failgitcat-file--batch-command="%(objectsize:disk)"2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_oid+EOF+test_grep"%(objectsize:disk) is currently not supported with remote-object-info"err+)+'++test_expect_success'remote-object-info fails on unspported filter option (deltabase)''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&++test_must_failgitcat-file--batch-command="%(deltabase)"2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_oid+EOF+test_grep"%(deltabase) is currently not supported with remote-object-info"err+)+'++test_expect_success'remote-object-info fails on server with legacy protocol''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&++test_must_failgit-cprotocol.version=0cat-file--batch-command="%(objectname) %(objectsize)"2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_oid+EOF+test_grep"remote-object-info requires protocol v2"err+)+'++test_expect_success'remote-object-info fails on server with legacy protocol fallback''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&++test_must_failgit-cprotocol.version=0cat-file--batch-command2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_oid+EOF+test_grep"remote-object-info requires protocol v2"err+)+'++test_expect_success'remote-object-info fails on malformed OID''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+malformed_object_id="this_id_is_not_valid"&&++test_must_failgitcat-file--batch-command="%(objectname) %(objectsize)"2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$malformed_object_id+EOF+test_grep"Not a valid object name '$malformed_object_id'"err+)+'++test_expect_success'remote-object-info fails on malformed OID fallback''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+malformed_object_id="this_id_is_not_valid"&&++test_must_failgitcat-file--batch-command2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$malformed_object_id+EOF+test_grep"Not a valid object name '$malformed_object_id'"err+)+'++test_expect_success'remote-object-info fails on missing OID''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+gitclone"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"missing_oid_repo&&+test_commit-Cmissing_oid_repomessage1c.txt&&+cdmissing_oid_repo&&++object_id=$(gitrev-parsemessage1:c.txt)&&+test_must_failgitcat-file--batch-command="%(objectname) %(objectsize)"2>err<<-EOF&&+remote-object-info"$HTTPD_URL/smart/http_parent"$object_id+EOF+test_grep"object-info: not our ref $object_id"err+)+'++# Test --batch-command remote-object-info with 'http://' transport and+# transfer.advertiseobjectinfo set to false, i.e. server does not have object-info capability++test_expect_success'remote-object-info fallback http://: fetch objects to client''+(+set_transport_variables"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"&&+git-C"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"configtransfer.advertiseobjectinfofalse&&+cd"$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty"&&++# Prove object is not on the client+echo"$hello_oid missing">expect&&+echo"$tree_oid missing">>expect&&+echo"$commit_oid missing">>expect&&+echo"$tag_oid missing">>expect&&++# These results prove remote-object-info can retrieve object info+echo"$hello_oid$hello_size">>expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++# These results are for the info command+# They prove objects are downloaded+echo"$hello_oid$hello_size">>expect&&+echo"$tree_oid$tree_size">>expect&&+echo"$commit_oid$commit_size">>expect&&+echo"$tag_oid$tag_size">>expect&&++gitcat-file--batch-command>actual<<-EOF&&+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+remote-object-info"$HTTPD_URL/smart/http_parent"$hello_oid$tree_oid$commit_oid$tag_oid+info$hello_oid+info$tree_oid+info$commit_oid+info$tag_oid+EOF++# revert server state back+git-C"$HTTPD_DOCUMENT_ROOT_PATH/http_parent"configtransfer.advertiseobjectinfotrue&&++test_cmpexpectactual+)+'++# DO NOT add non-httpd-specific tests here, because the last part of this+# test script is only executed when httpd is available and enabled.++test_done