[PATCH] fetch: show progress for packfile uri downloads

From: Albert Cui <redacted>

Git appears to hang when downloading packfiles as this part of the
fetch is silent, causing user confusion. This change implements
progress for the number of packfiles downloaded; a progress display
for bytes would involve deeper changes at the http-fetch layer
instead of fetch-pack, the caller.

Signed-off-by: Albert Cui <redacted>
---
    fetch: show progress for packfile uri downloads
    
    Git appears to hang when downloading packfiles as this part of the fetch
    is silent, causing user confusion. This change implements progress for
    the number of packfiles downloaded; a progress display for bytes would
    involve deeper changes at the http-fetch layer instead of fetch-pack,
    the caller.

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-907%2Falbertcui%2Fprogress-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-907/albertcui/progress-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/907

 fetch-pack.c           | 8 ++++++++
 t/t5702-protocol-v2.sh | 5 ++++-
 2 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/fetch-pack.c b/fetch-pack.c
index 6a61a464283e..5646d151293d 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c

@@ -23,6 +23,7 @@
 #include "fetch-negotiator.h"
 #include "fsck.h"
 #include "shallow.h"
+#include "progress.h"
 
 static int transfer_unpack_limit = -1;
 static int fetch_unpack_limit = -1;

@@ -1585,6 +1586,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
 	struct fetch_negotiator *negotiator;
 	int seen_ack = 0;
 	struct string_list packfile_uris = STRING_LIST_INIT_DUP;
+	struct progress *packfile_uri_progress;
 	int i;
 	struct strvec index_pack_args = STRVEC_INIT;
 	struct oidset gitmodules_oids = OIDSET_INIT;

@@ -1689,6 +1691,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
 		}
 	}
 
+	packfile_uri_progress = start_progress(_("Downloading packs"), packfile_uris.nr);
+
 	for (i = 0; i < packfile_uris.nr; i++) {
 		int j;
 		struct child_process cmd = CHILD_PROCESS_INIT;

@@ -1696,6 +1700,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
 		const char *uri = packfile_uris.items[i].string +
 			the_hash_algo->hexsz + 1;
 
+		display_progress(packfile_uri_progress, i+1);
 		strvec_push(&cmd.args, "http-fetch");
 		strvec_pushf(&cmd.args, "--packfile=%.*s",
 			     (int) the_hash_algo->hexsz,

@@ -1739,6 +1744,9 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
 						 get_object_directory(),
 						 packname));
 	}
+
+	stop_progress(&packfile_uri_progress);
+
 	string_list_clear(&packfile_uris, 0);
 	strvec_clear(&index_pack_args);
 

diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh
index 2e1243ca40b0..8964a4003678 100755
--- a/t/t5702-protocol-v2.sh
+++ b/t/t5702-protocol-v2.sh

@@ -851,7 +851,8 @@ test_expect_success 'part of packfile response provided as URI' '
 	GIT_TRACE=1 GIT_TRACE_PACKET="$(pwd)/log" GIT_TEST_SIDEBAND_ALL=1 \
 	git -c protocol.version=2 \
 		-c fetch.uriprotocols=http,https \
-		clone "$HTTPD_URL/smart/http_parent" http_child &&
+		clone "$HTTPD_URL/smart/http_parent" http_child \
+		--progress 2>progress &&
 
 	# Ensure that my-blob and other-blob are in separate packfiles.
 	for idx in http_child/.git/objects/pack/*.idx

@@ -875,6 +876,8 @@ test_expect_success 'part of packfile response provided as URI' '
 	test -f hfound &&
 	test -f h2found &&
 
+	test_i18ngrep "Downloading packs" progress &&
+
 	# Ensure that there are exactly 3 packfiles with associated .idx
 	ls http_child/.git/objects/pack/*.pack \
 	    http_child/.git/objects/pack/*.idx >filelist &&

base-commit: a5828ae6b52137b913b978e16cd2334482eb4c1f
-- 
gitgitgadget

On Wed, Mar 17, 2021 at 07:03:13PM +0000, Albert Cui via GitGitGadget wrote:

From: Albert Cui <redacted>

Git appears to hang when downloading packfiles as this part of the
fetch is silent, causing user confusion. This change implements
progress for the number of packfiles downloaded; a progress display
for bytes would involve deeper changes at the http-fetch layer
instead of fetch-pack, the caller.

I think this is an improvement, but I agree that a real byte display
would be much better. I actually worked on this a long time ago for a
very similar feature that we never quite pushed over the finish line.
See patches 11 and 12 here:

  https://lore.kernel.org/git/20111110074330.GA27925@sigill.intra.peff.net/

(it might need some of the earlier refactoring, too, I'm not sure; and
quite likely will need forward-porting as it has been 10 years).

-Peff

From: Albert Cui <redacted>

Git appears to hang when downloading packfiles as this part of the
fetch is silent, causing user confusion. This change implements
progress for the number of packfiles downloaded; a progress display
for bytes would involve deeper changes at the http-fetch layer
instead of fetch-pack, the caller, so we do not do that in this
patch.

Signed-off-by: Albert Cui <redacted>
---
    fetch: show progress for packfile uri downloads
    
    Git appears to hang when downloading packfiles as this part of the fetch
    is silent, causing user confusion. This change implements progress for
    the number of packfiles downloaded; a progress display for bytes would
    involve deeper changes at the http-fetch layer instead of fetch-pack,
    the caller.

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-907%2Falbertcui%2Fprogress-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-907/albertcui/progress-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/907

Range-diff vs v1:

 1:  d6cf8c195111 ! 1:  aedb1b33d0c4 fetch: show progress for packfile uri downloads
     @@ Commit message
          fetch is silent, causing user confusion. This change implements
          progress for the number of packfiles downloaded; a progress display
          for bytes would involve deeper changes at the http-fetch layer
     -    instead of fetch-pack, the caller.
     +    instead of fetch-pack, the caller, so we do not do that in this
     +    patch.
      
          Signed-off-by: Albert Cui [off-list ref]
      
     @@ fetch-pack.c: static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
       		const char *uri = packfile_uris.items[i].string +
       			the_hash_algo->hexsz + 1;
       
     -+		display_progress(packfile_uri_progress, i+1);
     ++		display_progress(packfile_uri_progress, i + 1);
       		strvec_push(&cmd.args, "http-fetch");
       		strvec_pushf(&cmd.args, "--packfile=%.*s",
       			     (int) the_hash_algo->hexsz,
     @@ fetch-pack.c: static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
      
       ## t/t5702-protocol-v2.sh ##
      @@ t/t5702-protocol-v2.sh: test_expect_success 'part of packfile response provided as URI' '
     - 	GIT_TRACE=1 GIT_TRACE_PACKET="$(pwd)/log" GIT_TEST_SIDEBAND_ALL=1 \
     + 	configure_exclusion "$P" my-blob >h &&
     + 	configure_exclusion "$P" other-blob >h2 &&
     + 
     +-	GIT_TRACE=1 GIT_TRACE_PACKET="$(pwd)/log" GIT_TEST_SIDEBAND_ALL=1 \
     ++	GIT_PROGRESS_DELAY=0 GIT_TRACE=1 GIT_TRACE2_EVENT=1 \
     ++	GIT_TRACE_PACKET="$(pwd)/log" GIT_TEST_SIDEBAND_ALL=1 \
       	git -c protocol.version=2 \
       		-c fetch.uriprotocols=http,https \
      -		clone "$HTTPD_URL/smart/http_parent" http_child &&


 fetch-pack.c           | 8 ++++++++
 t/t5702-protocol-v2.sh | 8 ++++++--
 2 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/fetch-pack.c b/fetch-pack.c
index 6a61a464283e..7ca85f40cf23 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c

@@ -23,6 +23,7 @@
 #include "fetch-negotiator.h"
 #include "fsck.h"
 #include "shallow.h"
+#include "progress.h"
 
 static int transfer_unpack_limit = -1;
 static int fetch_unpack_limit = -1;

@@ -1585,6 +1586,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
 	struct fetch_negotiator *negotiator;
 	int seen_ack = 0;
 	struct string_list packfile_uris = STRING_LIST_INIT_DUP;
+	struct progress *packfile_uri_progress;
 	int i;
 	struct strvec index_pack_args = STRVEC_INIT;
 	struct oidset gitmodules_oids = OIDSET_INIT;

@@ -1689,6 +1691,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
 		}
 	}
 
+	packfile_uri_progress = start_progress(_("Downloading packs"), packfile_uris.nr);
+
 	for (i = 0; i < packfile_uris.nr; i++) {
 		int j;
 		struct child_process cmd = CHILD_PROCESS_INIT;

@@ -1696,6 +1700,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
 		const char *uri = packfile_uris.items[i].string +
 			the_hash_algo->hexsz + 1;
 
+		display_progress(packfile_uri_progress, i + 1);
 		strvec_push(&cmd.args, "http-fetch");
 		strvec_pushf(&cmd.args, "--packfile=%.*s",
 			     (int) the_hash_algo->hexsz,

@@ -1739,6 +1744,9 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
 						 get_object_directory(),
 						 packname));
 	}
+
+	stop_progress(&packfile_uri_progress);
+
 	string_list_clear(&packfile_uris, 0);
 	strvec_clear(&index_pack_args);
 

diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh
index 2e1243ca40b0..0476b3f50455 100755
--- a/t/t5702-protocol-v2.sh
+++ b/t/t5702-protocol-v2.sh

@@ -848,10 +848,12 @@ test_expect_success 'part of packfile response provided as URI' '
 	configure_exclusion "$P" my-blob >h &&
 	configure_exclusion "$P" other-blob >h2 &&
 
-	GIT_TRACE=1 GIT_TRACE_PACKET="$(pwd)/log" GIT_TEST_SIDEBAND_ALL=1 \
+	GIT_PROGRESS_DELAY=0 GIT_TRACE=1 GIT_TRACE2_EVENT=1 \
+	GIT_TRACE_PACKET="$(pwd)/log" GIT_TEST_SIDEBAND_ALL=1 \
 	git -c protocol.version=2 \
 		-c fetch.uriprotocols=http,https \
-		clone "$HTTPD_URL/smart/http_parent" http_child &&
+		clone "$HTTPD_URL/smart/http_parent" http_child \
+		--progress 2>progress &&
 
 	# Ensure that my-blob and other-blob are in separate packfiles.
 	for idx in http_child/.git/objects/pack/*.idx

@@ -875,6 +877,8 @@ test_expect_success 'part of packfile response provided as URI' '
 	test -f hfound &&
 	test -f h2found &&
 
+	test_i18ngrep "Downloading packs" progress &&
+
 	# Ensure that there are exactly 3 packfiles with associated .idx
 	ls http_child/.git/objects/pack/*.pack \
 	    http_child/.git/objects/pack/*.idx >filelist &&

base-commit: a5828ae6b52137b913b978e16cd2334482eb4c1f
-- 
gitgitgadget

On Wed, Mar 17 2021, Albert Cui via GitGitGadget wrote:

This can just be "grep"

Hi,

Albert Cui wrote:

Git appears to hang when downloading packfiles as this part of the
fetch is silent, causing user confusion. This change implements
progress for the number of packfiles downloaded; a progress display
for bytes would involve deeper changes at the http-fetch layer
instead of fetch-pack, the caller, so we do not do that in this
patch.

Signed-off-by: Albert Cui <redacted>
---
 fetch-pack.c           | 8 ++++++++
 t/t5702-protocol-v2.sh | 8 ++++++--
 2 files changed, 14 insertions(+), 2 deletions(-)

This is something that came up at the last in-person Git Contributor
Summit; I'm glad to see it being taken care of.

It seems to be more idiomatic to initialize this to NULL.

That way, we can respect a --quiet option by making this remain NULL
when progress is not enabled:

	if (!args->quiet && !args->no_progress)
		packfile_uri_progress = ...;

[...]

This puts the trace in stderr mixed with other output.  Would it make
sense to put it in a separate file, like this?

	GIT_TRACE=1 GIT_TRACE_PACKET="$(pwd)/log" \
	GIT_PROGRESS_DELAY=0 GIT_TRACE2_EVENT="$(pwd)/trace2" \
	GIT_TEST_SIDEBAND_ALL=1 \
	git -c [etc]

[...]

That way, this "grep" could check the trace2 file which would contain
output intended for machines, and we wouldn't have to worry e.g. about
ANSII control codes potentially affecting the output around the space
some day in the progress output intended for a terminal.

With whatever subset of the changes described above make sense, this is
Reviewed-by: Jonathan Nieder <redacted>

Thanks.