[PATCH] fetch-pack: make packfile URIs work with transfer.fsckobjects

Subsystems: the rest

STALE2191d

7 messages, 2 authors, 2020-08-17 · open the first message on its own page

[PATCH] fetch-pack: make packfile URIs work with transfer.fsckobjects

From: Jonathan Tan <hidden>
Date: 2020-08-14 19:32:41

When fetching with packfile URIs and transfer.fsckobjects=1, use the
--fsck-objects instead of the --strict flag when invoking index-pack so
that links are not checked, only objects. This is because incomplete
links are expected. (A subsequent connectivity check will be done when
all the packs have been downloaded regardless of whether
transfer.fsckobjects is set.)

This is similar to 98a2ea46c2 ("fetch-pack: do not check links for
partial fetch", 2018-03-15), but for packfile URIs instead of partial
clones.

Signed-off-by: Jonathan Tan <redacted>
---
The subject is longer than 50 characters but I couldn't find a way to
shorten it, especially since I think it's important to mention packfile
URIs and transfer.fsckobjects. Any suggestions appreciated.
---
 fetch-pack.c           |  2 +-
 t/t5702-protocol-v2.sh | 53 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 54 insertions(+), 1 deletion(-)

diff --git a/fetch-pack.c b/fetch-pack.c
index 7f20eca4f8..66631d0034 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c

@@ -892,7 +892,7 @@ static int get_pack(struct fetch_pack_args *args,
 	    : transfer_fsck_objects >= 0
 	    ? transfer_fsck_objects
 	    : 0) {
-		if (args->from_promisor)
+		if (args->from_promisor || !only_packfile)
 			/*
 			 * We cannot use --strict in index-pack because it
 			 * checks both broken objects and links, but we only

diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh
index 5a60fbe3ed..8c6c67b10d 100755
--- a/t/t5702-protocol-v2.sh
+++ b/t/t5702-protocol-v2.sh

@@ -883,6 +883,59 @@ test_expect_success 'fetching with valid packfile URI but invalid hash fails' '
 	test_i18ngrep "pack downloaded from.*does not match expected hash" err
 '
 
+test_expect_success 'packfile-uri with transfer.fsckobjects' '
+	P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" &&
+	rm -rf "$P" http_child log &&
+
+	git init "$P" &&
+	git -C "$P" config "uploadpack.allowsidebandall" "true" &&
+
+	echo my-blob >"$P/my-blob" &&
+	git -C "$P" add my-blob &&
+	git -C "$P" commit -m x &&
+
+	configure_exclusion "$P" my-blob >h &&
+
+	sane_unset GIT_TEST_SIDEBAND_ALL &&
+	git -c protocol.version=2 -c transfer.fsckobjects=1 \
+		-c fetch.uriprotocols=http,https \
+		clone "$HTTPD_URL/smart/http_parent" http_child &&
+
+	# Ensure that there are exactly 4 files (2 .pack and 2 .idx).
+	ls http_child/.git/objects/pack/* >filelist &&
+	test_line_count = 4 filelist
+'
+
+test_expect_success 'packfile-uri with transfer.fsckobjects fails on bad object' '
+	P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" &&
+	rm -rf "$P" http_child log &&
+
+	git init "$P" &&
+	git -C "$P" config "uploadpack.allowsidebandall" "true" &&
+
+	cat >bogus-commit <<EOF &&
+tree $EMPTY_TREE
+author Bugs Bunny 1234567890 +0000
+committer Bugs Bunny <bugs@bun.ni> 1234567890 +0000
+
+This commit object intentionally broken
+EOF
+	BOGUS=$(git -C "$P" hash-object -t commit -w --stdin <bogus-commit) &&
+	git -C "$P" branch bogus-branch "$BOGUS" &&
+
+	echo my-blob >"$P/my-blob" &&
+	git -C "$P" add my-blob &&
+	git -C "$P" commit -m x &&
+
+	configure_exclusion "$P" my-blob >h &&
+
+	sane_unset GIT_TEST_SIDEBAND_ALL &&
+	test_must_fail git -c protocol.version=2 -c transfer.fsckobjects=1 \
+		-c fetch.uriprotocols=http,https \
+		clone "$HTTPD_URL/smart/http_parent" http_child 2>error &&
+	test_i18ngrep "invalid author/committer line - missing email" error
+'
+
 # DO NOT add non-httpd-specific tests here, because the last part of this
 # test script is only executed when httpd is available and enabled.

-- 
2.28.0.220.ged08abb693-goog

[PATCH v2 3/3] fetch-pack: make packfile URIs work with transfer.fsckobjects

From: Jonathan Tan <hidden>
Date: 2020-08-17 19:49:07

When fetching with packfile URIs and transfer.fsckobjects=1, use the
--fsck-objects instead of the --strict flag when invoking index-pack so
that links are not checked, only objects. This is because incomplete
links are expected. (A subsequent connectivity check will be done when
all the packs have been downloaded regardless of whether
transfer.fsckobjects is set.)

This is similar to 98a2ea46c2 ("fetch-pack: do not check links for
partial fetch", 2018-03-15), but for packfile URIs instead of partial
clones.

Signed-off-by: Jonathan Tan <redacted>
---
The subject is longer than 50 characters but I couldn't find a way to
shorten it, especially since I think it's important to mention packfile
URIs and transfer.fsckobjects. Any suggestions appreciated.
---
 fetch-pack.c           |  2 +-
 t/t5702-protocol-v2.sh | 53 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 54 insertions(+), 1 deletion(-)

diff --git a/fetch-pack.c b/fetch-pack.c
index 289121038e..0f1a84c061 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c

@@ -896,7 +896,7 @@ static int get_pack(struct fetch_pack_args *args,
 	    : transfer_fsck_objects >= 0
 	    ? transfer_fsck_objects
 	    : 0) {
-		if (args->from_promisor)
+		if (args->from_promisor || !only_packfile)
 			/*
 			 * We cannot use --strict in index-pack because it
 			 * checks both broken objects and links, but we only

diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh
index 5a60fbe3ed..8c6c67b10d 100755
--- a/t/t5702-protocol-v2.sh
+++ b/t/t5702-protocol-v2.sh

@@ -883,6 +883,59 @@ test_expect_success 'fetching with valid packfile URI but invalid hash fails' '
 	test_i18ngrep "pack downloaded from.*does not match expected hash" err
 '
 
+test_expect_success 'packfile-uri with transfer.fsckobjects' '
+	P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" &&
+	rm -rf "$P" http_child log &&
+
+	git init "$P" &&
+	git -C "$P" config "uploadpack.allowsidebandall" "true" &&
+
+	echo my-blob >"$P/my-blob" &&
+	git -C "$P" add my-blob &&
+	git -C "$P" commit -m x &&
+
+	configure_exclusion "$P" my-blob >h &&
+
+	sane_unset GIT_TEST_SIDEBAND_ALL &&
+	git -c protocol.version=2 -c transfer.fsckobjects=1 \
+		-c fetch.uriprotocols=http,https \
+		clone "$HTTPD_URL/smart/http_parent" http_child &&
+
+	# Ensure that there are exactly 4 files (2 .pack and 2 .idx).
+	ls http_child/.git/objects/pack/* >filelist &&
+	test_line_count = 4 filelist
+'
+
+test_expect_success 'packfile-uri with transfer.fsckobjects fails on bad object' '
+	P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" &&
+	rm -rf "$P" http_child log &&
+
+	git init "$P" &&
+	git -C "$P" config "uploadpack.allowsidebandall" "true" &&
+
+	cat >bogus-commit <<EOF &&
+tree $EMPTY_TREE
+author Bugs Bunny 1234567890 +0000
+committer Bugs Bunny <bugs@bun.ni> 1234567890 +0000
+
+This commit object intentionally broken
+EOF
+	BOGUS=$(git -C "$P" hash-object -t commit -w --stdin <bogus-commit) &&
+	git -C "$P" branch bogus-branch "$BOGUS" &&
+
+	echo my-blob >"$P/my-blob" &&
+	git -C "$P" add my-blob &&
+	git -C "$P" commit -m x &&
+
+	configure_exclusion "$P" my-blob >h &&
+
+	sane_unset GIT_TEST_SIDEBAND_ALL &&
+	test_must_fail git -c protocol.version=2 -c transfer.fsckobjects=1 \
+		-c fetch.uriprotocols=http,https \
+		clone "$HTTPD_URL/smart/http_parent" http_child 2>error &&
+	test_i18ngrep "invalid author/committer line - missing email" error
+'
+
 # DO NOT add non-httpd-specific tests here, because the last part of this
 # test script is only executed when httpd is available and enabled.

-- 
2.28.0.220.ged08abb693-goog

[PATCH v2 2/3] fetch-pack: document only_packfile in get_pack()

From: Jonathan Tan <hidden>
Date: 2020-08-17 19:49:13

dd4b732df7 ("upload-pack: send part of packfile response as uri",
2020-06-10) added the "only_packfile" parameter to get_pack() but did
not document it. Add documentation.

Signed-off-by: Jonathan Tan <redacted>
---
 fetch-pack.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/fetch-pack.c b/fetch-pack.c
index 7f20eca4f8..289121038e 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c

@@ -794,6 +794,10 @@ static void write_promisor_file(const char *keep_name,
 	strbuf_release(&promisor_name);
 }
 
+/*
+ * Pass 1 as "only_packfile" if the pack received is the only pack in this
+ * fetch request (that is, if there were no packfile URIs provided).
+ */
 static int get_pack(struct fetch_pack_args *args,
 		    int xd[2], struct string_list *pack_lockfiles,
 		    int only_packfile,

-- 
2.28.0.220.ged08abb693-goog

[PATCH v2 1/3] (various): document from_promisor parameter

From: Jonathan Tan <hidden>
Date: 2020-08-17 19:49:15

88e2f9ed8e ("introduce fetch-object: fetch one promisor object",
2017-12-05) plumbed through the from_promisor parameter but did
not document it everywhere it appeared. Add the documentation.

(It also plumbed through the no_dependents parameter, but I have left
that alone because it is being removed in a commit under review [1].)

[1] https://lore.kernel.org/git/e8f16d69089a5011c355d5939c56fa53b7a1eb2d.1597184949.git.jonathantanmy@google.com/

Signed-off-by: Jonathan Tan <redacted>
---
 fetch-pack.h  | 8 ++++++++
 remote-curl.c | 3 +++
 transport.h   | 3 +++
 3 files changed, 14 insertions(+)

diff --git a/fetch-pack.h b/fetch-pack.h
index 85d1e39fe7..5e747daea8 100644
--- a/fetch-pack.h
+++ b/fetch-pack.h

@@ -40,6 +40,14 @@ struct fetch_pack_args {
 	unsigned cloning:1;
 	unsigned update_shallow:1;
 	unsigned deepen:1;
+
+	/*
+	 * Indicate that the remote of this request is a promisor remote. The
+	 * pack received does not need all referred-to objects to be present in
+	 * the local object store, and fetch-pack will store the pack received
+	 * together with a ".promisor" file indicating that the aforementioned
+	 * pack is a promisor pack.
+	 */
 	unsigned from_promisor:1;
 
 	/*

diff --git a/remote-curl.c b/remote-curl.c
index 62b3a45cde..0c1833dcb6 100644
--- a/remote-curl.c
+++ b/remote-curl.c

@@ -39,7 +39,10 @@ struct options {
 		/* One of the SEND_PACK_PUSH_CERT_* constants. */
 		push_cert : 2,
 		deepen_relative : 1,
+
+		/* see documentation of corresponding flag in fetch-pack.h */
 		from_promisor : 1,
+
 		no_dependents : 1,
 		atomic : 1,
 		object_format : 1;

diff --git a/transport.h b/transport.h
index 1be4013dec..298d9eedc9 100644
--- a/transport.h
+++ b/transport.h

@@ -15,7 +15,10 @@ struct git_transport_options {
 	unsigned self_contained_and_connected : 1;
 	unsigned update_shallow : 1;
 	unsigned deepen_relative : 1;
+
+	/* see documentation of corresponding flag in fetch-pack.h */
 	unsigned from_promisor : 1;
+
 	unsigned no_dependents : 1;
 
 	/*

-- 
2.28.0.220.ged08abb693-goog

[PATCH v2 0/3] make packfile URIs work with transfer.fsckobjects

From: Jonathan Tan <hidden>
Date: 2020-08-17 19:49:17

Thanks, Junio, for taking a look.

Differences from v1:

 - Added 2 patches that document parts that Junio mentioned were
   underdocumented [1]. I'm not sure about only_packfile, since it's
   narrowly scoped (just an argument in a static function) but I've
   included it anyway.
 - Used <<-EOF instead of <<EOF.

[1] https://lore.kernel.org/git/xmqq8sehvw13.fsf@gitster.c.googlers.com/

Jonathan Tan (3):
  (various): document from_promisor parameter
  fetch-pack: document only_packfile in get_pack()
  fetch-pack: make packfile URIs work with transfer.fsckobjects

 fetch-pack.c           |  6 ++++-
 fetch-pack.h           |  8 +++++++
 remote-curl.c          |  3 +++
 t/t5702-protocol-v2.sh | 53 ++++++++++++++++++++++++++++++++++++++++++
 transport.h            |  3 +++
 5 files changed, 72 insertions(+), 1 deletion(-)

-- 
2.28.0.220.ged08abb693-goog

Re: [PATCH v2 3/3] fetch-pack: make packfile URIs work with transfer.fsckobjects

From: Eric Sunshine <hidden>
Date: 2020-08-17 19:52:57

On Mon, Aug 17, 2020 at 3:49 PM Jonathan Tan [off-list ref] wrote:

+test_expect_success 'packfile-uri with transfer.fsckobjects fails on bad object' '
+       P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" &&
+       rm -rf "$P" http_child log &&
+
+       git init "$P" &&
+       git -C "$P" config "uploadpack.allowsidebandall" "true" &&
+
+       cat >bogus-commit <<EOF &&
+tree $EMPTY_TREE
+author Bugs Bunny 1234567890 +0000
+committer Bugs Bunny [off-list ref] 1234567890 +0000
+
+This commit object intentionally broken
+EOF

The v2 cover letter says that this now uses <<-EOF so the here-doc
body can be indented, but that doesn't seem to be the case.

Re: [PATCH v2 3/3] fetch-pack: make packfile URIs work with transfer.fsckobjects

From: Jonathan Tan <hidden>
Date: 2020-08-17 20:22:41

On Mon, Aug 17, 2020 at 3:49 PM Jonathan Tan [off-list ref] wrote:

quoted

+test_expect_success 'packfile-uri with transfer.fsckobjects fails on bad object' '
+       P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" &&
+       rm -rf "$P" http_child log &&
+
+       git init "$P" &&
+       git -C "$P" config "uploadpack.allowsidebandall" "true" &&
+
+       cat >bogus-commit <<EOF &&
+tree $EMPTY_TREE
+author Bugs Bunny 1234567890 +0000
+committer Bugs Bunny [off-list ref] 1234567890 +0000
+
+This commit object intentionally broken
+EOF

The v2 cover letter says that this now uses <<-EOF so the here-doc
body can be indented, but that doesn't seem to be the case.

Ah, good catch. I forgot to amend my commit. If Junio doesn't do it
locally then I will send out a v3.

`h`	back out one level
`j`	next message in thread
`k`	previous message in thread
`l`	drill in
`Esc`	close help / fold thread tree
`?`	toggle this help