From: Martin Ågren <hidden> Date: 2020-05-17 18:16:52
On Sat, 16 May 2020 at 22:47, brian m. carlson
[off-list ref] wrote:
On 2020-05-16 at 11:18:12, Martin Ågren wrote:
quoted
On Wed, 13 May 2020 at 02:56, brian m. carlson
[off-list ref] wrote:
quoted
git index-pack is usually run in a repository, but need not be. Since
packs don't contains information on the algorithm in use, instead
relying on context, add an option to index-pack to tell it which one
we're using in case someone runs it outside of a repository.
quoted
Similar to an earlier patch where we modify `the_hash_algo` like this, I
feel a bit nervous. What happens if you pass in a "wrong" algo here,
i.e., SHA-1 in a SHA-256 repo? Or, given the motivation in the commit
message, should this only be allowed if we really *are* outside a repo?
Unfortunately, we can't prevent the user from being inside repository A,
which is SHA-1, while invoking git index-pack on repository B, which is
SHA-256.
Ah, I see.
That is valid without --stdin, if uncommon, and it needs to be
supported. I can prevent it from being used with --stdin, though.
Hmm, that might make sense. I suppose it could quickly get out of
control with bug reports coming in along the lines of "if I do this
really crazy git index-pack invocation, I manage to mess things up". The
easiest way to address this might be through documentation, i.e., "don't
use this option", "for internal use" or even "to be used by the test
suite only" for which there is even precedence in git-index-pack(1).
On the other hand, if we need to detect such hash mismatch even once the
SHA-256 work is 100% complete, then I suppose we really should try a
bit to catch bad invocations.
As a tangent, I see that v2.27.0 will come with `git init
--object-format=<format>` and `GIT_DEFAULT_HASH_ALGORITHM`. The docs for
the former mentions "(if enabled)". Should we add something more scary
to those to make it clear that they shouldn't be used and that you
basically shouldn't even try to figure out how to enable them? I can
already see the tweets and blog posts a few weeks from now about how you
can build Git from source setting a single switch, run
git init --object-format=sha256
and you're in the future! Which will just lead to pain some days or
weeks later.... "I've done lots of work. How do I convert my repo to
SHA-1 so I can share it?"...
We've added "experimental" things before and tried to document the
experimental nature. Maybe here we're not even "experimental" -- more
like "if you use this in production, you *will* suffer"?
If you pass in a wrong algorithm, we usually blow up with an inflate
error because we consume more bytes than expected with our ref deltas.
I'm not aware of any cases where we segfault or access invalid memory;
we just blow up in a nonobvious way. That's true, too, if you manually
tamper with the algorithm in extensions.objectformat; usually we blow up
(but not segfault) because the index is "corrupt".
Ok, I see. I suppose "some time", we could tweak error messages to hint
about an object-format mismatch, but I don't think that needs to block
your work here now.
Martin
From: brian m. carlson <hidden> Date: 2020-05-17 20:52:54
On 2020-05-17 at 18:16:37, Martin Ågren wrote:
On Sat, 16 May 2020 at 22:47, brian m. carlson
[off-list ref] wrote:
quoted
That is valid without --stdin, if uncommon, and it needs to be
supported. I can prevent it from being used with --stdin, though.
Hmm, that might make sense. I suppose it could quickly get out of
control with bug reports coming in along the lines of "if I do this
really crazy git index-pack invocation, I manage to mess things up". The
easiest way to address this might be through documentation, i.e., "don't
use this option", "for internal use" or even "to be used by the test
suite only" for which there is even precedence in git-index-pack(1).
On the other hand, if we need to detect such hash mismatch even once the
SHA-256 work is 100% complete, then I suppose we really should try a
bit to catch bad invocations.
I can add documentation and a warning there.
If we actually verified the checksum at the end of the pack first, then
we'd be able to distinguish the two cases, because we'd try to compute a
clearly invalid hash over the body, and the likelihood of it matching
would be very small. We don't at the moment, for reasons I'm unclear
about, but it's probably performance.
As a tangent, I see that v2.27.0 will come with `git init
--object-format=<format>` and `GIT_DEFAULT_HASH_ALGORITHM`. The docs for
the former mentions "(if enabled)". Should we add something more scary
to those to make it clear that they shouldn't be used and that you
basically shouldn't even try to figure out how to enable them? I can
already see the tweets and blog posts a few weeks from now about how you
can build Git from source setting a single switch, run
git init --object-format=sha256
and you're in the future! Which will just lead to pain some days or
weeks later.... "I've done lots of work. How do I convert my repo to
SHA-1 so I can share it?"...
We've added "experimental" things before and tried to document the
experimental nature. Maybe here we're not even "experimental" -- more
like "if you use this in production, you *will* suffer"?
Well, the option is there, but it produces the following:
% git init --object-format=sha256
fatal: The hash algorithm sha256 is not supported in this build.
which can be distinguished from this:
% git init --object-format=blake2b
fatal: unknown hash algorithm 'blake2b'
Right now it's pretty broken without this series, so you can't use it.
I mean, you have the source and can remove the check, but it doesn't
work as it stands, so I'm not too worried about people trying to do that
at the moment. I'll sneak in some documentation for the end product,
though.
--
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204
From: brian m. carlson <hidden> Date: 2020-05-17 22:37:57
On 2020-05-16 at 11:14:16, Martin Ågren wrote:
On Wed, 13 May 2020 at 02:58, brian m. carlson
[off-list ref] wrote:
quoted
When using protocol v2, we need to know what hash algorithm is used by
the remote end. See if the server has sent us an object-format
capability, and if so, use it to determine the hash algorithm in use and
set that value in the packet reader. Parse the refs using this
algorithm.
Note that we use memcpy instead of oidcpy for copying values, since
oidcpy is intentionally limited to the current hash algorithm length,
and the copy will be too short if the server side uses SHA-256 but the
client side has not had a repository set up (and therefore defaults to
SHA-1).
Might an `oidcpy_algop()` prove useful over time?
oidcpy_algop(&ref->old_oid, &old_oid, reader->hash_algo);
I think I can just omit this chunk, because oidcpy now copies the entire
struct for speed.
--
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204
From: brian m. carlson <hidden> Date: 2020-05-25 19:59:48
The pattern here looking for failures is specific to SHA-1. Let's
create a variable that matches the regex or glob pattern for a path
within the objects directory.
Signed-off-by: brian m. carlson <redacted>
---
t/t1050-large.sh | 2 +-
t/test-lib.sh | 1 +
2 files changed, 2 insertions(+), 1 deletion(-)
@@ -64,7 +64,7 @@ test_expect_success 'add a large file or two' 'test$count=1&&cnt=$(gitshow-index<"$idx"|wc-l)&&test$cnt=2&&-forlin.git/objects/??/??????????????????????????????????????+forlin.git/objects/$OIDPATH_REGEXdotest_path_is_file"$l"||continuebad=t
From: brian m. carlson <hidden> Date: 2020-05-25 19:59:49
Document a capability that indicates which hash algorithms are in use by
both sides of a remote connection. Use the term "object-format", since
this is the term used for the repository extension as well.
Signed-off-by: brian m. carlson <redacted>
---
Documentation/technical/protocol-capabilities.txt | 15 +++++++++++++++
1 file changed, 15 insertions(+)
@@ -176,6 +176,21 @@ agent strings are purely informative for statistics and debugging purposes, and MUST NOT be used to programmatically assume the presence or absence of particular features.+object-format+-------------++This capability, which takes a hash algorithm as an argument, indicates+that the server supports the given hash algorithms. It may be sent+multiple times; if so, the first one given is the one used in the ref+advertisement.++When provided by the client, this indicates that it intends to use the+given hash algorithm to communicate. The algorithm provided must be one+that the server supports.++If this capability is not provided, it is assumed that the only+supported algorithm is SHA-1.+ symref ------
From: brian m. carlson <hidden> Date: 2020-05-25 19:59:50
This is part 2 of 3 of the SHA-256 work. This series adds all of the
protocol logic to work with SHA-256 repositories.
Changes from v1:
* Fix spurious line additions and deletions.
* Rename len to linelen for easier understanding.
* Move the documentation comment for xstrncmpz to the header.
* Drop a useless variable (found).
* Update several commit messages to better explain things as suggested
by Junio and Martin.
* Name the parameters for parse_feature_value for better documentation.
* Reduce the scope of variables when possible.
* Add explicit handling for missing object-format capabilities.
* Rename all new options to --object-format.
* Use oidcpy where possible.
* Test more failure cases.
* Have index-pack fail if --stdin and --object-format are both
specified.
* Move and simplify t5704.
* Other miscellaneous cleanups to respond to review feedback.
Range-diff below.
brian m. carlson (44):
t1050: match object ID paths in a hash-insensitive way
Documentation: document v1 protocol object-format capability
connect: have ref processing code take struct packet_reader
wrapper: add function to compare strings with different NUL
termination
remote: advertise the object-format capability on the server side
connect: add function to parse multiple v1 capability values
connect: add function to fetch value of a v2 server capability
pkt-line: add a member for hash algorithm
transport: add a hash algorithm member
connect: add function to detect supported v1 hash functions
send-pack: detect when the server doesn't support our hash
connect: make parse_feature_value extern
fetch-pack: detect when the server doesn't support our hash
connect: detect algorithm when fetching refs
builtin/receive-pack: detect when the server doesn't support our hash
docs: update remote helper docs for object-format extensions
transport-helper: implement object-format extensions
remote-curl: implement object-format extensions
builtin/clone: initialize hash algorithm properly
t5562: pass object-format in synthesized test data
fetch-pack: parse and advertise the object-format capability
setup: set the_repository's hash algo when checking format
t3200: mark assertion with SHA1 prerequisite
packfile: compute and use the index CRC offset
t5302: modernize test formatting
builtin/show-index: provide options to determine hash algo
t1302: expect repo format version 1 for SHA-256
Documentation/technical: document object-format for protocol v2
connect: pass full packet reader when parsing v2 refs
connect: parse v2 refs with correct hash algorithm
serve: advertise object-format capability for protocol v2
t5500: make hash independent
builtin/ls-remote: initialize repository based on fetch
remote-curl: detect algorithm for dumb HTTP by size
builtin/index-pack: add option to specify hash algorithm
t1050: pass algorithm to index-pack when outside repo
remote-curl: avoid truncating refs with ls-remote
t/helper: initialize the repository for test-sha1-array
t5702: offer an object-format capability in the test
t5703: use object-format serve option
t5704: send object-format capability with SHA-256
t5300: pass --object-format to git index-pack
bundle: detect hash algorithm when reading refs
remote-testgit: adapt for object-format
Documentation/git-index-pack.txt | 8 +
Documentation/git-show-index.txt | 11 +-
Documentation/gitremote-helpers.txt | 33 +-
.../technical/protocol-capabilities.txt | 15 +
Documentation/technical/protocol-v2.txt | 9 +
builtin/clone.c | 9 +
builtin/index-pack.c | 14 +-
builtin/ls-remote.c | 4 +
builtin/receive-pack.c | 10 +
builtin/show-index.c | 29 +-
bundle.c | 22 +-
bundle.h | 1 +
connect.c | 138 +++++--
connect.h | 3 +
fetch-pack.c | 14 +
git-compat-util.h | 6 +
git.c | 2 +-
object-store.h | 1 +
packfile.c | 1 +
pkt-line.c | 1 +
pkt-line.h | 3 +
remote-curl.c | 46 ++-
send-pack.c | 6 +
serve.c | 27 ++
setup.c | 1 +
t/helper/test-oid-array.c | 3 +
t/t1050-large.sh | 6 +-
t/t1302-repo-version.sh | 6 +-
t/t3200-branch.sh | 2 +-
t/t5300-pack-object.sh | 9 +-
t/t5302-pack-index.sh | 360 +++++++++---------
t/t5500-fetch-pack.sh | 5 +-
t/t5562-http-backend-content-length.sh | 5 +-
t/t5701-git-serve.sh | 25 ++
t/t5702-protocol-v2.sh | 2 +
t/t5703-upload-pack-ref-in-want.sh | 19 +-
t/t5704-protocol-violations.sh | 2 +
t/t5801/git-remote-testgit | 6 +
t/test-lib.sh | 1 +
transport-helper.c | 24 +-
transport.c | 18 +-
transport.h | 8 +
upload-pack.c | 3 +-
wrapper.c | 8 +
44 files changed, 678 insertions(+), 248 deletions(-)
Range-diff against v1:
1: 82a0a5beae = 1: 5878fe6a98 t1050: match object ID paths in a hash-insensitive way
2: 95e84f6457 ! 2: 402864eaa3 Documentation: document v1 protocol object-format capability
@@ Documentation/technical/protocol-capabilities.txt: agent strings are purely info
symref
------
-@@ Documentation/technical/protocol-capabilities.txt: refs being sent.
-
- Clients MAY use the parameters from this capability to select the proper initial
- branch when cloning a repository.
--
- shallow
- -------
-
3: 7c82e91a11 ! 3: d124692e2f connect: have ref processing code take struct packet_reader
@@ Commit message
code take pointers to struct reader instead of having to pass multiple
arguments to each function.
+ Rename the len variable to "linelen" to make it clearer what the
+ variable does in light of the variable change.
+
Signed-off-by: brian m. carlson [off-list ref]
## connect.c ##
@@ connect.c: static void annotate_refs_with_symref_info(struct ref *ref)
}
-static void process_capabilities(const char *line, int *len)
-+static void process_capabilities(struct packet_reader *reader, int *len)
++static void process_capabilities(struct packet_reader *reader, int *linelen)
{
+ const char *line = reader->line;
int nul_location = strlen(line);
- if (nul_location == *len)
+- if (nul_location == *len)
++ if (nul_location == *linelen)
return;
-@@ connect.c: static void process_capabilities(const char *line, int *len)
- *len = nul_location;
+ server_capabilities_v1 = xstrdup(line + nul_location + 1);
+- *len = nul_location;
++ *linelen = nul_location;
}
-static int process_dummy_ref(const char *line)
4: a78234de04 < -: ---------- wrapper: add function to compare strings with different NUL termination
-: ---------- > 4: cce29662b4 wrapper: add function to compare strings with different NUL termination
5: 628ecec99a = 5: 3b207e304b remote: advertise the object-format capability on the server side
6: 9990767072 = 6: 235d7f5b8f connect: add function to parse multiple v1 capability values
7: 5ce2b7afde = 7: 0324e126b1 connect: add function to fetch value of a v2 server capability
8: e5d58b48f3 = 8: cdba3122ce pkt-line: add a member for hash algorithm
9: bce9ba0538 = 9: c8233c3b42 transport: add a hash algorithm member
10: 2d016e3870 ! 10: b9273c4021 connect: add function to detect supported v1 hash functions
@@ connect.c: static const char *parse_feature_value(const char *feature_list, cons
+int server_supports_hash(const char *desired, int *feature_supported)
+{
+ int offset = 0;
-+ int len, found = 0;
++ int len;
+ const char *hash;
+
+ hash = next_server_feature_value("object-format", &len, &offset);
@@ connect.c: static const char *parse_feature_value(const char *feature_list, cons
+ }
+ while (hash) {
+ if (!xstrncmpz(desired, hash, len))
-+ found = 1;
-+
-+ if (found)
+ return 1;
++
+ hash = next_server_feature_value("object-format", &len, &offset);
+ }
+ return 0;
11: 9fdc67b825 ! 11: e2d37b75c8 send-pack: detect when the server doesn't support our hash
@@ Commit message
send-pack: detect when the server doesn't support our hash
Detect when the server doesn't support our hash algorithm and abort.
+ If the server does support our hash, advertise it as part of our
+ capabilities.
Signed-off-by: brian m. carlson [off-list ref]
12: 91a1fb0a7d ! 12: 602734cbbb connect: make parse_feature_value extern
@@ connect.h: struct packet_reader;
enum protocol_version discover_version(struct packet_reader *reader);
int server_supports_hash(const char *desired, int *feature_supported);
-+const char *parse_feature_value(const char *, const char *, int *, int *);
++const char *parse_feature_value(const char *feature_list, const char *feature, int *lenp, int *offset);
int server_supports_v2(const char *c, int die_on_error);
int server_feature_v2(const char *c, const char **v);
int server_supports_feature(const char *c, const char *feature,
13: fd82e5f755 = 13: d97fa2c8aa fetch-pack: detect when the server doesn't support our hash
14: b62f751fe4 ! 14: ba052f1da7 connect: detect algorithm when fetching refs
@@ Commit message
If we're fetching refs, detect the hash algorithm and parse the refs
using that algorithm.
+ As mentioned in the documentation, if multiple versions of the
+ object-format capability are provided, we use the first. No known
+ implementation supports multiple algorithms now, but they may in the
+ future.
+
Signed-off-by: brian m. carlson [off-list ref]
## connect.c ##
@@ connect.c: static void annotate_refs_with_symref_info(struct ref *ref)
- static void process_capabilities(struct packet_reader *reader, int *len)
+ static void process_capabilities(struct packet_reader *reader, int *linelen)
{
+ const char *feat_val;
+ int feat_len;
-+ int hash_algo;
const char *line = reader->line;
int nul_location = strlen(line);
- if (nul_location == *len)
+ if (nul_location == *linelen)
return;
server_capabilities_v1 = xstrdup(line + nul_location + 1);
- *len = nul_location;
+ *linelen = nul_location;
+
+ feat_val = server_feature_value("object-format", &feat_len);
+ if (feat_val) {
+ char *hash_name = xstrndup(feat_val, feat_len);
-+ hash_algo = hash_algo_by_name(hash_name);
++ int hash_algo = hash_algo_by_name(hash_name);
+ if (hash_algo != GIT_HASH_UNKNOWN)
+ reader->hash_algo = &hash_algos[hash_algo];
+ free(hash_name);
++ } else {
++ reader->hash_algo = &hash_algos[GIT_HASH_SHA1];
+ }
}
15: 29b4219411 ! 15: 661d94d4de builtin/receive-pack: detect when the server doesn't support our hash
@@ builtin/receive-pack.c: static struct command *read_head_info(struct packet_read
linelen = strlen(reader->line);
if (linelen < reader->pktlen) {
const char *feature_list = reader->line + linelen + 1;
-+ const char *hash;
++ const char *hash = NULL;
+ int len = 0;
if (parse_feature_request(feature_list, "report-status"))
report_status = 1;
16: f8eb8c96f8 = 16: fd8b85390c docs: update remote helper docs for object-format extensions
17: 93bf7005a8 = 17: 32285e611f transport-helper: implement object-format extensions
18: ed75c102a3 = 18: a33d1ed9a0 remote-curl: implement object-format extensions
19: bf16872e73 = 19: fffdf0780d builtin/clone: initialize hash algorithm properly
20: ce77713343 ! 20: f616f85b4b t5562: pass object-format in synthesized test data
@@ t/t5562-http-backend-content-length.sh: test_expect_success 'setup' '
printf 0000 &&
echo "$hash_next" | git pack-objects --stdout
} >push_body &&
-@@ t/t5562-http-backend-content-length.sh: test_expect_success GZIP 'push plain' '
- test_cmp act.head exp.head
- '
-
-+test_expect_success GZIP 'push plain with SHA-1' '
-+ test_when_finished "git branch -D newbranch" &&
-+ test_http_env receive push_body &&
-+ verify_http_result "200 OK" &&
-+ git rev-parse newbranch >act.head &&
-+ echo "$hash_next" >exp.head &&
-+ test_cmp act.head exp.head
-+'
-+
- test_expect_success 'push plain truncated' '
- test_http_env receive push_body.trunc &&
- ! verify_http_result "200 OK"
21: e4dd90fa9d < -: ---------- t5704: send object-format capability with SHA-256
22: 626d6e9018 ! 21: eca43da42e fetch-pack: parse and advertise the object-format capability
@@ fetch-pack.c: static int send_fetch_request(struct fetch_negotiator *negotiator,
+ die(_("mismatched algorithms: client %s; server %s"),
+ the_hash_algo->name, hash_name);
+ packet_write_fmt(fd_out, "object-format=%s", the_hash_algo->name);
-+ }
-+ else if (hash_algo_by_ptr(the_hash_algo) != GIT_HASH_SHA1)
++ } else if (hash_algo_by_ptr(the_hash_algo) != GIT_HASH_SHA1) {
+ die(_("the server does not support algorithm '%s'"),
+ the_hash_algo->name);
++ }
+
packet_buf_delim(&req_buf);
if (args->use_thin_pack)
23: 8c675b5117 ! 22: 22c1a62e10 setup: set the_repository's hash algo when checking format
@@ Commit message
the same time. This ensures that we perform a suitable initialization
early enough to avoid confusing any parts of the code. If we defer
until later, we can end up with portions of the code which are confused
- about the hash algorithm, resulting in segfaults.
+ about the hash algorithm, resulting in segfaults when working with
+ SHA-256 repositories.
Signed-off-by: brian m. carlson [off-list ref]
24: b714d4accc = 23: 7c7f2263d5 t3200: mark assertion with SHA1 prerequisite
25: eacde58fda = 24: ee8a71a926 packfile: compute and use the index CRC offset
26: 81bb8cdb18 = 25: 6afecf0b09 t5302: modernize test formatting
27: 8623f21715 ! 26: 99a847ba4e builtin/show-index: provide options to determine hash algo
@@ Metadata
## Commit message ##
builtin/show-index: provide options to determine hash algo
- It's possible to use a variety of index formats with show-index, and we
- need a way to indicate the hash algorithm which is in use for a
- particular index we'd like to show. Default to using the value for the
- repository we're in by calling setup_git_directory_gently, and allow
- overriding it by using a --hash argument.
+ show-index is capable of reading any possible index file whether or not
+ the index is inside a repository. However, because our index files lack
+ metadata about the hash algorithm in use, it's not possible to
+ autodetect the algorithm that a particular index file is using.
+
+ In order to allow us to read index files of any algorithm, let's set up
+ the .git directory gently so that we default to the algorithm for the
+ current repository, and add an --object-format option to allow users to
+ override this setting and continue to run show-index outside of a
+ repository altogether. Let's also document this new option so that
+ people can find it and use it.
Signed-off-by: brian m. carlson [off-list ref]
+ ## Documentation/git-show-index.txt ##
+@@ Documentation/git-show-index.txt: git-show-index - Show packed archive index
+ SYNOPSIS
+ --------
+ [verse]
+-'git show-index'
++'git show-index' [--object-format=<hash-algorithm>]
+
+
+ DESCRIPTION
+@@ Documentation/git-show-index.txt: Note that you can get more information on a packfile by calling
+ linkgit:git-verify-pack[1]. However, as this command considers only the
+ index file itself, it's both faster and more flexible.
+
++OPTIONS
++-------
++
++--object-format=<hash-algorithm>::
++ Specify the given object format (hash algorithm) for the index file. The
++ valid values are 'sha1' and (if enabled) 'sha256'. The default is the
++ algorithm for the current repository (set by `extensions.objectFormat`), or
++ 'sha1' if no value is set or outside a repository..
++
+ GIT
+ ---
+ Part of the linkgit:git[1] suite
+
## builtin/show-index.c ##
@@
#include "builtin.h"
@@ builtin/show-index.c
-static const char show_index_usage[] =
-"git show-index";
+static const char *const show_index_usage[] = {
-+ "git show-index [--hash=HASH]",
++ "git show-index [--object-format=<hash-algorithm>]",
+ NULL
+};
@@ builtin/show-index.c: int cmd_show_index(int argc, const char **argv, const char
+ const char *hash_name = NULL;
+ int hash_algo;
+ const struct option show_index_options[] = {
-+ OPT_STRING(0, "hash", &hash_name, N_("hash"),
++ OPT_STRING(0, "object-format", &hash_name, N_("hash-algorithm"),
+ N_("specify the hash algorithm to use")),
+ OPT_END()
+ };
28: bb3d2f566a = 27: 9f7c7bafaf t1302: expect repo format version 1 for SHA-256
29: cc25069cb6 = 28: d0ea597d63 Documentation/technical: document object-format for protocol v2
30: efdac6383f ! 29: 51848df542 connect: pass full packet reader when parsing v2 refs
@@ connect.c: static int process_ref_v2(const char *line, struct ref ***list)
/*
* Ref lines have a number of fields which are space deliminated. The
@@ connect.c: struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,
- }
- packet_flush(fd_out);
-+
/* Process response from server */
while (packet_reader_read(reader) == PACKET_READ_NORMAL) {
- if (!process_ref_v2(reader->line, &list))
31: 602405e436 ! 30: b57361f3b8 connect: parse v2 refs with correct hash algorithm
@@ Commit message
set that value in the packet reader. Parse the refs using this
algorithm.
- Note that we use memcpy instead of oidcpy for copying values, since
- oidcpy is intentionally limited to the current hash algorithm length,
- and the copy will be too short if the server side uses SHA-256 but the
- client side has not had a repository set up (and therefore defaults to
- SHA-1).
-
Signed-off-by: brian m. carlson [off-list ref]
## connect.c ##
+@@ connect.c: static int process_ref(const struct packet_reader *reader, int len,
+ die(_("protocol error: unexpected capabilities^{}"));
+ } else if (check_ref(name, flags)) {
+ struct ref *ref = alloc_ref(name);
+- memcpy(ref->old_oid.hash, old_oid.hash, reader->hash_algo->rawsz);
++ oidcpy(&ref->old_oid, &old_oid);
+ **list = ref;
+ *list = &ref->next;
+ }
@@ connect.c: static int process_ref_v2(struct packet_reader *reader, struct ref ***list)
goto out;
}
32: 96236ac9ae ! 31: a0c0f0f7a3 serve: advertise object-format capability for protocol v2
@@ Commit message
In the test, when we're using an algorithm other than SHA-1, we need to
specify the algorithm in use so we don't get a failure with an "unknown
- format" message. Add a wrapper function that specifies this header if
- required. Skip specifying this header for SHA-1 to test that it works
- both with and without this header.
+ format" message. Add a test that we handle a mismatched algorithm.
+ Remove the test_oid_init call since it's no longer necessary.
Signed-off-by: brian m. carlson [off-list ref]
+ ## connect.c ##
+@@ connect.c: struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,
+ die(_("unknown object format '%s' specified by server"), hash_name);
+ reader->hash_algo = &hash_algos[hash_algo];
+ packet_write_fmt(fd_out, "object-format=%s", reader->hash_algo->name);
++ } else {
++ reader->hash_algo = &hash_algos[GIT_HASH_SHA1];
+ }
+
+ if (server_options && server_options->nr &&
+
## serve.c ##
@@ serve.c: static int agent_advertise(struct repository *r,
return 1;
@@ serve.c: static int process_request(void)
## t/t5701-git-serve.sh ##
@@ t/t5701-git-serve.sh: test_description='test protocol v2 server commands'
-
. ./test-lib.sh
-+write_command () {
-+ echo "command=$1"
-+
-+ if test "$(test_oid algo)" != sha1
-+ then
-+ echo "object-format=$(test_oid algo)"
-+ fi
-+}
-+
test_expect_success 'test capability advertisement' '
-+ test_oid_init &&
++ test_oid_cache <<-EOF &&
++ wrong_algo sha1:sha256
++ wrong_algo sha256:sha1
++ EOF
cat >expect <<-EOF &&
version 2
agent=git/$(git version | cut -d" " -f3)
@@ t/t5701-git-serve.sh: test_expect_success 'request invalid capability' '
EOF
test_must_fail test-tool serve-v2 --stateless-rpc 2>err <in &&
@@ t/t5701-git-serve.sh: test_expect_success 'request with no command' '
-
test_expect_success 'request invalid command' '
test-tool pkt-line pack >in <<-EOF &&
-- command=foo
-+ $(write_command foo)
+ command=foo
++ object-format=$(test_oid algo)
agent=git/test
0000
EOF
-@@ t/t5701-git-serve.sh: test_expect_success 'setup some refs and tags' '
+@@ t/t5701-git-serve.sh: test_expect_success 'request invalid command' '
+ test_i18ngrep "invalid command" err
+ '
++test_expect_success 'wrong object-format' '
++ test-tool pkt-line pack >in <<-EOF &&
++ command=fetch
++ agent=git/test
++ object-format=$(test_oid wrong_algo)
++ 0000
++ EOF
++ test_must_fail test-tool serve-v2 --stateless-rpc 2>err <in &&
++ test_i18ngrep "mismatched object format" err
++'
++
+ # Test the basics of ls-refs
+ #
+ test_expect_success 'setup some refs and tags' '
+@@ t/t5701-git-serve.sh: test_expect_success 'setup some refs and tags' '
test_expect_success 'basics of ls-refs' '
test-tool pkt-line pack >in <<-EOF &&
-- command=ls-refs
-+ $(write_command ls-refs)
+ command=ls-refs
++ object-format=$(test_oid algo)
0000
EOF
@@ t/t5701-git-serve.sh: test_expect_success 'basics of ls-refs' '
-
test_expect_success 'basic ref-prefixes' '
test-tool pkt-line pack >in <<-EOF &&
-- command=ls-refs
-+ $(write_command ls-refs)
+ command=ls-refs
++ object-format=$(test_oid algo)
0001
ref-prefix refs/heads/master
ref-prefix refs/tags/one
@@ t/t5701-git-serve.sh: test_expect_success 'basic ref-prefixes' '
-
test_expect_success 'refs/heads prefix' '
test-tool pkt-line pack >in <<-EOF &&
-- command=ls-refs
-+ $(write_command ls-refs)
+ command=ls-refs
++ object-format=$(test_oid algo)
0001
ref-prefix refs/heads/
0000
@@ t/t5701-git-serve.sh: test_expect_success 'refs/heads prefix' '
-
test_expect_success 'peel parameter' '
test-tool pkt-line pack >in <<-EOF &&
-- command=ls-refs
-+ $(write_command ls-refs)
+ command=ls-refs
++ object-format=$(test_oid algo)
0001
peel
ref-prefix refs/tags/
@@ t/t5701-git-serve.sh: test_expect_success 'peel parameter' '
-
test_expect_success 'symrefs parameter' '
test-tool pkt-line pack >in <<-EOF &&
-- command=ls-refs
-+ $(write_command ls-refs)
+ command=ls-refs
++ object-format=$(test_oid algo)
0001
symrefs
ref-prefix refs/heads/
@@ t/t5701-git-serve.sh: test_expect_success 'symrefs parameter' '
-
test_expect_success 'sending server-options' '
test-tool pkt-line pack >in <<-EOF &&
-- command=ls-refs
-+ $(write_command ls-refs)
+ command=ls-refs
++ object-format=$(test_oid algo)
server-option=hello
server-option=world
0001
@@ t/t5701-git-serve.sh: test_expect_success 'unexpected lines are not allowed in fetch request' '
- git init server &&
test-tool pkt-line pack >in <<-EOF &&
-- command=fetch
-+ $(write_command fetch)
+ command=fetch
++ object-format=$(test_oid algo)
0001
this-is-not-a-command
0000
33: 57f3bbb709 = 32: 1694f3f838 t5500: make hash independent
34: 8242e65747 ! 33: 902b394667 builtin/ls-remote: initialize repository based on fetch
@@ Commit message
the refs to 40 hex characters, since that's the length of the default
hash algorithm (SHA-1).
+ Note that technically this is not a correct setting of the repository
+ hash algorithm since, if we are in a repository, it might be one of a
+ different hash algorithm from the remote side. However, our current
+ code paths don't handle multiple algorithms and won't for some time, so
+ this is the best we can do. We rely on the fact that ls-remote never
+ modifies the current repository, which is a reasonable assumption to
+ make.
+
Signed-off-by: brian m. carlson [off-list ref]
## builtin/ls-remote.c ##
35: c5664b646f ! 34: cc12b9b51f remote-curl: detect algorithm for dumb HTTP by size
@@ Commit message
provide one. Detect the hash algorithm in use by the size of the first
object ID.
+ We anonymize the URL like elsewhere in the function in case the user has
+ decided to include a secret in the URL.
+
Signed-off-by: brian m. carlson [off-list ref]
## remote-curl.c ##
@@ remote-curl.c: static struct ref *parse_info_refs(struct discovery *heads)
+ if (!options.hash_algo)
+ die("%sinfo/refs not valid: could not determine hash algorithm; "
+ "is this a git repository?",
-+ url.buf);
++ transport_anonymize_url(url.buf));
+
data = heads->buf;
start = NULL;
36: 31cd59a221 < -: ---------- builtin/index-pack: add option to specify hash algorithm
-: ---------- > 35: b5425c9f54 builtin/index-pack: add option to specify hash algorithm
37: 658b787e8c = 36: 5c70c24d7a t1050: pass algorithm to index-pack when outside repo
38: 64429337ba = 37: 460d6008e8 remote-curl: avoid truncating refs with ls-remote
39: cde2128520 = 38: 60a98d9b53 t/helper: initialize the repository for test-sha1-array
40: 0af00a7681 = 39: b66c3ead37 t5702: offer an object-format capability in the test
41: 74278d4c1c = 40: af43274a1f t5703: use object-format serve option
-: ---------- > 41: f5085b1f3f t5704: send object-format capability with SHA-256
42: 4f735c8bb5 = 42: a1b01babda t5300: pass --object-format to git index-pack
43: 3854f70427 = 43: dbb5f7195e bundle: detect hash algorithm when reading refs
44: 103be1f4d6 = 44: 6c823bbe68 remote-testgit: adapt for object-format
From: brian m. carlson <hidden> Date: 2020-05-25 19:59:50
When parsing capabilities for the pack protocol, there are times we'll
want to compare the value of a capability to a NUL-terminated string.
Since the data we're reading will be space-terminated, not
NUL-terminated, we need a function that compares the two strings, but
also checks that they're the same length. Otherwise, if we used strncmp
to compare these strings, we might accidentally accept a parameter that
was a prefix of the expected value.
Add a function, xstrncmpz, that takes a NUL-terminated string and a
non-NUL-terminated string, plus a length, and compares them, ensuring
that they are the same length.
Signed-off-by: brian m. carlson <redacted>
---
git-compat-util.h | 6 ++++++
wrapper.c | 8 ++++++++
2 files changed, 14 insertions(+)
From: brian m. carlson <hidden> Date: 2020-05-25 19:59:54
In a future patch, we'll want to access multiple members from struct
packet_reader when parsing references. Therefore, have the ref parsing
code take pointers to struct reader instead of having to pass multiple
arguments to each function.
Rename the len variable to "linelen" to make it clearer what the
variable does in light of the variable change.
Signed-off-by: brian m. carlson <redacted>
---
connect.c | 27 ++++++++++++++++-----------
1 file changed, 16 insertions(+), 11 deletions(-)
From: brian m. carlson <hidden> Date: 2020-05-25 19:59:55
If we're fetching refs, detect the hash algorithm and parse the refs
using that algorithm.
As mentioned in the documentation, if multiple versions of the
object-format capability are provided, we use the first. No known
implementation supports multiple algorithms now, but they may in the
future.
Signed-off-by: brian m. carlson <redacted>
---
connect.c | 21 +++++++++++++++++----
1 file changed, 17 insertions(+), 4 deletions(-)
@@ -258,7 +271,7 @@ static int process_ref(const struct packet_reader *reader, int len,structobject_idold_oid;constchar*name;-if(parse_oid_hex(line,&old_oid,&name))+if(parse_oid_hex_algop(line,&old_oid,&name,reader->hash_algo))return0;if(*name!=' ')return0;
@@ -270,7 +283,7 @@ static int process_ref(const struct packet_reader *reader, int len,die(_("protocol error: unexpected capabilities^{}"));}elseif(check_ref(name,flags)){structref*ref=alloc_ref(name);-oidcpy(&ref->old_oid,&old_oid);+memcpy(ref->old_oid.hash,old_oid.hash,reader->hash_algo->rawsz);**list=ref;*list=&ref->next;}
@@ -288,7 +301,7 @@ static int process_shallow(const struct packet_reader *reader, int len,if(!skip_prefix(line,"shallow ",&arg))return0;-if(get_oid_hex(arg,&old_oid))+if(get_oid_hex_algop(arg,&old_oid,reader->hash_algo))die(_("protocol error: expected shallow sha-1, got '%s'"),arg);if(!shallow_points)die(_("repository on the other end cannot be shallow"));
From: brian m. carlson <hidden> Date: 2020-05-25 19:59:57
Detect when the server doesn't support our hash algorithm and abort.
Signed-off-by: brian m. carlson <redacted>
---
fetch-pack.c | 2 ++
1 file changed, 2 insertions(+)
@@ -1040,6 +1040,8 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,print_verbose(args,_("Server supports %s"),"deepen-relative");elseif(args->deepen_relative)die(_("Server does not support --deepen"));+if(!server_supports_hash(the_hash_algo->name,NULL))+die(_("Server does not support this repository's object format"));if(!args->no_dependents){mark_complete_and_common_ref(negotiator,args,&ref);
From: brian m. carlson <hidden> Date: 2020-05-25 19:59:58
We're going to be using this function in other files, so no longer mark
this function static.
Signed-off-by: brian m. carlson <redacted>
---
connect.c | 3 +--
connect.h | 1 +
2 files changed, 2 insertions(+), 2 deletions(-)
From: brian m. carlson <hidden> Date: 2020-05-25 19:59:58
Update the remote helper docs to document the object-format extensions
we will implement in remote-curl and the transport helper code shortly.
Signed-off-by: brian m. carlson <redacted>
---
Documentation/gitremote-helpers.txt | 33 +++++++++++++++++++++++++----
1 file changed, 29 insertions(+), 4 deletions(-)
@@ -238,6 +238,9 @@ the remote repository. `--signed-tags=verbatim` to linkgit:git-fast-export[1]. In the absence of this capability, Git will use `--signed-tags=warn-strip`.+'object-format'::+ This indicates that the helper is able to interact with the remote+ side using an explicit hash algorithm extension. COMMANDS
@@ -257,12 +260,14 @@ Support for this command is mandatory. 'list':: Lists the refs, one per line, in the format "<value> <name> [<attr> ...]". The value may be a hex sha1 hash, "@<dest>" for- a symref, or "?" to indicate that the helper could not get the- value of the ref. A space-separated list of attributes follows- the name; unrecognized attributes are ignored. The list ends- with a blank line.+ a symref, ":<keyword> <value>" for a key-value pair, or+ "?" to indicate that the helper could not get the value of the+ ref. A space-separated list of attributes follows the name;+ unrecognized attributes are ignored. The list ends with a+ blank line. + See REF LIST ATTRIBUTES for a list of currently defined attributes.+See REF LIST KEYWORDS for a list of currently defined keywords. + Supported if the helper has the "fetch" or "import" capability.
@@ -430,6 +435,18 @@ attributes are defined. This ref is unchanged since the last import or fetch, although the helper cannot necessarily determine what value that produced.+REF LIST KEYWORDS+-----------------++The 'list' command may produce a list of key-value pairs.+The following keys are defined.++'object-format'::+ The refs are using the given hash algorithm. This keyword is only+ used if the server and client both support the object-format+ extension.++ OPTIONS -------
@@ -514,6 +531,14 @@ set by Git if the remote helper has the 'option' capability. transaction. If successful, all refs will be updated, or none will. If the remote side does not support this capability, the push will fail.+'option object-format' {'true'|algorithm}::+ If 'true', indicate that the caller wants hash algorithm information+ to be passed back from the remote. This mode is used when fetching+ refs.+++If set to an algorithm, indicate that the caller wants to interact with+the remote side using that algorithm.+ SEE ALSO -------- linkgit:git-remote[1]
From: brian m. carlson <hidden> Date: 2020-05-25 19:59:59
Detect when the server doesn't support our hash algorithm and abort.
If the server does support our hash, advertise it as part of our
capabilities.
Signed-off-by: brian m. carlson <redacted>
---
send-pack.c | 6 ++++++
1 file changed, 6 insertions(+)
@@ -363,6 +363,7 @@ int send_pack(struct send_pack_args *args,intatomic_supported=0;intuse_push_options=0;intpush_options_supported=0;+intobject_format_supported=0;unsignedcmds_sent=0;intret;structasyncdemux;
@@ -389,6 +390,9 @@ int send_pack(struct send_pack_args *args,if(server_supports("push-options"))push_options_supported=1;+if(!server_supports_hash(the_hash_algo->name,&object_format_supported))+die(_("the receiving end does not support this repository's hash algorithm"));+if(args->push_cert!=SEND_PACK_PUSH_CERT_NEVER){intlen;push_cert_nonce=server_feature_value("push-cert",&len);
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:02
When using SHA-256, we need to take advantage of the extensions section
in the config file, so we need to use repository format version 1.
Update the test to look for the correct value.
Note that test_oid produces a value without a trailing newline, so use
echo to ensure we print a trailing newline to compare it correctly
against the actual results.
Signed-off-by: brian m. carlson <redacted>
---
t/t1302-repo-version.sh | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:03
Our style these days is to place the description and the opening quote
of the body on the same line as test_expect_success (if it fits), to
place the trailing quote on a line by itself after the body, and to use
tabs. Since we're going to be making several significant changes to
this test, modernize the style to aid in readability of the subsequent
patches.
This patch should have no functional change.
Signed-off-by: brian m. carlson <redacted>
---
t/t5302-pack-index.sh | 360 +++++++++++++++++++++---------------------
1 file changed, 184 insertions(+), 176 deletions(-)
@@ -7,65 +7,65 @@ test_description='pack index with 64-bit offsets and object CRC' ../test-lib.sh test_expect_success'setup''-test_oid_init&&-rawsz=$(test_oidrawsz)&&-rm-rf.git&&-gitinit&&-gitconfigpack.threads1&&-i=1&&-whiletest$i-le100-do-iii=$(printf'%03i'$i)-test-toolgenrandom"bar"200>wide_delta_$iii&&-test-toolgenrandom"baz $iii"50>>wide_delta_$iii&&-test-toolgenrandom"foo"$i100>deep_delta_$iii&&-test-toolgenrandom"foo"$(expr$i+1)100>>deep_delta_$iii&&-test-toolgenrandom"foo"$(expr$i+2)100>>deep_delta_$iii&&-echo$iii>file_$iii&&-test-toolgenrandom"$iii"8192>>file_$iii&&-gitupdate-index--addfile_$iiideep_delta_$iiiwide_delta_$iii&&-i=$(expr$i+1)||return1-done&&-{echo101&&test-toolgenrandom1008192;}>file_101&&-gitupdate-index--addfile_101&&-tree=$(gitwrite-tree)&&-commit=$(gitcommit-tree$tree</dev/null)&&{-echo$tree&&-gitls-tree$tree|sed-e"s/.* \\([0-9a-f]*\\) .*/\\1/"-}>obj-list&&-gitupdate-refHEAD$commit+test_oid_init&&+rawsz=$(test_oidrawsz)&&+rm-rf.git&&+gitinit&&+gitconfigpack.threads1&&+i=1&&+whiletest$i-le100+do+iii=$(printf'%03i'$i)+test-toolgenrandom"bar"200>wide_delta_$iii&&+test-toolgenrandom"baz $iii"50>>wide_delta_$iii&&+test-toolgenrandom"foo"$i100>deep_delta_$iii&&+test-toolgenrandom"foo"$(expr$i+1)100>>deep_delta_$iii&&+test-toolgenrandom"foo"$(expr$i+2)100>>deep_delta_$iii&&+echo$iii>file_$iii&&+test-toolgenrandom"$iii"8192>>file_$iii&&+gitupdate-index--addfile_$iiideep_delta_$iiiwide_delta_$iii&&+i=$(expr$i+1)||return1+done&&+{echo101&&test-toolgenrandom1008192;}>file_101&&+gitupdate-index--addfile_101&&+tree=$(gitwrite-tree)&&+commit=$(gitcommit-tree$tree</dev/null)&&{+echo$tree&&+gitls-tree$tree|sed-e"s/.* \\([0-9a-f]*\\) .*/\\1/"+}>obj-list&&+gitupdate-refHEAD$commit'-test_expect_success\-'pack-objects with index version 1'\-'pack1=$(gitpack-objects--index-version=1test-1<obj-list)&&-gitverify-pack-v"test-1-${pack1}.pack"'+test_expect_success'pack-objects with index version 1''+pack1=$(gitpack-objects--index-version=1test-1<obj-list)&&+gitverify-pack-v"test-1-${pack1}.pack"+'-test_expect_success\-'pack-objects with index version 2'\-'pack2=$(gitpack-objects--index-version=2test-2<obj-list)&&-gitverify-pack-v"test-2-${pack2}.pack"'+test_expect_success'pack-objects with index version 2''+pack2=$(gitpack-objects--index-version=2test-2<obj-list)&&+gitverify-pack-v"test-2-${pack2}.pack"+'-test_expect_success\-'both packs should be identical'\-'cmp "test-1-${pack1}.pack" "test-2-${pack2}.pack"'+test_expect_success'both packs should be identical''+cmp"test-1-${pack1}.pack""test-2-${pack2}.pack"+'-test_expect_success\-'index v1 and index v2 should be different'\-'! cmp "test-1-${pack1}.idx" "test-2-${pack2}.idx"'+test_expect_success'index v1 and index v2 should be different''+!cmp"test-1-${pack1}.idx""test-2-${pack2}.idx"+'-test_expect_success\-'index-pack with index version 1'\-'git index-pack --index-version=1 -o 1.idx "test-1-${pack1}.pack"'+test_expect_success'index-pack with index version 1''+gitindex-pack--index-version=1-o1.idx"test-1-${pack1}.pack"+'-test_expect_success\-'index-pack with index version 2'\-'git index-pack --index-version=2 -o 2.idx "test-1-${pack1}.pack"'+test_expect_success'index-pack with index version 2''+gitindex-pack--index-version=2-o2.idx"test-1-${pack1}.pack"+'-test_expect_success\-'index-pack results should match pack-objects ones'\-'cmp"test-1-${pack1}.idx""1.idx"&&-cmp"test-2-${pack2}.idx""2.idx"'+test_expect_success'index-pack results should match pack-objects ones''+cmp"test-1-${pack1}.idx""1.idx"&&+cmp"test-2-${pack2}.idx""2.idx"+' test_expect_success'index-pack --verify on index version 1''gitindex-pack--verify"test-1-${pack1}.pack"
@@ -75,13 +75,13 @@ test_expect_success 'index-pack --verify on index version 2' 'gitindex-pack--verify"test-2-${pack2}.pack"'-test_expect_success\-'pack-objects --index-version=2, is not accepted'\-'test_must_fail git pack-objects --index-version=2, test-3 <obj-list'+test_expect_success'pack-objects --index-version=2, is not accepted''+test_must_failgitpack-objects--index-version=2,test-3<obj-list+'-test_expect_success\-'index v2: force some 64-bit offsets with pack-objects'\-'pack3=$(git pack-objects --index-version=2,0x40000 test-3 <obj-list)'+test_expect_success'index v2: force some 64-bit offsets with pack-objects''+pack3=$(gitpack-objects--index-version=2,0x40000test-3<obj-list)+'ifmsg=$(gitverify-pack-v"test-3-${pack3}.pack"2>&1)||!(echo"$msg"|grep"pack too large .* off_t")
@@ -91,21 +91,21 @@ elsesay"# skipping tests concerning 64-bit offsets"fi-test_expect_successOFF64_T\-'index v2: verify a pack with some 64-bit offsets'\-'git verify-pack -v "test-3-${pack3}.pack"'+test_expect_successOFF64_T'index v2: verify a pack with some 64-bit offsets''+gitverify-pack-v"test-3-${pack3}.pack"+'-test_expect_successOFF64_T\-'64-bit offsets: should be different from previous index v2 results'\-'! cmp "test-2-${pack2}.idx" "test-3-${pack3}.idx"'+test_expect_successOFF64_T'64-bit offsets: should be different from previous index v2 results''+!cmp"test-2-${pack2}.idx""test-3-${pack3}.idx"+'-test_expect_successOFF64_T\-'index v2: force some 64-bit offsets with index-pack'\-'git index-pack --index-version=2,0x40000 -o 3.idx "test-1-${pack1}.pack"'+test_expect_successOFF64_T'index v2: force some 64-bit offsets with index-pack''+gitindex-pack--index-version=2,0x40000-o3.idx"test-1-${pack1}.pack"+'-test_expect_successOFF64_T\-'64-bit offsets: index-pack result should match pack-objects one'\-'cmp "test-3-${pack3}.idx" "3.idx"'+test_expect_successOFF64_T'64-bit offsets: index-pack result should match pack-objects one''+cmp"test-3-${pack3}.idx""3.idx"+' test_expect_successOFF64_T'index-pack --verify on 64-bit offset v2 (cheat)''# This cheats by knowing which lower offset should still be encoded
@@ -120,135 +120,143 @@ test_expect_success OFF64_T 'index-pack --verify on 64-bit offset v2' '# returns the object number for given object in given pack index index_obj_nr(){-idx_file=$1-object_sha1=$2-nr=0-gitshow-index<$idx_file|-whilereadoffssha1extra-do-nr=$(($nr+1))-test"$sha1"="$object_sha1"||continue-echo"$(($nr-1))"-break-done+idx_file=$1+object_sha1=$2+nr=0+gitshow-index<$idx_file|+whilereadoffssha1extra+do+nr=$(($nr+1))+test"$sha1"="$object_sha1"||continue+echo"$(($nr-1))"+break+done}# returns the pack offset for given object as found in given pack index index_obj_offset(){-idx_file=$1-object_sha1=$2-gitshow-index<$idx_file|grep$object_sha1|-(readoffsextra&&echo"$offs")+idx_file=$1+object_sha1=$2+gitshow-index<$idx_file|grep$object_sha1|+(readoffsextra&&echo"$offs")}-test_expect_success\-'[index v1] 1) stream pack to repository'\-'gitindex-pack--index-version=1--stdin<"test-1-${pack1}.pack"&&-gitprune-packed&&-gitcount-objects|(readnrrest&&test"$nr"-eq1)&&-cmp"test-1-${pack1}.pack"".git/objects/pack/pack-${pack1}.pack"&&-cmp"test-1-${pack1}.idx"".git/objects/pack/pack-${pack1}.idx"'+test_expect_success'[index v1] 1) stream pack to repository''+gitindex-pack--index-version=1--stdin<"test-1-${pack1}.pack"&&+gitprune-packed&&+gitcount-objects|(readnrrest&&test"$nr"-eq1)&&+cmp"test-1-${pack1}.pack"".git/objects/pack/pack-${pack1}.pack"&&+cmp"test-1-${pack1}.idx"".git/objects/pack/pack-${pack1}.idx"+' test_expect_success\-'[index v1] 2) create a stealth corruption in a delta base reference'\-'# This test assumes file_101 is a delta smaller than 16 bytes.-# It should be against file_100 but we substitute its base for file_099-sha1_101=$(githash-objectfile_101)&&-sha1_099=$(githash-objectfile_099)&&-offs_101=$(index_obj_offset1.idx$sha1_101)&&-nr_099=$(index_obj_nr1.idx$sha1_099)&&-chmod+w".git/objects/pack/pack-${pack1}.pack"&&-recordsz=$((rawsz+4))&&-ddof=".git/objects/pack/pack-${pack1}.pack"seek=$(($offs_101+1))\-if=".git/objects/pack/pack-${pack1}.idx"\-skip=$((4+256*4+$nr_099*recordsz))\-bs=1count=$rawszconv=notrunc&&-gitcat-fileblob$sha1_101>file_101_foo1'+'[index v1] 2) create a stealth corruption in a delta base reference''+# This test assumes file_101 is a delta smaller than 16 bytes.+# It should be against file_100 but we substitute its base for file_099+sha1_101=$(githash-objectfile_101)&&+sha1_099=$(githash-objectfile_099)&&+offs_101=$(index_obj_offset1.idx$sha1_101)&&+nr_099=$(index_obj_nr1.idx$sha1_099)&&+chmod+w".git/objects/pack/pack-${pack1}.pack"&&+recordsz=$((rawsz+4))&&+ddof=".git/objects/pack/pack-${pack1}.pack"seek=$(($offs_101+1))\+if=".git/objects/pack/pack-${pack1}.idx"\+skip=$((4+256*4+$nr_099*recordsz))\+bs=1count=$rawszconv=notrunc&&+gitcat-fileblob$sha1_101>file_101_foo1+' test_expect_success\-'[index v1] 3) corrupted delta happily returned wrong data'\-'test -f file_101_foo1 && ! cmp file_101 file_101_foo1'+'[index v1] 3) corrupted delta happily returned wrong data''+test-ffile_101_foo1&&!cmpfile_101file_101_foo1+' test_expect_success\-'[index v1] 4) confirm that the pack is actually corrupted'\-'test_must_fail git fsck --full $commit'+'[index v1] 4) confirm that the pack is actually corrupted''+test_must_failgitfsck--full$commit+' test_expect_success\-'[index v1] 5) pack-objects happily reuses corrupted data'\-'pack4=$(gitpack-objectstest-4<obj-list)&&-test-f"test-4-${pack4}.pack"'+'[index v1] 5) pack-objects happily reuses corrupted data''+pack4=$(gitpack-objectstest-4<obj-list)&&+test-f"test-4-${pack4}.pack"+'++test_expect_success'[index v1] 6) newly created pack is BAD !''+test_must_failgitverify-pack-v"test-4-${pack4}.pack"+'++test_expect_success'[index v2] 1) stream pack to repository''+rm-f.git/objects/pack/*&&+gitindex-pack--index-version=2--stdin<"test-1-${pack1}.pack"&&+gitprune-packed&&+gitcount-objects|(readnrrest&&test"$nr"-eq1)&&+cmp"test-1-${pack1}.pack"".git/objects/pack/pack-${pack1}.pack"&&+cmp"test-2-${pack1}.idx"".git/objects/pack/pack-${pack1}.idx"+' test_expect_success\-'[index v1] 6) newly created pack is BAD !'\-'test_must_fail git verify-pack -v "test-4-${pack4}.pack"'+'[index v2] 2) create a stealth corruption in a delta base reference''+# This test assumes file_101 is a delta smaller than 16 bytes.+# It should be against file_100 but we substitute its base for file_099+sha1_101=$(githash-objectfile_101)&&+sha1_099=$(githash-objectfile_099)&&+offs_101=$(index_obj_offset1.idx$sha1_101)&&+nr_099=$(index_obj_nr1.idx$sha1_099)&&+chmod+w".git/objects/pack/pack-${pack1}.pack"&&+ddof=".git/objects/pack/pack-${pack1}.pack"seek=$(($offs_101+1))\+if=".git/objects/pack/pack-${pack1}.idx"\+skip=$((8+256*4+$nr_099*rawsz))\+bs=1count=$rawszconv=notrunc&&+gitcat-fileblob$sha1_101>file_101_foo2+' test_expect_success\-'[index v2] 1) stream pack to repository'\-'rm-f.git/objects/pack/*&&-gitindex-pack--index-version=2--stdin<"test-1-${pack1}.pack"&&-gitprune-packed&&-gitcount-objects|(readnrrest&&test"$nr"-eq1)&&-cmp"test-1-${pack1}.pack"".git/objects/pack/pack-${pack1}.pack"&&-cmp"test-2-${pack1}.idx"".git/objects/pack/pack-${pack1}.idx"'+'[index v2] 3) corrupted delta happily returned wrong data''+test-ffile_101_foo2&&!cmpfile_101file_101_foo2+' test_expect_success\-'[index v2] 2) create a stealth corruption in a delta base reference'\-'# This test assumes file_101 is a delta smaller than 16 bytes.-# It should be against file_100 but we substitute its base for file_099-sha1_101=$(githash-objectfile_101)&&-sha1_099=$(githash-objectfile_099)&&-offs_101=$(index_obj_offset1.idx$sha1_101)&&-nr_099=$(index_obj_nr1.idx$sha1_099)&&-chmod+w".git/objects/pack/pack-${pack1}.pack"&&-ddof=".git/objects/pack/pack-${pack1}.pack"seek=$(($offs_101+1))\-if=".git/objects/pack/pack-${pack1}.idx"\-skip=$((8+256*4+$nr_099*rawsz))\-bs=1count=$rawszconv=notrunc&&-gitcat-fileblob$sha1_101>file_101_foo2'+'[index v2] 4) confirm that the pack is actually corrupted''+test_must_failgitfsck--full$commit+' test_expect_success\-'[index v2] 3) corrupted delta happily returned wrong data'\-'test -f file_101_foo2 && ! cmp file_101 file_101_foo2'+'[index v2] 5) pack-objects refuses to reuse corrupted data''+test_must_failgitpack-objectstest-5<obj-list&&+test_must_failgitpack-objects--no-reuse-objecttest-6<obj-list+' test_expect_success\-'[index v2] 4) confirm that the pack is actually corrupted'\-'test_must_fail git fsck --full $commit'--test_expect_success\-'[index v2] 5) pack-objects refuses to reuse corrupted data'\-'test_must_failgitpack-objectstest-5<obj-list&&-test_must_failgitpack-objects--no-reuse-objecttest-6<obj-list'--test_expect_success\-'[index v2] 6) verify-pack detects CRC mismatch'\-'rm-f.git/objects/pack/*&&-gitindex-pack--index-version=2--stdin<"test-1-${pack1}.pack"&&-gitverify-pack".git/objects/pack/pack-${pack1}.pack"&&-obj=$(githash-objectfile_001)&&-nr=$(index_obj_nr".git/objects/pack/pack-${pack1}.idx"$obj)&&-chmod+w".git/objects/pack/pack-${pack1}.idx"&&-printfxxxx|ddof=".git/objects/pack/pack-${pack1}.idx"conv=notrunc\-bs=1count=4seek=$((8+256*4+$(wc-l<obj-list)*rawsz+$nr*4))&&-(whilereadobj-dogitcat-file-p$obj>/dev/null||exit1-done<obj-list)&&-test_must_failgitverify-pack".git/objects/pack/pack-${pack1}.pack"+'[index v2] 6) verify-pack detects CRC mismatch''+rm-f.git/objects/pack/*&&+gitindex-pack--index-version=2--stdin<"test-1-${pack1}.pack"&&+gitverify-pack".git/objects/pack/pack-${pack1}.pack"&&+obj=$(githash-objectfile_001)&&+nr=$(index_obj_nr".git/objects/pack/pack-${pack1}.idx"$obj)&&+chmod+w".git/objects/pack/pack-${pack1}.idx"&&+printfxxxx|ddof=".git/objects/pack/pack-${pack1}.idx"conv=notrunc\+bs=1count=4seek=$((8+256*4+$(wc-l<obj-list)*rawsz+$nr*4))&&+(whilereadobj+dogitcat-file-p$obj>/dev/null||exit1+done<obj-list)&&+test_must_failgitverify-pack".git/objects/pack/pack-${pack1}.pack"' test_expect_success'running index-pack in the object store''-rm-f.git/objects/pack/*&&-cptest-1-${pack1}.pack.git/objects/pack/pack-${pack1}.pack&&-(-cd.git/objects/pack&&-gitindex-packpack-${pack1}.pack-)&&-test-f.git/objects/pack/pack-${pack1}.idx+rm-f.git/objects/pack/*&&+cptest-1-${pack1}.pack.git/objects/pack/pack-${pack1}.pack&&+(+cd.git/objects/pack&&+gitindex-packpack-${pack1}.pack+)&&+test-f.git/objects/pack/pack-${pack1}.idx' test_expect_success'index-pack --strict warns upon missing tagger in tag''-sha=$(gitrev-parseHEAD)&&-cat>wrong-tag<<EOF&&+sha=$(gitrev-parseHEAD)&&+cat>wrong-tag<<EOF&& object$shatypecommit taggutentag
@@ -256,18 +264,18 @@ tag guten tag Thisisaninvalidtag. EOF-tag=$(githash-object-ttag-w--stdin<wrong-tag)&&-pack1=$(echo$tag$sha|gitpack-objectstag-test)&&-echoremovetagobject&&-thirtyeight=${tag#??}&&-rm-f.git/objects/${tag%$thirtyeight}/$thirtyeight&&-gitindex-pack--stricttag-test-${pack1}.pack2>err&&-grep"^warning:.* expected .tagger. line"err+tag=$(githash-object-ttag-w--stdin<wrong-tag)&&+pack1=$(echo$tag$sha|gitpack-objectstag-test)&&+echoremovetagobject&&+thirtyeight=${tag#??}&&+rm-f.git/objects/${tag%$thirtyeight}/$thirtyeight&&+gitindex-pack--stricttag-test-${pack1}.pack2>err&&+grep"^warning:.* expected .tagger. line"err' test_expect_success'index-pack --fsck-objects also warns upon missing tagger in tag''-gitindex-pack--fsck-objectstag-test-${pack1}.pack2>err&&-grep"^warning:.* expected .tagger. line"err+gitindex-pack--fsck-objectstag-test-${pack1}.pack2>err&&+grep"^warning:.* expected .tagger. line"err' test_done
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:04
This test has hard-coded pkt-lines with object IDs. The pkt-line
lengths necessarily differ between hash algorithms, so generate these
lines with the packetize helper so they're always the right size. In
addition, we will require an object-format capability for SHA-256, so
pass that capability on to the upload-pack process.
Signed-off-by: brian m. carlson <redacted>
---
t/t5500-fetch-pack.sh | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:05
When reading the info/refs file for a repository, we have no explicit
way to detect which hash algorithm is in use because the file doesn't
provide one. Detect the hash algorithm in use by the size of the first
object ID.
We anonymize the URL like elsewhere in the function in case the user has
decided to include a secret in the URL.
Signed-off-by: brian m. carlson <redacted>
---
remote-curl.c | 23 +++++++++++++++++++++--
1 file changed, 21 insertions(+), 2 deletions(-)
@@ -262,6 +275,12 @@ static struct ref *parse_info_refs(struct discovery *heads)structref*ref=NULL;structref*last_ref=NULL;+options.hash_algo=detect_hash_algo(heads);+if(!options.hash_algo)+die("%sinfo/refs not valid: could not determine hash algorithm; "+"is this a git repository?",+transport_anonymize_url(url.buf));+data=heads->buf;start=NULL;mid=data;
@@ -272,13 +291,13 @@ static struct ref *parse_info_refs(struct discovery *heads)if(data[i]=='\t')mid=&data[i];if(data[i]=='\n'){-if(mid-start!=the_hash_algo->hexsz)+if(mid-start!=options.hash_algo->hexsz)die(_("%sinfo/refs not valid: is this a git repository?"),transport_anonymize_url(url.buf));data[i]=0;ref_name=mid+1;ref=alloc_ref(ref_name);-get_oid_hex(start,&ref->old_oid);+get_oid_hex_algop(start,&ref->old_oid,options.hash_algo);if(!refs)refs=ref;if(last_ref)
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:07
git index-pack is usually run in a repository, but need not be. Since
packs don't contains information on the algorithm in use, instead
relying on context, add an option to index-pack to tell it which one
we're using in case someone runs it outside of a repository. Since
using --stdin necessarily implies a repository, don't allow specifying
an object format if it's provided to prevent users from passing an
option that won't work. Add documentation for this option.
Signed-off-by: brian m. carlson <redacted>
---
Documentation/git-index-pack.txt | 8 ++++++++
builtin/index-pack.c | 8 ++++++++
2 files changed, 16 insertions(+)
@@ -93,6 +93,14 @@ OPTIONS --max-input-size=<size>:: Die, if the pack is larger than <size>.+--object-format=<hash-algorithm>::+ Specify the given object format (hash algorithm) for the pack. The valid+ values are 'sha1' and (if enabled) 'sha256'. The default is the algorithm for+ the current repository (set by `extensions.objectFormat`), or 'sha1' if no+ value is set or outside a repository.+++This option cannot be used with --stdin.+ NOTES -----
@@ -1667,6 +1667,7 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)unsignedcharpack_hash[GIT_MAX_RAWSZ];unsignedforeign_nr=1;/* zero is a "good" value, assume bad */intreport_end_of_input=0;+inthash_algo=0;/**index-packneverneedstofetchmissingobjectsexceptwhen
@@ -1776,6 +1782,8 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)die(_("--fix-thin cannot be used without --stdin"));if(from_stdin&&!startup_info->have_repository)die(_("--stdin requires a git repository"));+if(from_stdin&&hash_algo)+die(_("--object-format cannot be used with --stdin"));if(!index_name&&pack_name)index_name=derive_filename(pack_name,"idx",&index_name_buf);
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:08
Much like with the dumb HTTP transport, there isn't a way to explicitly
specify the hash algorithm when dealing with a bundle, so detect the
algorithm based on the length of the object IDs in the prerequisites and
ref advertisements.
Signed-off-by: brian m. carlson <redacted>
---
bundle.c | 22 +++++++++++++++++++++-
bundle.h | 1 +
transport.c | 10 ++++++++--
3 files changed, 30 insertions(+), 3 deletions(-)
@@ -143,6 +143,9 @@ static struct ref *get_refs_from_bundle(struct transport *transport,data->fd=read_bundle_header(transport->url,&data->header);if(data->fd<0)die(_("could not read bundle '%s'"),transport->url);++transport->hash_algo=data->header.hash_algo;+for(i=0;i<data->header.references.nr;i++){structref_list_entry*e=data->header.references.list+i;structref*ref=alloc_ref(e->name);
@@ -157,11 +160,14 @@ static int fetch_refs_from_bundle(struct transport *transport,intnr_heads,structref**to_fetch){structbundle_transport_data*data=transport->data;+intret;if(!data->get_refs_from_bundle_called)get_refs_from_bundle(transport,0,NULL);-returnunbundle(the_repository,&data->header,data->fd,-transport->progress?BUNDLE_VERBOSE:0);+ret=unbundle(the_repository,&data->header,data->fd,+transport->progress?BUNDLE_VERBOSE:0);+transport->hash_algo=data->header.hash_algo;+returnret;}staticintclose_bundle(structtransport*transport)
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:09
In order to make this test work with SHA-256, offer an object-format
capability so that both sides use the same algorithm.
Signed-off-by: brian m. carlson <redacted>
---
t/t5702-protocol-v2.sh | 2 ++
1 file changed, 2 insertions(+)
@@ -13,6 +13,7 @@ start_git_daemon --export-all --enable=receive-packdaemon_parent=$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent test_expect_success'create repo to be served by git-daemon''+test_oid_init&&gitinit"$daemon_parent"&&test_commit-C"$daemon_parent"one'
@@ -394,6 +395,7 @@ test_expect_success 'even with handcrafted request, filter does not work if not# Custom request that tries to filter even though it is not advertised.test-toolpkt-linepack>in<<-EOF&&command=fetch+object-format=$(test_oidalgo)0001want$(git-Cserverrev-parsemaster)filterblob:none
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:12
When using an algorithm other than SHA-1, we need the remote helper to
advertise support for the object-format extension and provide
information back to us so that we can properly parse refs and return
data. Ensure that the test remote helper understands these extensions.
Signed-off-by: brian m. carlson <redacted>
---
t/t5801/git-remote-testgit | 6 ++++++
1 file changed, 6 insertions(+)
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:13
test-sha1-array uses the_hash_algo under the hood. Since t0064 wants to
use the value that is correct for the hash algorithm that we're testing,
make sure the test helper initializes the repository to set
the_hash_algo correctly.
Signed-off-by: brian m. carlson <redacted>
---
t/helper/test-oid-array.c | 3 +++
1 file changed, 3 insertions(+)
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:13
Normally, the remote-curl transport helper is aware of the hash
algorithm we're using because we're in a repo with the appropriate hash
algorithm set. However, when using git ls-remote outside of a
repository, we won't have initialized the hash algorithm properly, so
use hash_to_hex_algop to print the ref corresponding to the algorithm
we've detected.
Signed-off-by: brian m. carlson <redacted>
---
remote-curl.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:15
When outside a repository, git index-pack is unable to guess the hash
algorithm in use for a pack, since packs don't contain any information
on the algorithm in use. Pass an option to index-pack to help it out in
this test.
Signed-off-by: brian m. carlson <redacted>
---
t/t1050-large.sh | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
@@ -12,6 +12,7 @@ file_size () {} test_expect_successsetup'+test_oid_init&&# clone does not allow us to pass core.bigfilethreshold to# new repos, so set core.bigfilethreshold globallygitconfig--globalcore.bigfilethreshold200k&&
@@ -177,7 +178,8 @@ test_expect_success 'git-show a large file' ' test_expect_success'index-pack''gitclonefile://"$(pwd)"/.gitfoo&&-GIT_DIR=non-existentgitindex-pack--strict--verifyfoo/.git/objects/pack/*.pack+GIT_DIR=non-existentgitindex-pack--object-format=$(test_oidalgo)\+--strict--verifyfoo/.git/objects/pack/*.pack' test_expect_success'repack''
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:17
ls-remote may or may not operate within a repository, and as such will
not have been initialized with the repository's hash algorithm. Even if
it were, the remote side could be using a different algorithm and we
would still want to display those refs properly. Find the hash
algorithm used by the remote side by querying the transport object and
set our hash algorithm accordingly.
Without this change, if the remote side is using SHA-256, we truncate
the refs to 40 hex characters, since that's the length of the default
hash algorithm (SHA-1).
Note that technically this is not a correct setting of the repository
hash algorithm since, if we are in a repository, it might be one of a
different hash algorithm from the remote side. However, our current
code paths don't handle multiple algorithms and won't for some time, so
this is the best we can do. We rely on the fact that ls-remote never
modifies the current repository, which is a reasonable assumption to
make.
Signed-off-by: brian m. carlson <redacted>
---
builtin/ls-remote.c | 4 ++++
1 file changed, 4 insertions(+)
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:18
When using protocol v2, we need to know what hash algorithm is used by
the remote end. See if the server has sent us an object-format
capability, and if so, use it to determine the hash algorithm in use and
set that value in the packet reader. Parse the refs using this
algorithm.
Signed-off-by: brian m. carlson <redacted>
---
connect.c | 21 ++++++++++++++++-----
1 file changed, 16 insertions(+), 5 deletions(-)
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:19
Advertise the current hash algorithm in use by using the object-format
capability as part of the ref advertisement.
Signed-off-by: brian m. carlson <redacted>
---
builtin/receive-pack.c | 1 +
upload-pack.c | 3 ++-
2 files changed, 3 insertions(+), 1 deletion(-)
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:23
One of the test assertions in this test checks that git branch -m works
even without a .git/config file. However, if the repository requires
configuration extensions, such as because it uses a non-SHA-1 algorithm,
this assertion will fail. Mark the assertion as requiring SHA-1.
Signed-off-by: brian m. carlson <redacted>
---
t/t3200-branch.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:23
When we're parsing refs, we need to know not only what the line we're
parsing is, but also the hash algorithm we should use to parse it, which
is stored in the reader object. Pass the packet reader object through
to the protocol v2 ref parsing function.
Signed-off-by: brian m. carlson <redacted>
---
connect.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
@@ -374,7 +374,7 @@ struct ref **get_remote_heads(struct packet_reader *reader,}/* Returns 1 when a valid ref has been added to `list`, 0 otherwise */-staticintprocess_ref_v2(constchar*line,structref***list)+staticintprocess_ref_v2(structpacket_reader*reader,structref***list){intret=1;inti=0;
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:27
When connecting to a remote system, we need to know what hash algorithm
it will be using to talk to us. Add a hash_algo member to struct
transport and add a function to read this data from the transport
object.
Signed-off-by: brian m. carlson <redacted>
---
transport.c | 8 ++++++++
transport.h | 8 ++++++++
2 files changed, 16 insertions(+)
@@ -311,6 +311,7 @@ static struct ref *handshake(struct transport *transport, int for_push,BUG("unknown protocol version");}data->got_remote_heads=1;+transport->hash_algo=reader.hash_algo;if(reader.line_peeked)BUG("buffer must be empty at the end of handshake()");
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:27
Add a function, server_supports_hash, to see if the remote server
supports a particular hash algorithm when speaking protocol v1.
Signed-off-by: brian m. carlson <redacted>
---
connect.c | 22 ++++++++++++++++++++++
connect.h | 1 +
2 files changed, 23 insertions(+)
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:29
So far in protocol v2, all of our server capabilities that have values
have not had values that we've been interested in parsing. For example,
we receive but ignore the agent value.
However, in a future commit, we're going to want to parse out the value
of a server capability. To make this easy, add a function,
server_feature_v2, that can fetch the value provided as part of the
server capability.
Signed-off-by: brian m. carlson <redacted>
---
connect.c | 15 +++++++++++++++
connect.h | 1 +
2 files changed, 16 insertions(+)
@@ -84,6 +84,21 @@ int server_supports_v2(const char *c, int die_on_error)return0;}+intserver_feature_v2(constchar*c,constchar**v)+{+inti;++for(i=0;i<server_capabilities_v2.argc;i++){+constchar*out;+if(skip_prefix(server_capabilities_v2.argv[i],c,&out)&&+(*out=='=')){+*v=out+1;+return1;+}+}+return0;+}+intserver_supports_feature(constchar*c,constchar*feature,intdie_on_error){
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:30
Add a member for the hash algorithm currently in use to the packet
reader so it can parse references correctly.
Signed-off-by: brian m. carlson <redacted>
---
pkt-line.c | 1 +
pkt-line.h | 3 +++
2 files changed, 4 insertions(+)
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:32
In a capability response, we can have multiple symref entries. In the
future, we will also allow for multiple hash algorithms to be specified.
To avoid duplication, expand the parse_feature_value function to take an
optional offset where the parsing should begin next time. Add a wrapper
function that allows us to query the next server feature value, and use
it in the existing symref parsing code.
Signed-off-by: brian m. carlson <redacted>
---
connect.c | 30 +++++++++++++++++++++---------
1 file changed, 21 insertions(+), 9 deletions(-)
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:33
Parse the server's object-format capability and respond accordingly,
dying if there is a mismatch.
Signed-off-by: brian m. carlson <redacted>
---
fetch-pack.c | 12 ++++++++++++
1 file changed, 12 insertions(+)
@@ -1180,6 +1180,7 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,intsideband_all,intseen_ack){intret=0;+constchar*hash_name;structstrbufreq_buf=STRBUF_INIT;if(server_supports_v2("fetch",1))
@@ -1194,6 +1195,17 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,args->server_options->items[i].string);}+if(server_feature_v2("object-format",&hash_name)){+inthash_algo=hash_algo_by_name(hash_name);+if(hash_algo_by_ptr(the_hash_algo)!=hash_algo)+die(_("mismatched algorithms: client %s; server %s"),+the_hash_algo->name,hash_name);+packet_write_fmt(fd_out,"object-format=%s",the_hash_algo->name);+}elseif(hash_algo_by_ptr(the_hash_algo)!=GIT_HASH_SHA1){+die(_("the server does not support algorithm '%s'"),+the_hash_algo->name);+}+packet_buf_delim(&req_buf);if(args->use_thin_pack)packet_buf_write(&req_buf,"thin-pack");
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:34
When performing a clone, we don't know what hash algorithm the other end
will support. Currently, we don't support fetching data belonging to a
different algorithm, so we must know what algorithm the remote side is
using in order to properly initialize the repository. We can know that
only after fetching the refs, so if the remote side has any references,
use that information to reinitialize the repository with the correct
hash algorithm information.
Signed-off-by: brian m. carlson <redacted>
---
builtin/clone.c | 9 +++++++++
1 file changed, 9 insertions(+)
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:35
Ensure that we pass the object-format capability in the synthesized test
data so that this test works with algorithms other than SHA-1.
In addition, add an additional test using the old data for when we're
using SHA-1 so that we can be sure that we preserve backwards
compatibility with servers not offering the object-format capability.
Signed-off-by: brian m. carlson <redacted>
---
t/t5562-http-backend-content-length.sh | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:36
When we're using an algorithm other than SHA-1, we need to specify the
algorithm in use so we don't get a failure with an "unknown format"
message. Add a wrapper function that specifies this header if required.
Skip specifying this header for SHA-1 to test that it works both with an
without this header.
Signed-off-by: brian m. carlson <redacted>
---
t/t5703-upload-pack-ref-in-want.sh | 19 ++++++++++++++-----
1 file changed, 14 insertions(+), 5 deletions(-)
@@ -129,7 +138,7 @@ test_expect_success 'mix want and want-ref' 'gitrev-parseef>expected_commits&&test-toolpkt-linepack>in<<-EOF&&-command=fetch+$(write_commandfetch)0001no-progresswant-refrefs/heads/master
@@ -152,7 +161,7 @@ test_expect_success 'want-ref with ref we already have commit for' 'oid=$(gitrev-parsec)&&test-toolpkt-linepack>in<<-EOF&&-command=fetch+$(write_commandfetch)0001no-progresswant-refrefs/heads/o/foo
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:37
When we speak protocol v2 in this test, we must pass the object-format
header if the algorithm is not SHA-1. Otherwise, git upload-pack fails
because the hash algorithm doesn't match and not because we've failed to
speak the protocol correctly. Pass the header so that our assertions
test what we're really interested in.
Signed-off-by: brian m. carlson <redacted>
---
t/t5704-protocol-violations.sh | 2 ++
1 file changed, 2 insertions(+)
@@ -9,6 +9,7 @@ making sure that we do not segfault or otherwise behave badly.' test_expect_success'extra delim packet in v2 ls-refs args''{packetizecommand=ls-refs&&+packetize"object-format=$(test_oidalgo)"&&printf0001&&# protocol expects 0000 flush hereprintf0001
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:39
git index-pack by default reads the repository to determine the object
format. However, when outside of a repository, it's necessary to specify
the hash algorithm in use so that the pack can be properly indexed. Add
an --object-format argument when invoking git index-pack outside of a
repository.
Signed-off-by: brian m. carlson <redacted>
---
t/t5300-pack-object.sh | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
@@ -12,7 +12,8 @@ TRASH=$(pwd) test_expect_success\'setup'\-'rm-f.git/index*&&+'test_oid_init&&+rm-f.git/index*&&perl-e"print \"a\" x 4096;">a&&perl-e"print \"b\" x 4096;">b&&perl-e"print \"c\" x 4096;">c&&
@@ -412,18 +413,18 @@ test_expect_success 'set up pack for non-repo tests' '' test_expect_success'index-pack --stdin complains of non-repo''-nongittest_must_failgitindex-pack--stdin<foo.pack&&+nongittest_must_failgitindex-pack--object-format=$(test_oidalgo)--stdin<foo.pack&&test_path_is_missingnon-repo/.git' test_expect_success'index-pack <pack> works in non-repo''-nongitgitindex-pack../foo.pack&&+nongitgitindex-pack--object-format=$(test_oidalgo)../foo.pack&&test_path_is_filefoo.idx' test_expect_success'index-pack --strict <pack> works in non-repo''rm-ffoo.idx&&-nongitgitindex-pack--strict../foo.pack&&+nongitgitindex-pack--strict--object-format=$(test_oidalgo)../foo.pack&&test_path_is_filefoo.idx'
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:40
In order to communicate the protocol supported by the server side, add
support for advertising the object-format capability. We check that the
client side sends us an identical algorithm if it sends us its own
object-format capability, and assume it speaks SHA-1 if not.
In the test, when we're using an algorithm other than SHA-1, we need to
specify the algorithm in use so we don't get a failure with an "unknown
format" message. Add a test that we handle a mismatched algorithm.
Remove the test_oid_init call since it's no longer necessary.
Signed-off-by: brian m. carlson <redacted>
---
connect.c | 2 ++
serve.c | 27 +++++++++++++++++++++++++++
t/t5701-git-serve.sh | 25 +++++++++++++++++++++++++
3 files changed, 54 insertions(+)
@@ -45,6 +50,7 @@ test_expect_success 'request invalid capability' ' test_expect_success'request with no command''test-toolpkt-linepack>in<<-EOF&&agent=git/test+object-format=$(test_oidalgo)0000EOFtest_must_failtest-toolserve-v2--stateless-rpc2>err<in&&
@@ -54,6 +60,7 @@ test_expect_success 'request with no command' ' test_expect_success'request invalid command''test-toolpkt-linepack>in<<-EOF&&command=foo+object-format=$(test_oidalgo)agent=git/test0000EOF
@@ -61,6 +68,17 @@ test_expect_success 'request invalid command' 'test_i18ngrep"invalid command"err'+test_expect_success'wrong object-format''+test-toolpkt-linepack>in<<-EOF&&+command=fetch+agent=git/test+object-format=$(test_oidwrong_algo)+0000+EOF+test_must_failtest-toolserve-v2--stateless-rpc2>err<in&&+test_i18ngrep"mismatched object format"err+'+# Test the basics of ls-refs# test_expect_success'setup some refs and tags''
@@ -74,6 +92,7 @@ test_expect_success 'setup some refs and tags' ' test_expect_success'basics of ls-refs''test-toolpkt-linepack>in<<-EOF&&command=ls-refs+object-format=$(test_oidalgo)0000EOF
@@ -200,6 +224,7 @@ test_expect_success 'unexpected lines are not allowed in fetch request' 'test-toolpkt-linepack>in<<-EOF&&command=fetch+object-format=$(test_oidalgo)0001this-is-not-a-command0000
@@ -453,3 +453,12 @@ included in a request. This is done by sending each option as a a request. The provided options must not contain a NUL or LF character.++ object-format+~~~~~~~~~~~~~~~++The server can advertise the `object-format` capability with a value `X` (in the+form `object-format=X`) to notify the client that the server is able to deal+with objects using hash algorithm X. If not specified, the server is assumed to+only handle SHA-1. If the client would like to use a hash algorithm other than+SHA-1, it should specify its object-format string.
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:43
show-index is capable of reading any possible index file whether or not
the index is inside a repository. However, because our index files lack
metadata about the hash algorithm in use, it's not possible to
autodetect the algorithm that a particular index file is using.
In order to allow us to read index files of any algorithm, let's set up
the .git directory gently so that we default to the algorithm for the
current repository, and add an --object-format option to allow users to
override this setting and continue to run show-index outside of a
repository altogether. Let's also document this new option so that
people can find it and use it.
Signed-off-by: brian m. carlson <redacted>
---
Documentation/git-show-index.txt | 11 ++++++++++-
builtin/show-index.c | 29 ++++++++++++++++++++++++-----
git.c | 2 +-
3 files changed, 35 insertions(+), 7 deletions(-)
@@ -9,7 +9,7 @@ git-show-index - Show packed archive index SYNOPSIS -------- [verse]-'git show-index'+'git show-index' [--object-format=<hash-algorithm>] DESCRIPTION
@@ -36,6 +36,15 @@ Note that you can get more information on a packfile by calling linkgit:git-verify-pack[1]. However, as this command considers only the index file itself, it's both faster and more flexible.+OPTIONS+-------++--object-format=<hash-algorithm>::+ Specify the given object format (hash algorithm) for the index file. The+ valid values are 'sha1' and (if enabled) 'sha256'. The default is the+ algorithm for the current repository (set by `extensions.objectFormat`), or+ 'sha1' if no value is set or outside a repository..+ GIT --- Part of the linkgit:git[1] suite
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:45
Both v2 pack index files and the v3 format specified as part of the
NewHash work have similar data starting at the CRC table. Much of the
existing code wants to read either this table or the offset entries
following it, and in doing so computes the offset each time.
In order to share as much code between v2 and v3, compute the offset of
the CRC table and store it when the pack is opened. Use this value to
compute offsets to not only the CRC table, but to the offset entries
beyond it.
Signed-off-by: brian m. carlson <redacted>
---
builtin/index-pack.c | 6 +-----
object-store.h | 1 +
packfile.c | 1 +
3 files changed, 3 insertions(+), 5 deletions(-)
@@ -1555,13 +1555,9 @@ static void read_v2_anomalous_offsets(struct packed_git *p,{constuint32_t*idx1,*idx2;uint32_ti;-constuint32_thashwords=the_hash_algo->rawsz/sizeof(uint32_t);/* The address of the 4-byte offset table */-idx1=(((constuint32_t*)p->index_data)-+2/* 8-byte header */-+256/* fan out */-+hashwords*p->num_objects/* object ID table */+idx1=(((constuint32_t*)((constuint8_t*)p->index_data+p->crc_offset))+p->num_objects/* CRC32 table */);
@@ -178,6 +178,7 @@ int load_idx(const char *path, const unsigned int hashsz, void *idx_map,*/(sizeof(off_t)<=4))returnerror("pack too large for current definition of off_t in %s",path);+p->crc_offset=8+4*256+nr*hashsz;}p->index_version=version;
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:46
When we're checking the repository's format, set the hash algorithm at
the same time. This ensures that we perform a suitable initialization
early enough to avoid confusing any parts of the code. If we defer
until later, we can end up with portions of the code which are confused
about the hash algorithm, resulting in segfaults when working with
SHA-256 repositories.
Signed-off-by: brian m. carlson <redacted>
---
setup.c | 1 +
1 file changed, 1 insertion(+)
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:47
Detect when the server doesn't support our hash algorithm and abort.
Signed-off-by: brian m. carlson <redacted>
---
builtin/receive-pack.c | 9 +++++++++
1 file changed, 9 insertions(+)
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:48
Implement the object-format extensions that let us determine the hash
algorithm in use when pushing, pulling, and fetching.
Signed-off-by: brian m. carlson <redacted>
---
remote-curl.c | 19 ++++++++++++++++++-
1 file changed, 18 insertions(+), 1 deletion(-)
From: brian m. carlson <hidden> Date: 2020-05-25 20:00:50
Implement the object-format extensions that let us determine the hash
algorithm in use when pushing or pulling data.
Signed-off-by: brian m. carlson <redacted>
---
transport-helper.c | 24 ++++++++++++++++++++++--
1 file changed, 22 insertions(+), 2 deletions(-)
From: brian m. carlson <hidden> Date: 2020-06-19 17:56:17
Document a capability that indicates which hash algorithms are in use by
both sides of a remote connection. Use the term "object-format", since
this is the term used for the repository extension as well.
Signed-off-by: brian m. carlson <redacted>
---
Documentation/technical/protocol-capabilities.txt | 15 +++++++++++++++
1 file changed, 15 insertions(+)
@@ -176,6 +176,21 @@ agent strings are purely informative for statistics and debugging purposes, and MUST NOT be used to programmatically assume the presence or absence of particular features.+object-format+-------------++This capability, which takes a hash algorithm as an argument, indicates+that the server supports the given hash algorithms. It may be sent+multiple times; if so, the first one given is the one used in the ref+advertisement.++When provided by the client, this indicates that it intends to use the+given hash algorithm to communicate. The algorithm provided must be one+that the server supports.++If this capability is not provided, it is assumed that the only+supported algorithm is SHA-1.+ symref ------
From: brian m. carlson <hidden> Date: 2020-06-19 17:56:18
This is part 2 of 3 of the SHA-256 work. This series adds all of the
protocol logic to work with SHA-256 repositories.
v3 fixes a bug in patch 34 which prevented cloning an empty repository
with the dumb HTTP protocol. We look up the hash algorithm by length of
the data in the info/refs file and if we have no refs, we have no
entries.
Previously, we just failed and complained, which isn't really helpful,
nor is it backward compatible. So now we use whatever the default is
for the current repository. That means we honor GIT_DEFAULT_HASH or git
clone -c, and default to SHA-1 otherwise. Users are encouraged to
switch to the smart protocol if they need to distinguish the remote
side's hash algorithm when the repository is empty.
There are tests for the default hash behavior, but not for git clone -c,
because the extensions.objectformat option doesn't exist yet. I have
tested that it does indeed work, though.
Otherwise, this series is the same as v2 except for a rebase (for my
convenience and Junio's).
Changes from v2:
* Rebase onto master.
* Fix cloning an empty repository with the dumb HTTP protocol.
Changes from v1:
* Fix spurious line additions and deletions.
* Rename len to linelen for easier understanding.
* Move the documentation comment for xstrncmpz to the header.
* Drop a useless variable (found).
* Update several commit messages to better explain things as suggested
by Junio and Martin.
* Name the parameters for parse_feature_value for better documentation.
* Reduce the scope of variables when possible.
* Add explicit handling for missing object-format capabilities.
* Rename all new options to --object-format.
* Use oidcpy where possible.
* Test more failure cases.
* Have index-pack fail if --stdin and --object-format are both
specified.
* Move and simplify t5704.
* Other miscellaneous cleanups to respond to review feedback.
Range-diff below.
brian m. carlson (44):
t1050: match object ID paths in a hash-insensitive way
Documentation: document v1 protocol object-format capability
connect: have ref processing code take struct packet_reader
wrapper: add function to compare strings with different NUL
termination
remote: advertise the object-format capability on the server side
connect: add function to parse multiple v1 capability values
connect: add function to fetch value of a v2 server capability
pkt-line: add a member for hash algorithm
transport: add a hash algorithm member
connect: add function to detect supported v1 hash functions
send-pack: detect when the server doesn't support our hash
connect: make parse_feature_value extern
fetch-pack: detect when the server doesn't support our hash
connect: detect algorithm when fetching refs
builtin/receive-pack: detect when the server doesn't support our hash
docs: update remote helper docs for object-format extensions
transport-helper: implement object-format extensions
remote-curl: implement object-format extensions
builtin/clone: initialize hash algorithm properly
t5562: pass object-format in synthesized test data
fetch-pack: parse and advertise the object-format capability
setup: set the_repository's hash algo when checking format
t3200: mark assertion with SHA1 prerequisite
packfile: compute and use the index CRC offset
t5302: modernize test formatting
builtin/show-index: provide options to determine hash algo
t1302: expect repo format version 1 for SHA-256
Documentation/technical: document object-format for protocol v2
connect: pass full packet reader when parsing v2 refs
connect: parse v2 refs with correct hash algorithm
serve: advertise object-format capability for protocol v2
t5500: make hash independent
builtin/ls-remote: initialize repository based on fetch
remote-curl: detect algorithm for dumb HTTP by size
builtin/index-pack: add option to specify hash algorithm
t1050: pass algorithm to index-pack when outside repo
remote-curl: avoid truncating refs with ls-remote
t/helper: initialize the repository for test-sha1-array
t5702: offer an object-format capability in the test
t5703: use object-format serve option
t5704: send object-format capability with SHA-256
t5300: pass --object-format to git index-pack
bundle: detect hash algorithm when reading refs
remote-testgit: adapt for object-format
Documentation/git-index-pack.txt | 8 +
Documentation/git-show-index.txt | 11 +-
Documentation/gitremote-helpers.txt | 33 +-
.../technical/protocol-capabilities.txt | 15 +
Documentation/technical/protocol-v2.txt | 9 +
builtin/clone.c | 9 +
builtin/index-pack.c | 14 +-
builtin/ls-remote.c | 4 +
builtin/receive-pack.c | 10 +
builtin/show-index.c | 29 +-
bundle.c | 22 +-
bundle.h | 1 +
connect.c | 138 +++++--
connect.h | 3 +
fetch-pack.c | 14 +
git-compat-util.h | 6 +
git.c | 2 +-
object-store.h | 1 +
packfile.c | 1 +
pkt-line.c | 1 +
pkt-line.h | 3 +
remote-curl.c | 46 ++-
send-pack.c | 6 +
serve.c | 27 ++
setup.c | 1 +
t/helper/test-oid-array.c | 3 +
t/t1050-large.sh | 6 +-
t/t1302-repo-version.sh | 6 +-
t/t3200-branch.sh | 2 +-
t/t5300-pack-object.sh | 9 +-
t/t5302-pack-index.sh | 360 +++++++++---------
t/t5500-fetch-pack.sh | 5 +-
t/t5550-http-fetch-dumb.sh | 18 +
t/t5562-http-backend-content-length.sh | 5 +-
t/t5701-git-serve.sh | 25 ++
t/t5702-protocol-v2.sh | 2 +
t/t5703-upload-pack-ref-in-want.sh | 19 +-
t/t5704-protocol-violations.sh | 2 +
t/t5801/git-remote-testgit | 6 +
t/test-lib.sh | 1 +
transport-helper.c | 24 +-
transport.c | 18 +-
transport.h | 8 +
upload-pack.c | 3 +-
wrapper.c | 8 +
45 files changed, 696 insertions(+), 248 deletions(-)
Range-diff against v2:
1: 5878fe6a98 = 1: 3504602e31 t1050: match object ID paths in a hash-insensitive way
2: 402864eaa3 = 2: 150ccddb98 Documentation: document v1 protocol object-format capability
3: d124692e2f = 3: b86ec9fffe connect: have ref processing code take struct packet_reader
4: cce29662b4 = 4: f048e638e5 wrapper: add function to compare strings with different NUL termination
5: 3b207e304b ! 5: 99261e8221 remote: advertise the object-format capability on the server side
@@ upload-pack.c
@@ upload-pack.c: static int send_ref(const char *refname, const struct object_id *oid,
struct strbuf symref_info = STRBUF_INIT;
- format_symref_info(&symref_info, cb_data);
+ format_symref_info(&symref_info, &data->symref);
- packet_write_fmt(1, "%s %s%c%s%s%s%s%s%s agent=%s\n",
+ packet_write_fmt(1, "%s %s%c%s%s%s%s%s%s object-format=%s agent=%s\n",
oid_to_hex(oid), refname_nons,
0, capabilities,
(allow_unadvertised_object_request & ALLOW_TIP_SHA1) ?
@@ upload-pack.c: static int send_ref(const char *refname, const struct object_id *oid,
- stateless_rpc ? " no-done" : "",
+ data->stateless_rpc ? " no-done" : "",
symref_info.buf,
allow_filter ? " filter" : "",
+ the_hash_algo->name,
6: 235d7f5b8f = 6: 5504199a26 connect: add function to parse multiple v1 capability values
7: 0324e126b1 = 7: 59d1b463bf connect: add function to fetch value of a v2 server capability
8: cdba3122ce = 8: 1b2789cab7 pkt-line: add a member for hash algorithm
9: c8233c3b42 = 9: 971d05e2c7 transport: add a hash algorithm member
10: b9273c4021 = 10: 7b90abd41a connect: add function to detect supported v1 hash functions
11: e2d37b75c8 = 11: 578676762d send-pack: detect when the server doesn't support our hash
12: 602734cbbb = 12: 131e98603a connect: make parse_feature_value extern
13: d97fa2c8aa = 13: a786478005 fetch-pack: detect when the server doesn't support our hash
14: ba052f1da7 = 14: 3436a6db7b connect: detect algorithm when fetching refs
15: 661d94d4de = 15: c8d0760e3f builtin/receive-pack: detect when the server doesn't support our hash
16: fd8b85390c = 16: 944bf6ab9a docs: update remote helper docs for object-format extensions
17: 32285e611f = 17: 9f072d34dc transport-helper: implement object-format extensions
18: a33d1ed9a0 = 18: 2bdc53a8d9 remote-curl: implement object-format extensions
19: fffdf0780d = 19: 87c6cd32f7 builtin/clone: initialize hash algorithm properly
20: f616f85b4b = 20: e2cc4d34fe t5562: pass object-format in synthesized test data
21: eca43da42e = 21: 34b712a983 fetch-pack: parse and advertise the object-format capability
22: 22c1a62e10 = 22: 3e43a7d314 setup: set the_repository's hash algo when checking format
23: 7c7f2263d5 = 23: fcf0ef64f0 t3200: mark assertion with SHA1 prerequisite
24: ee8a71a926 = 24: 192578b5dd packfile: compute and use the index CRC offset
25: 6afecf0b09 = 25: a560ed3194 t5302: modernize test formatting
26: 99a847ba4e = 26: 9a079f99a0 builtin/show-index: provide options to determine hash algo
27: 9f7c7bafaf = 27: 56d97c9c94 t1302: expect repo format version 1 for SHA-256
28: d0ea597d63 = 28: 8c25898ded Documentation/technical: document object-format for protocol v2
29: 51848df542 = 29: 39af318fb0 connect: pass full packet reader when parsing v2 refs
30: b57361f3b8 ! 30: 5b334cb9b9 connect: parse v2 refs with correct hash algorithm
@@ connect.c: static int process_ref_v2(struct packet_reader *reader, struct ref **
*list = &peeled->next;
@@ connect.c: struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,
- const struct string_list *server_options)
+ int stateless_rpc)
{
int i;
+ const char *hash_name;
31: a0c0f0f7a3 = 31: 955ea0b4cb serve: advertise object-format capability for protocol v2
32: 1694f3f838 = 32: 5172e56115 t5500: make hash independent
33: 902b394667 = 33: 79b576238f builtin/ls-remote: initialize repository based on fetch
34: cc12b9b51f ! 34: 0ecf5c1a1f remote-curl: detect algorithm for dumb HTTP by size
@@ Commit message
provide one. Detect the hash algorithm in use by the size of the first
object ID.
+ If we have an empty repository, we don't know what the hash algorithm is
+ on the remote side, so default to whatever the local side has
+ configured. Without doing this, we cannot clone an empty repository
+ since we don't know its hash algorithm. Test this case appropriately,
+ since we currently have no tests for cloning an empty repository with
+ the dumb HTTP protocol.
+
We anonymize the URL like elsewhere in the function in case the user has
decided to include a secret in the URL.
@@ remote-curl.c: static struct ref *parse_git_refs(struct discovery *heads, int fo
+ const char *p = memchr(heads->buf, '\t', heads->len);
+ int algo;
+ if (!p)
-+ return NULL;
++ return the_hash_algo;
+
+ algo = hash_algo_by_length((p - heads->buf) / 2);
+ if (algo == GIT_HASH_UNKNOWN)
@@ remote-curl.c: static struct ref *parse_info_refs(struct discovery *heads)
if (!refs)
refs = ref;
if (last_ref)
+
+ ## t/t5550-http-fetch-dumb.sh ##
+@@ t/t5550-http-fetch-dumb.sh: test_expect_success 'create password-protected repository' '
+ "$HTTPD_DOCUMENT_ROOT_PATH/auth/dumb/repo.git"
+ '
+
++test_expect_success 'create empty remote repository' '
++ git init --bare "$HTTPD_DOCUMENT_ROOT_PATH/empty.git" &&
++ (cd "$HTTPD_DOCUMENT_ROOT_PATH/empty.git" &&
++ mkdir -p hooks &&
++ write_script "hooks/post-update" <<-\EOF &&
++ exec git update-server-info
++ EOF
++ hooks/post-update
++ )
++'
++
++test_expect_success 'empty dumb HTTP repository has default hash algorithm' '
++ test_when_finished "rm -fr clone-empty" &&
++ git clone $HTTPD_URL/dumb/empty.git clone-empty &&
++ git -C clone-empty rev-parse --show-object-format >empty-format &&
++ test "$(cat empty-format)" = "$(test_oid algo)"
++'
++
+ setup_askpass_helper
+
+ test_expect_success 'cloning password-protected repository can fail' '
35: b5425c9f54 = 35: 95bc1ef6fb builtin/index-pack: add option to specify hash algorithm
36: 5c70c24d7a = 36: 89c389100e t1050: pass algorithm to index-pack when outside repo
37: 460d6008e8 = 37: bd93aabe6b remote-curl: avoid truncating refs with ls-remote
38: 60a98d9b53 = 38: a8e31600cc t/helper: initialize the repository for test-sha1-array
39: b66c3ead37 = 39: d217758a83 t5702: offer an object-format capability in the test
40: af43274a1f = 40: 6d1964142c t5703: use object-format serve option
41: f5085b1f3f = 41: 84cad9b3ba t5704: send object-format capability with SHA-256
42: a1b01babda = 42: 1e96dbf3dc t5300: pass --object-format to git index-pack
43: dbb5f7195e = 43: 14cb067334 bundle: detect hash algorithm when reading refs
44: 6c823bbe68 = 44: 816d08eb2e remote-testgit: adapt for object-format
From: brian m. carlson <hidden> Date: 2020-06-19 17:56:22
In a future patch, we'll want to access multiple members from struct
packet_reader when parsing references. Therefore, have the ref parsing
code take pointers to struct reader instead of having to pass multiple
arguments to each function.
Rename the len variable to "linelen" to make it clearer what the
variable does in light of the variable change.
Signed-off-by: brian m. carlson <redacted>
---
connect.c | 27 ++++++++++++++++-----------
1 file changed, 16 insertions(+), 11 deletions(-)
From: brian m. carlson <hidden> Date: 2020-06-19 17:56:26
Add a function, server_supports_hash, to see if the remote server
supports a particular hash algorithm when speaking protocol v1.
Signed-off-by: brian m. carlson <redacted>
---
connect.c | 22 ++++++++++++++++++++++
connect.h | 1 +
2 files changed, 23 insertions(+)
From: brian m. carlson <hidden> Date: 2020-06-19 17:56:36
Implement the object-format extensions that let us determine the hash
algorithm in use when pushing or pulling data.
Signed-off-by: brian m. carlson <redacted>
---
transport-helper.c | 24 ++++++++++++++++++++++--
1 file changed, 22 insertions(+), 2 deletions(-)
From: brian m. carlson <hidden> Date: 2020-06-19 17:56:40
When we're checking the repository's format, set the hash algorithm at
the same time. This ensures that we perform a suitable initialization
early enough to avoid confusing any parts of the code. If we defer
until later, we can end up with portions of the code which are confused
about the hash algorithm, resulting in segfaults when working with
SHA-256 repositories.
Signed-off-by: brian m. carlson <redacted>
---
setup.c | 1 +
1 file changed, 1 insertion(+)
From: brian m. carlson <hidden> Date: 2020-06-19 17:56:41
Parse the server's object-format capability and respond accordingly,
dying if there is a mismatch.
Signed-off-by: brian m. carlson <redacted>
---
fetch-pack.c | 12 ++++++++++++
1 file changed, 12 insertions(+)
@@ -1180,6 +1180,7 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,intsideband_all,intseen_ack){intret=0;+constchar*hash_name;structstrbufreq_buf=STRBUF_INIT;if(server_supports_v2("fetch",1))
@@ -1194,6 +1195,17 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,args->server_options->items[i].string);}+if(server_feature_v2("object-format",&hash_name)){+inthash_algo=hash_algo_by_name(hash_name);+if(hash_algo_by_ptr(the_hash_algo)!=hash_algo)+die(_("mismatched algorithms: client %s; server %s"),+the_hash_algo->name,hash_name);+packet_write_fmt(fd_out,"object-format=%s",the_hash_algo->name);+}elseif(hash_algo_by_ptr(the_hash_algo)!=GIT_HASH_SHA1){+die(_("the server does not support algorithm '%s'"),+the_hash_algo->name);+}+packet_buf_delim(&req_buf);if(args->use_thin_pack)packet_buf_write(&req_buf,"thin-pack");
From: brian m. carlson <hidden> Date: 2020-06-19 17:56:44
show-index is capable of reading any possible index file whether or not
the index is inside a repository. However, because our index files lack
metadata about the hash algorithm in use, it's not possible to
autodetect the algorithm that a particular index file is using.
In order to allow us to read index files of any algorithm, let's set up
the .git directory gently so that we default to the algorithm for the
current repository, and add an --object-format option to allow users to
override this setting and continue to run show-index outside of a
repository altogether. Let's also document this new option so that
people can find it and use it.
Signed-off-by: brian m. carlson <redacted>
---
Documentation/git-show-index.txt | 11 ++++++++++-
builtin/show-index.c | 29 ++++++++++++++++++++++++-----
git.c | 2 +-
3 files changed, 35 insertions(+), 7 deletions(-)
@@ -9,7 +9,7 @@ git-show-index - Show packed archive index SYNOPSIS -------- [verse]-'git show-index'+'git show-index' [--object-format=<hash-algorithm>] DESCRIPTION
@@ -36,6 +36,15 @@ Note that you can get more information on a packfile by calling linkgit:git-verify-pack[1]. However, as this command considers only the index file itself, it's both faster and more flexible.+OPTIONS+-------++--object-format=<hash-algorithm>::+ Specify the given object format (hash algorithm) for the index file. The+ valid values are 'sha1' and (if enabled) 'sha256'. The default is the+ algorithm for the current repository (set by `extensions.objectFormat`), or+ 'sha1' if no value is set or outside a repository..+ GIT --- Part of the linkgit:git[1] suite
From: brian m. carlson <hidden> Date: 2020-06-19 17:56:45
Our style these days is to place the description and the opening quote
of the body on the same line as test_expect_success (if it fits), to
place the trailing quote on a line by itself after the body, and to use
tabs. Since we're going to be making several significant changes to
this test, modernize the style to aid in readability of the subsequent
patches.
This patch should have no functional change.
Signed-off-by: brian m. carlson <redacted>
---
t/t5302-pack-index.sh | 360 +++++++++++++++++++++---------------------
1 file changed, 184 insertions(+), 176 deletions(-)
@@ -7,65 +7,65 @@ test_description='pack index with 64-bit offsets and object CRC' ../test-lib.sh test_expect_success'setup''-test_oid_init&&-rawsz=$(test_oidrawsz)&&-rm-rf.git&&-gitinit&&-gitconfigpack.threads1&&-i=1&&-whiletest$i-le100-do-iii=$(printf'%03i'$i)-test-toolgenrandom"bar"200>wide_delta_$iii&&-test-toolgenrandom"baz $iii"50>>wide_delta_$iii&&-test-toolgenrandom"foo"$i100>deep_delta_$iii&&-test-toolgenrandom"foo"$(expr$i+1)100>>deep_delta_$iii&&-test-toolgenrandom"foo"$(expr$i+2)100>>deep_delta_$iii&&-echo$iii>file_$iii&&-test-toolgenrandom"$iii"8192>>file_$iii&&-gitupdate-index--addfile_$iiideep_delta_$iiiwide_delta_$iii&&-i=$(expr$i+1)||return1-done&&-{echo101&&test-toolgenrandom1008192;}>file_101&&-gitupdate-index--addfile_101&&-tree=$(gitwrite-tree)&&-commit=$(gitcommit-tree$tree</dev/null)&&{-echo$tree&&-gitls-tree$tree|sed-e"s/.* \\([0-9a-f]*\\) .*/\\1/"-}>obj-list&&-gitupdate-refHEAD$commit+test_oid_init&&+rawsz=$(test_oidrawsz)&&+rm-rf.git&&+gitinit&&+gitconfigpack.threads1&&+i=1&&+whiletest$i-le100+do+iii=$(printf'%03i'$i)+test-toolgenrandom"bar"200>wide_delta_$iii&&+test-toolgenrandom"baz $iii"50>>wide_delta_$iii&&+test-toolgenrandom"foo"$i100>deep_delta_$iii&&+test-toolgenrandom"foo"$(expr$i+1)100>>deep_delta_$iii&&+test-toolgenrandom"foo"$(expr$i+2)100>>deep_delta_$iii&&+echo$iii>file_$iii&&+test-toolgenrandom"$iii"8192>>file_$iii&&+gitupdate-index--addfile_$iiideep_delta_$iiiwide_delta_$iii&&+i=$(expr$i+1)||return1+done&&+{echo101&&test-toolgenrandom1008192;}>file_101&&+gitupdate-index--addfile_101&&+tree=$(gitwrite-tree)&&+commit=$(gitcommit-tree$tree</dev/null)&&{+echo$tree&&+gitls-tree$tree|sed-e"s/.* \\([0-9a-f]*\\) .*/\\1/"+}>obj-list&&+gitupdate-refHEAD$commit'-test_expect_success\-'pack-objects with index version 1'\-'pack1=$(gitpack-objects--index-version=1test-1<obj-list)&&-gitverify-pack-v"test-1-${pack1}.pack"'+test_expect_success'pack-objects with index version 1''+pack1=$(gitpack-objects--index-version=1test-1<obj-list)&&+gitverify-pack-v"test-1-${pack1}.pack"+'-test_expect_success\-'pack-objects with index version 2'\-'pack2=$(gitpack-objects--index-version=2test-2<obj-list)&&-gitverify-pack-v"test-2-${pack2}.pack"'+test_expect_success'pack-objects with index version 2''+pack2=$(gitpack-objects--index-version=2test-2<obj-list)&&+gitverify-pack-v"test-2-${pack2}.pack"+'-test_expect_success\-'both packs should be identical'\-'cmp "test-1-${pack1}.pack" "test-2-${pack2}.pack"'+test_expect_success'both packs should be identical''+cmp"test-1-${pack1}.pack""test-2-${pack2}.pack"+'-test_expect_success\-'index v1 and index v2 should be different'\-'! cmp "test-1-${pack1}.idx" "test-2-${pack2}.idx"'+test_expect_success'index v1 and index v2 should be different''+!cmp"test-1-${pack1}.idx""test-2-${pack2}.idx"+'-test_expect_success\-'index-pack with index version 1'\-'git index-pack --index-version=1 -o 1.idx "test-1-${pack1}.pack"'+test_expect_success'index-pack with index version 1''+gitindex-pack--index-version=1-o1.idx"test-1-${pack1}.pack"+'-test_expect_success\-'index-pack with index version 2'\-'git index-pack --index-version=2 -o 2.idx "test-1-${pack1}.pack"'+test_expect_success'index-pack with index version 2''+gitindex-pack--index-version=2-o2.idx"test-1-${pack1}.pack"+'-test_expect_success\-'index-pack results should match pack-objects ones'\-'cmp"test-1-${pack1}.idx""1.idx"&&-cmp"test-2-${pack2}.idx""2.idx"'+test_expect_success'index-pack results should match pack-objects ones''+cmp"test-1-${pack1}.idx""1.idx"&&+cmp"test-2-${pack2}.idx""2.idx"+' test_expect_success'index-pack --verify on index version 1''gitindex-pack--verify"test-1-${pack1}.pack"
@@ -75,13 +75,13 @@ test_expect_success 'index-pack --verify on index version 2' 'gitindex-pack--verify"test-2-${pack2}.pack"'-test_expect_success\-'pack-objects --index-version=2, is not accepted'\-'test_must_fail git pack-objects --index-version=2, test-3 <obj-list'+test_expect_success'pack-objects --index-version=2, is not accepted''+test_must_failgitpack-objects--index-version=2,test-3<obj-list+'-test_expect_success\-'index v2: force some 64-bit offsets with pack-objects'\-'pack3=$(git pack-objects --index-version=2,0x40000 test-3 <obj-list)'+test_expect_success'index v2: force some 64-bit offsets with pack-objects''+pack3=$(gitpack-objects--index-version=2,0x40000test-3<obj-list)+'ifmsg=$(gitverify-pack-v"test-3-${pack3}.pack"2>&1)||!(echo"$msg"|grep"pack too large .* off_t")
@@ -91,21 +91,21 @@ elsesay"# skipping tests concerning 64-bit offsets"fi-test_expect_successOFF64_T\-'index v2: verify a pack with some 64-bit offsets'\-'git verify-pack -v "test-3-${pack3}.pack"'+test_expect_successOFF64_T'index v2: verify a pack with some 64-bit offsets''+gitverify-pack-v"test-3-${pack3}.pack"+'-test_expect_successOFF64_T\-'64-bit offsets: should be different from previous index v2 results'\-'! cmp "test-2-${pack2}.idx" "test-3-${pack3}.idx"'+test_expect_successOFF64_T'64-bit offsets: should be different from previous index v2 results''+!cmp"test-2-${pack2}.idx""test-3-${pack3}.idx"+'-test_expect_successOFF64_T\-'index v2: force some 64-bit offsets with index-pack'\-'git index-pack --index-version=2,0x40000 -o 3.idx "test-1-${pack1}.pack"'+test_expect_successOFF64_T'index v2: force some 64-bit offsets with index-pack''+gitindex-pack--index-version=2,0x40000-o3.idx"test-1-${pack1}.pack"+'-test_expect_successOFF64_T\-'64-bit offsets: index-pack result should match pack-objects one'\-'cmp "test-3-${pack3}.idx" "3.idx"'+test_expect_successOFF64_T'64-bit offsets: index-pack result should match pack-objects one''+cmp"test-3-${pack3}.idx""3.idx"+' test_expect_successOFF64_T'index-pack --verify on 64-bit offset v2 (cheat)''# This cheats by knowing which lower offset should still be encoded
@@ -120,135 +120,143 @@ test_expect_success OFF64_T 'index-pack --verify on 64-bit offset v2' '# returns the object number for given object in given pack index index_obj_nr(){-idx_file=$1-object_sha1=$2-nr=0-gitshow-index<$idx_file|-whilereadoffssha1extra-do-nr=$(($nr+1))-test"$sha1"="$object_sha1"||continue-echo"$(($nr-1))"-break-done+idx_file=$1+object_sha1=$2+nr=0+gitshow-index<$idx_file|+whilereadoffssha1extra+do+nr=$(($nr+1))+test"$sha1"="$object_sha1"||continue+echo"$(($nr-1))"+break+done}# returns the pack offset for given object as found in given pack index index_obj_offset(){-idx_file=$1-object_sha1=$2-gitshow-index<$idx_file|grep$object_sha1|-(readoffsextra&&echo"$offs")+idx_file=$1+object_sha1=$2+gitshow-index<$idx_file|grep$object_sha1|+(readoffsextra&&echo"$offs")}-test_expect_success\-'[index v1] 1) stream pack to repository'\-'gitindex-pack--index-version=1--stdin<"test-1-${pack1}.pack"&&-gitprune-packed&&-gitcount-objects|(readnrrest&&test"$nr"-eq1)&&-cmp"test-1-${pack1}.pack"".git/objects/pack/pack-${pack1}.pack"&&-cmp"test-1-${pack1}.idx"".git/objects/pack/pack-${pack1}.idx"'+test_expect_success'[index v1] 1) stream pack to repository''+gitindex-pack--index-version=1--stdin<"test-1-${pack1}.pack"&&+gitprune-packed&&+gitcount-objects|(readnrrest&&test"$nr"-eq1)&&+cmp"test-1-${pack1}.pack"".git/objects/pack/pack-${pack1}.pack"&&+cmp"test-1-${pack1}.idx"".git/objects/pack/pack-${pack1}.idx"+' test_expect_success\-'[index v1] 2) create a stealth corruption in a delta base reference'\-'# This test assumes file_101 is a delta smaller than 16 bytes.-# It should be against file_100 but we substitute its base for file_099-sha1_101=$(githash-objectfile_101)&&-sha1_099=$(githash-objectfile_099)&&-offs_101=$(index_obj_offset1.idx$sha1_101)&&-nr_099=$(index_obj_nr1.idx$sha1_099)&&-chmod+w".git/objects/pack/pack-${pack1}.pack"&&-recordsz=$((rawsz+4))&&-ddof=".git/objects/pack/pack-${pack1}.pack"seek=$(($offs_101+1))\-if=".git/objects/pack/pack-${pack1}.idx"\-skip=$((4+256*4+$nr_099*recordsz))\-bs=1count=$rawszconv=notrunc&&-gitcat-fileblob$sha1_101>file_101_foo1'+'[index v1] 2) create a stealth corruption in a delta base reference''+# This test assumes file_101 is a delta smaller than 16 bytes.+# It should be against file_100 but we substitute its base for file_099+sha1_101=$(githash-objectfile_101)&&+sha1_099=$(githash-objectfile_099)&&+offs_101=$(index_obj_offset1.idx$sha1_101)&&+nr_099=$(index_obj_nr1.idx$sha1_099)&&+chmod+w".git/objects/pack/pack-${pack1}.pack"&&+recordsz=$((rawsz+4))&&+ddof=".git/objects/pack/pack-${pack1}.pack"seek=$(($offs_101+1))\+if=".git/objects/pack/pack-${pack1}.idx"\+skip=$((4+256*4+$nr_099*recordsz))\+bs=1count=$rawszconv=notrunc&&+gitcat-fileblob$sha1_101>file_101_foo1+' test_expect_success\-'[index v1] 3) corrupted delta happily returned wrong data'\-'test -f file_101_foo1 && ! cmp file_101 file_101_foo1'+'[index v1] 3) corrupted delta happily returned wrong data''+test-ffile_101_foo1&&!cmpfile_101file_101_foo1+' test_expect_success\-'[index v1] 4) confirm that the pack is actually corrupted'\-'test_must_fail git fsck --full $commit'+'[index v1] 4) confirm that the pack is actually corrupted''+test_must_failgitfsck--full$commit+' test_expect_success\-'[index v1] 5) pack-objects happily reuses corrupted data'\-'pack4=$(gitpack-objectstest-4<obj-list)&&-test-f"test-4-${pack4}.pack"'+'[index v1] 5) pack-objects happily reuses corrupted data''+pack4=$(gitpack-objectstest-4<obj-list)&&+test-f"test-4-${pack4}.pack"+'++test_expect_success'[index v1] 6) newly created pack is BAD !''+test_must_failgitverify-pack-v"test-4-${pack4}.pack"+'++test_expect_success'[index v2] 1) stream pack to repository''+rm-f.git/objects/pack/*&&+gitindex-pack--index-version=2--stdin<"test-1-${pack1}.pack"&&+gitprune-packed&&+gitcount-objects|(readnrrest&&test"$nr"-eq1)&&+cmp"test-1-${pack1}.pack"".git/objects/pack/pack-${pack1}.pack"&&+cmp"test-2-${pack1}.idx"".git/objects/pack/pack-${pack1}.idx"+' test_expect_success\-'[index v1] 6) newly created pack is BAD !'\-'test_must_fail git verify-pack -v "test-4-${pack4}.pack"'+'[index v2] 2) create a stealth corruption in a delta base reference''+# This test assumes file_101 is a delta smaller than 16 bytes.+# It should be against file_100 but we substitute its base for file_099+sha1_101=$(githash-objectfile_101)&&+sha1_099=$(githash-objectfile_099)&&+offs_101=$(index_obj_offset1.idx$sha1_101)&&+nr_099=$(index_obj_nr1.idx$sha1_099)&&+chmod+w".git/objects/pack/pack-${pack1}.pack"&&+ddof=".git/objects/pack/pack-${pack1}.pack"seek=$(($offs_101+1))\+if=".git/objects/pack/pack-${pack1}.idx"\+skip=$((8+256*4+$nr_099*rawsz))\+bs=1count=$rawszconv=notrunc&&+gitcat-fileblob$sha1_101>file_101_foo2+' test_expect_success\-'[index v2] 1) stream pack to repository'\-'rm-f.git/objects/pack/*&&-gitindex-pack--index-version=2--stdin<"test-1-${pack1}.pack"&&-gitprune-packed&&-gitcount-objects|(readnrrest&&test"$nr"-eq1)&&-cmp"test-1-${pack1}.pack"".git/objects/pack/pack-${pack1}.pack"&&-cmp"test-2-${pack1}.idx"".git/objects/pack/pack-${pack1}.idx"'+'[index v2] 3) corrupted delta happily returned wrong data''+test-ffile_101_foo2&&!cmpfile_101file_101_foo2+' test_expect_success\-'[index v2] 2) create a stealth corruption in a delta base reference'\-'# This test assumes file_101 is a delta smaller than 16 bytes.-# It should be against file_100 but we substitute its base for file_099-sha1_101=$(githash-objectfile_101)&&-sha1_099=$(githash-objectfile_099)&&-offs_101=$(index_obj_offset1.idx$sha1_101)&&-nr_099=$(index_obj_nr1.idx$sha1_099)&&-chmod+w".git/objects/pack/pack-${pack1}.pack"&&-ddof=".git/objects/pack/pack-${pack1}.pack"seek=$(($offs_101+1))\-if=".git/objects/pack/pack-${pack1}.idx"\-skip=$((8+256*4+$nr_099*rawsz))\-bs=1count=$rawszconv=notrunc&&-gitcat-fileblob$sha1_101>file_101_foo2'+'[index v2] 4) confirm that the pack is actually corrupted''+test_must_failgitfsck--full$commit+' test_expect_success\-'[index v2] 3) corrupted delta happily returned wrong data'\-'test -f file_101_foo2 && ! cmp file_101 file_101_foo2'+'[index v2] 5) pack-objects refuses to reuse corrupted data''+test_must_failgitpack-objectstest-5<obj-list&&+test_must_failgitpack-objects--no-reuse-objecttest-6<obj-list+' test_expect_success\-'[index v2] 4) confirm that the pack is actually corrupted'\-'test_must_fail git fsck --full $commit'--test_expect_success\-'[index v2] 5) pack-objects refuses to reuse corrupted data'\-'test_must_failgitpack-objectstest-5<obj-list&&-test_must_failgitpack-objects--no-reuse-objecttest-6<obj-list'--test_expect_success\-'[index v2] 6) verify-pack detects CRC mismatch'\-'rm-f.git/objects/pack/*&&-gitindex-pack--index-version=2--stdin<"test-1-${pack1}.pack"&&-gitverify-pack".git/objects/pack/pack-${pack1}.pack"&&-obj=$(githash-objectfile_001)&&-nr=$(index_obj_nr".git/objects/pack/pack-${pack1}.idx"$obj)&&-chmod+w".git/objects/pack/pack-${pack1}.idx"&&-printfxxxx|ddof=".git/objects/pack/pack-${pack1}.idx"conv=notrunc\-bs=1count=4seek=$((8+256*4+$(wc-l<obj-list)*rawsz+$nr*4))&&-(whilereadobj-dogitcat-file-p$obj>/dev/null||exit1-done<obj-list)&&-test_must_failgitverify-pack".git/objects/pack/pack-${pack1}.pack"+'[index v2] 6) verify-pack detects CRC mismatch''+rm-f.git/objects/pack/*&&+gitindex-pack--index-version=2--stdin<"test-1-${pack1}.pack"&&+gitverify-pack".git/objects/pack/pack-${pack1}.pack"&&+obj=$(githash-objectfile_001)&&+nr=$(index_obj_nr".git/objects/pack/pack-${pack1}.idx"$obj)&&+chmod+w".git/objects/pack/pack-${pack1}.idx"&&+printfxxxx|ddof=".git/objects/pack/pack-${pack1}.idx"conv=notrunc\+bs=1count=4seek=$((8+256*4+$(wc-l<obj-list)*rawsz+$nr*4))&&+(whilereadobj+dogitcat-file-p$obj>/dev/null||exit1+done<obj-list)&&+test_must_failgitverify-pack".git/objects/pack/pack-${pack1}.pack"' test_expect_success'running index-pack in the object store''-rm-f.git/objects/pack/*&&-cptest-1-${pack1}.pack.git/objects/pack/pack-${pack1}.pack&&-(-cd.git/objects/pack&&-gitindex-packpack-${pack1}.pack-)&&-test-f.git/objects/pack/pack-${pack1}.idx+rm-f.git/objects/pack/*&&+cptest-1-${pack1}.pack.git/objects/pack/pack-${pack1}.pack&&+(+cd.git/objects/pack&&+gitindex-packpack-${pack1}.pack+)&&+test-f.git/objects/pack/pack-${pack1}.idx' test_expect_success'index-pack --strict warns upon missing tagger in tag''-sha=$(gitrev-parseHEAD)&&-cat>wrong-tag<<EOF&&+sha=$(gitrev-parseHEAD)&&+cat>wrong-tag<<EOF&& object$shatypecommit taggutentag
@@ -256,18 +264,18 @@ tag guten tag Thisisaninvalidtag. EOF-tag=$(githash-object-ttag-w--stdin<wrong-tag)&&-pack1=$(echo$tag$sha|gitpack-objectstag-test)&&-echoremovetagobject&&-thirtyeight=${tag#??}&&-rm-f.git/objects/${tag%$thirtyeight}/$thirtyeight&&-gitindex-pack--stricttag-test-${pack1}.pack2>err&&-grep"^warning:.* expected .tagger. line"err+tag=$(githash-object-ttag-w--stdin<wrong-tag)&&+pack1=$(echo$tag$sha|gitpack-objectstag-test)&&+echoremovetagobject&&+thirtyeight=${tag#??}&&+rm-f.git/objects/${tag%$thirtyeight}/$thirtyeight&&+gitindex-pack--stricttag-test-${pack1}.pack2>err&&+grep"^warning:.* expected .tagger. line"err' test_expect_success'index-pack --fsck-objects also warns upon missing tagger in tag''-gitindex-pack--fsck-objectstag-test-${pack1}.pack2>err&&-grep"^warning:.* expected .tagger. line"err+gitindex-pack--fsck-objectstag-test-${pack1}.pack2>err&&+grep"^warning:.* expected .tagger. line"err' test_done
From: brian m. carlson <hidden> Date: 2020-06-19 17:56:48
When using protocol v2, we need to know what hash algorithm is used by
the remote end. See if the server has sent us an object-format
capability, and if so, use it to determine the hash algorithm in use and
set that value in the packet reader. Parse the refs using this
algorithm.
Signed-off-by: brian m. carlson <redacted>
---
connect.c | 21 ++++++++++++++++-----
1 file changed, 16 insertions(+), 5 deletions(-)
From: brian m. carlson <hidden> Date: 2020-06-19 17:56:48
When reading the info/refs file for a repository, we have no explicit
way to detect which hash algorithm is in use because the file doesn't
provide one. Detect the hash algorithm in use by the size of the first
object ID.
If we have an empty repository, we don't know what the hash algorithm is
on the remote side, so default to whatever the local side has
configured. Without doing this, we cannot clone an empty repository
since we don't know its hash algorithm. Test this case appropriately,
since we currently have no tests for cloning an empty repository with
the dumb HTTP protocol.
We anonymize the URL like elsewhere in the function in case the user has
decided to include a secret in the URL.
Signed-off-by: brian m. carlson <redacted>
---
remote-curl.c | 23 +++++++++++++++++++++--
t/t5550-http-fetch-dumb.sh | 18 ++++++++++++++++++
2 files changed, 39 insertions(+), 2 deletions(-)
@@ -262,6 +275,12 @@ static struct ref *parse_info_refs(struct discovery *heads)structref*ref=NULL;structref*last_ref=NULL;+options.hash_algo=detect_hash_algo(heads);+if(!options.hash_algo)+die("%sinfo/refs not valid: could not determine hash algorithm; "+"is this a git repository?",+transport_anonymize_url(url.buf));+data=heads->buf;start=NULL;mid=data;
@@ -272,13 +291,13 @@ static struct ref *parse_info_refs(struct discovery *heads)if(data[i]=='\t')mid=&data[i];if(data[i]=='\n'){-if(mid-start!=the_hash_algo->hexsz)+if(mid-start!=options.hash_algo->hexsz)die(_("%sinfo/refs not valid: is this a git repository?"),transport_anonymize_url(url.buf));data[i]=0;ref_name=mid+1;ref=alloc_ref(ref_name);-get_oid_hex(start,&ref->old_oid);+get_oid_hex_algop(start,&ref->old_oid,options.hash_algo);if(!refs)refs=ref;if(last_ref)
From: brian m. carlson <hidden> Date: 2020-06-19 17:56:49
When we're parsing refs, we need to know not only what the line we're
parsing is, but also the hash algorithm we should use to parse it, which
is stored in the reader object. Pass the packet reader object through
to the protocol v2 ref parsing function.
Signed-off-by: brian m. carlson <redacted>
---
connect.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
@@ -376,7 +376,7 @@ struct ref **get_remote_heads(struct packet_reader *reader,}/* Returns 1 when a valid ref has been added to `list`, 0 otherwise */-staticintprocess_ref_v2(constchar*line,structref***list)+staticintprocess_ref_v2(structpacket_reader*reader,structref***list){intret=1;inti=0;
From: brian m. carlson <hidden> Date: 2020-06-19 17:56:50
In order to communicate the protocol supported by the server side, add
support for advertising the object-format capability. We check that the
client side sends us an identical algorithm if it sends us its own
object-format capability, and assume it speaks SHA-1 if not.
In the test, when we're using an algorithm other than SHA-1, we need to
specify the algorithm in use so we don't get a failure with an "unknown
format" message. Add a test that we handle a mismatched algorithm.
Remove the test_oid_init call since it's no longer necessary.
Signed-off-by: brian m. carlson <redacted>
---
connect.c | 2 ++
serve.c | 27 +++++++++++++++++++++++++++
t/t5701-git-serve.sh | 25 +++++++++++++++++++++++++
3 files changed, 54 insertions(+)
@@ -45,6 +50,7 @@ test_expect_success 'request invalid capability' ' test_expect_success'request with no command''test-toolpkt-linepack>in<<-EOF&&agent=git/test+object-format=$(test_oidalgo)0000EOFtest_must_failtest-toolserve-v2--stateless-rpc2>err<in&&
@@ -54,6 +60,7 @@ test_expect_success 'request with no command' ' test_expect_success'request invalid command''test-toolpkt-linepack>in<<-EOF&&command=foo+object-format=$(test_oidalgo)agent=git/test0000EOF
@@ -61,6 +68,17 @@ test_expect_success 'request invalid command' 'test_i18ngrep"invalid command"err'+test_expect_success'wrong object-format''+test-toolpkt-linepack>in<<-EOF&&+command=fetch+agent=git/test+object-format=$(test_oidwrong_algo)+0000+EOF+test_must_failtest-toolserve-v2--stateless-rpc2>err<in&&+test_i18ngrep"mismatched object format"err+'+# Test the basics of ls-refs# test_expect_success'setup some refs and tags''
@@ -74,6 +92,7 @@ test_expect_success 'setup some refs and tags' ' test_expect_success'basics of ls-refs''test-toolpkt-linepack>in<<-EOF&&command=ls-refs+object-format=$(test_oidalgo)0000EOF
@@ -200,6 +224,7 @@ test_expect_success 'unexpected lines are not allowed in fetch request' 'test-toolpkt-linepack>in<<-EOF&&command=fetch+object-format=$(test_oidalgo)0001this-is-not-a-command0000
From: brian m. carlson <hidden> Date: 2020-06-19 17:56:51
Normally, the remote-curl transport helper is aware of the hash
algorithm we're using because we're in a repo with the appropriate hash
algorithm set. However, when using git ls-remote outside of a
repository, we won't have initialized the hash algorithm properly, so
use hash_to_hex_algop to print the ref corresponding to the algorithm
we've detected.
Signed-off-by: brian m. carlson <redacted>
---
remote-curl.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
From: brian m. carlson <hidden> Date: 2020-06-19 17:56:55
ls-remote may or may not operate within a repository, and as such will
not have been initialized with the repository's hash algorithm. Even if
it were, the remote side could be using a different algorithm and we
would still want to display those refs properly. Find the hash
algorithm used by the remote side by querying the transport object and
set our hash algorithm accordingly.
Without this change, if the remote side is using SHA-256, we truncate
the refs to 40 hex characters, since that's the length of the default
hash algorithm (SHA-1).
Note that technically this is not a correct setting of the repository
hash algorithm since, if we are in a repository, it might be one of a
different hash algorithm from the remote side. However, our current
code paths don't handle multiple algorithms and won't for some time, so
this is the best we can do. We rely on the fact that ls-remote never
modifies the current repository, which is a reasonable assumption to
make.
Signed-off-by: brian m. carlson <redacted>
---
builtin/ls-remote.c | 4 ++++
1 file changed, 4 insertions(+)
@@ -455,3 +455,12 @@ included in a request. This is done by sending each option as a a request. The provided options must not contain a NUL or LF character.++ object-format+~~~~~~~~~~~~~~~++The server can advertise the `object-format` capability with a value `X` (in the+form `object-format=X`) to notify the client that the server is able to deal+with objects using hash algorithm X. If not specified, the server is assumed to+only handle SHA-1. If the client would like to use a hash algorithm other than+SHA-1, it should specify its object-format string.
From: brian m. carlson <hidden> Date: 2020-06-19 17:56:59
In order to make this test work with SHA-256, offer an object-format
capability so that both sides use the same algorithm.
Signed-off-by: brian m. carlson <redacted>
---
t/t5702-protocol-v2.sh | 2 ++
1 file changed, 2 insertions(+)
@@ -13,6 +13,7 @@ start_git_daemon --export-all --enable=receive-packdaemon_parent=$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent test_expect_success'create repo to be served by git-daemon''+test_oid_init&&gitinit"$daemon_parent"&&test_commit-C"$daemon_parent"one'
@@ -394,6 +395,7 @@ test_expect_success 'even with handcrafted request, filter does not work if not# Custom request that tries to filter even though it is not advertised.test-toolpkt-linepack>in<<-EOF&&command=fetch+object-format=$(test_oidalgo)0001want$(git-Cserverrev-parsemaster)filterblob:none
From: brian m. carlson <hidden> Date: 2020-06-19 17:57:01
When we speak protocol v2 in this test, we must pass the object-format
header if the algorithm is not SHA-1. Otherwise, git upload-pack fails
because the hash algorithm doesn't match and not because we've failed to
speak the protocol correctly. Pass the header so that our assertions
test what we're really interested in.
Signed-off-by: brian m. carlson <redacted>
---
t/t5704-protocol-violations.sh | 2 ++
1 file changed, 2 insertions(+)
@@ -9,6 +9,7 @@ making sure that we do not segfault or otherwise behave badly.' test_expect_success'extra delim packet in v2 ls-refs args''{packetizecommand=ls-refs&&+packetize"object-format=$(test_oidalgo)"&&printf0001&&# protocol expects 0000 flush hereprintf0001
From: brian m. carlson <hidden> Date: 2020-06-19 17:57:03
The pattern here looking for failures is specific to SHA-1. Let's
create a variable that matches the regex or glob pattern for a path
within the objects directory.
Signed-off-by: brian m. carlson <redacted>
---
t/t1050-large.sh | 2 +-
t/test-lib.sh | 1 +
2 files changed, 2 insertions(+), 1 deletion(-)
@@ -64,7 +64,7 @@ test_expect_success 'add a large file or two' 'test$count=1&&cnt=$(gitshow-index<"$idx"|wc-l)&&test$cnt=2&&-forlin.git/objects/??/??????????????????????????????????????+forlin.git/objects/$OIDPATH_REGEXdotest_path_is_file"$l"||continuebad=t
From: brian m. carlson <hidden> Date: 2020-06-19 17:57:05
This test has hard-coded pkt-lines with object IDs. The pkt-line
lengths necessarily differ between hash algorithms, so generate these
lines with the packetize helper so they're always the right size. In
addition, we will require an object-format capability for SHA-256, so
pass that capability on to the upload-pack process.
Signed-off-by: brian m. carlson <redacted>
---
t/t5500-fetch-pack.sh | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
From: brian m. carlson <hidden> Date: 2020-06-19 17:57:06
Update the remote helper docs to document the object-format extensions
we will implement in remote-curl and the transport helper code shortly.
Signed-off-by: brian m. carlson <redacted>
---
Documentation/gitremote-helpers.txt | 33 +++++++++++++++++++++++++----
1 file changed, 29 insertions(+), 4 deletions(-)
@@ -238,6 +238,9 @@ the remote repository. `--signed-tags=verbatim` to linkgit:git-fast-export[1]. In the absence of this capability, Git will use `--signed-tags=warn-strip`.+'object-format'::+ This indicates that the helper is able to interact with the remote+ side using an explicit hash algorithm extension. COMMANDS
@@ -257,12 +260,14 @@ Support for this command is mandatory. 'list':: Lists the refs, one per line, in the format "<value> <name> [<attr> ...]". The value may be a hex sha1 hash, "@<dest>" for- a symref, or "?" to indicate that the helper could not get the- value of the ref. A space-separated list of attributes follows- the name; unrecognized attributes are ignored. The list ends- with a blank line.+ a symref, ":<keyword> <value>" for a key-value pair, or+ "?" to indicate that the helper could not get the value of the+ ref. A space-separated list of attributes follows the name;+ unrecognized attributes are ignored. The list ends with a+ blank line. + See REF LIST ATTRIBUTES for a list of currently defined attributes.+See REF LIST KEYWORDS for a list of currently defined keywords. + Supported if the helper has the "fetch" or "import" capability.
@@ -432,6 +437,18 @@ attributes are defined. This ref is unchanged since the last import or fetch, although the helper cannot necessarily determine what value that produced.+REF LIST KEYWORDS+-----------------++The 'list' command may produce a list of key-value pairs.+The following keys are defined.++'object-format'::+ The refs are using the given hash algorithm. This keyword is only+ used if the server and client both support the object-format+ extension.++ OPTIONS -------
@@ -516,6 +533,14 @@ set by Git if the remote helper has the 'option' capability. transaction. If successful, all refs will be updated, or none will. If the remote side does not support this capability, the push will fail.+'option object-format' {'true'|algorithm}::+ If 'true', indicate that the caller wants hash algorithm information+ to be passed back from the remote. This mode is used when fetching+ refs.+++If set to an algorithm, indicate that the caller wants to interact with+the remote side using that algorithm.+ SEE ALSO -------- linkgit:git-remote[1]
From: brian m. carlson <hidden> Date: 2020-06-19 17:57:15
When connecting to a remote system, we need to know what hash algorithm
it will be using to talk to us. Add a hash_algo member to struct
transport and add a function to read this data from the transport
object.
Signed-off-by: brian m. carlson <redacted>
---
transport.c | 8 ++++++++
transport.h | 8 ++++++++
2 files changed, 16 insertions(+)
@@ -312,6 +312,7 @@ static struct ref *handshake(struct transport *transport, int for_push,BUG("unknown protocol version");}data->got_remote_heads=1;+transport->hash_algo=reader.hash_algo;if(reader.line_peeked)BUG("buffer must be empty at the end of handshake()");
From: brian m. carlson <hidden> Date: 2020-06-19 17:57:18
If we're fetching refs, detect the hash algorithm and parse the refs
using that algorithm.
As mentioned in the documentation, if multiple versions of the
object-format capability are provided, we use the first. No known
implementation supports multiple algorithms now, but they may in the
future.
Signed-off-by: brian m. carlson <redacted>
---
connect.c | 21 +++++++++++++++++----
1 file changed, 17 insertions(+), 4 deletions(-)
@@ -259,7 +272,7 @@ static int process_ref(const struct packet_reader *reader, int len,structobject_idold_oid;constchar*name;-if(parse_oid_hex(line,&old_oid,&name))+if(parse_oid_hex_algop(line,&old_oid,&name,reader->hash_algo))return0;if(*name!=' ')return0;
@@ -271,7 +284,7 @@ static int process_ref(const struct packet_reader *reader, int len,die(_("protocol error: unexpected capabilities^{}"));}elseif(check_ref(name,flags)){structref*ref=alloc_ref(name);-oidcpy(&ref->old_oid,&old_oid);+memcpy(ref->old_oid.hash,old_oid.hash,reader->hash_algo->rawsz);**list=ref;*list=&ref->next;}
@@ -289,7 +302,7 @@ static int process_shallow(const struct packet_reader *reader, int len,if(!skip_prefix(line,"shallow ",&arg))return0;-if(get_oid_hex(arg,&old_oid))+if(get_oid_hex_algop(arg,&old_oid,reader->hash_algo))die(_("protocol error: expected shallow sha-1, got '%s'"),arg);if(!shallow_points)die(_("repository on the other end cannot be shallow"));
From: brian m. carlson <hidden> Date: 2020-06-19 17:57:19
So far in protocol v2, all of our server capabilities that have values
have not had values that we've been interested in parsing. For example,
we receive but ignore the agent value.
However, in a future commit, we're going to want to parse out the value
of a server capability. To make this easy, add a function,
server_feature_v2, that can fetch the value provided as part of the
server capability.
Signed-off-by: brian m. carlson <redacted>
---
connect.c | 15 +++++++++++++++
connect.h | 1 +
2 files changed, 16 insertions(+)
@@ -84,6 +84,21 @@ int server_supports_v2(const char *c, int die_on_error)return0;}+intserver_feature_v2(constchar*c,constchar**v)+{+inti;++for(i=0;i<server_capabilities_v2.argc;i++){+constchar*out;+if(skip_prefix(server_capabilities_v2.argv[i],c,&out)&&+(*out=='=')){+*v=out+1;+return1;+}+}+return0;+}+intserver_supports_feature(constchar*c,constchar*feature,intdie_on_error){
From: brian m. carlson <hidden> Date: 2020-06-19 17:57:19
Advertise the current hash algorithm in use by using the object-format
capability as part of the ref advertisement.
Signed-off-by: brian m. carlson <redacted>
---
builtin/receive-pack.c | 1 +
upload-pack.c | 3 ++-
2 files changed, 3 insertions(+), 1 deletion(-)
From: brian m. carlson <hidden> Date: 2020-06-19 17:57:23
In a capability response, we can have multiple symref entries. In the
future, we will also allow for multiple hash algorithms to be specified.
To avoid duplication, expand the parse_feature_value function to take an
optional offset where the parsing should begin next time. Add a wrapper
function that allows us to query the next server feature value, and use
it in the existing symref parsing code.
Signed-off-by: brian m. carlson <redacted>
---
connect.c | 30 +++++++++++++++++++++---------
1 file changed, 21 insertions(+), 9 deletions(-)
From: brian m. carlson <hidden> Date: 2020-06-19 17:57:26
When using an algorithm other than SHA-1, we need the remote helper to
advertise support for the object-format extension and provide
information back to us so that we can properly parse refs and return
data. Ensure that the test remote helper understands these extensions.
Signed-off-by: brian m. carlson <redacted>
---
t/t5801/git-remote-testgit | 6 ++++++
1 file changed, 6 insertions(+)