From: Johan Herland <hidden> Date: 2016-06-15 22:51:18
Finally, I found some time to re-roll this series. Here's a quick
overview of the changes since the previous iteration:
- Rebased onto 'next' to include the deadlock fix from Peff and J6t.
- Reshuffle the patch series to leave the more contentious patches
towards the end of the series.
- (patch #3) Implement tighter matching rules in server_supports(),
as suggested by Junio in the previous thread.
- Remove --max-object-count from pack-objects and limit-object-count
capability, since object count is not considered a useful metric
for limiting pushes. However, keep the server-side object count
limit, since it is the only metric we can cheaply check on the
server-side (unless we change the pack format to include pack size
and/or commit count). This is now found in patch #10.
- (patch #9) In pack-objects, attempt to estimate the pack size before
we start writing out pack data. Abort as early as possible if the
estimated pack size exceeds the pack size limit. The estimate is
based on the in-pack size of already packed objects (assumed to be
reused as-is). The patch does not attempt to estimate the packed
size of currently loose objects. Therefore, whenever we're pushing
unpacked objects, we end up underestimating the pack size. This is
suboptimal, but ok, since we will still abort the transfer if we
exceed the pack size limit while writing out the pack data.
Have fun! :)
...Johan
Johan Herland (10):
Update technical docs to reflect side-band-64k capability in receive-pack
send-pack: Attempt to retrieve remote status even if pack-objects fails
Tighten rules for matching server capabilities in server_supports()
receive-pack: Prepare for addition of the new 'limit-*' family of capabilities
pack-objects: Teach new option --max-commit-count, limiting #commits in pack
send-pack/receive-pack: Allow server to refuse pushes with too many commits
pack-objects: Allow --max-pack-size to be used together with --stdout
send-pack/receive-pack: Allow server to refuse pushing too large packs
pack-objects: Estimate pack size; abort early if pack size limit is exceeded
receive-pack: Allow server to refuse pushes with too many objects
Documentation/config.txt | 27 ++++
Documentation/git-pack-objects.txt | 11 ++
Documentation/technical/pack-protocol.txt | 5 +-
Documentation/technical/protocol-capabilities.txt | 29 ++++-
builtin/pack-objects.c | 56 ++++++-
builtin/receive-pack.c | 42 +++++-
builtin/send-pack.c | 31 +++--
cache.h | 2 +-
connect.c | 30 ++++-
send-pack.h | 2 +
t/t5300-pack-object.sh | 77 +++++++++
t/t5400-send-pack.sh | 171 +++++++++++++++++++++
12 files changed, 453 insertions(+), 30 deletions(-)
--
1.7.5.rc1.3.g4d7b
From: Johan Herland <hidden> Date: 2016-06-15 22:51:18
When pushing, send-pack uses pack-objects to write the pack data to the
receive-pack process running on the remote end. The scenarios where
pack-objects dies unexpectedly, can be roughly divided based on whether
the reason for the failure is _local_ (i.e. something in pack-objects
caused it to fail of its own accord), or _remote_ (i.e. something in
the remote receive-pack process caused it to fail, leaving the local
pack-objects process with a broken pipe)
If the reason for the failure is local, we expect pack-objects to report
an appropriate error message to the user.
However, if the reason for the failure is remote, pack-objects will merely
abort because of the broken pipe, and the user is left with no clue as to
the reason why the remote receive-pack process died.
In certain cases, though, the receive-pack process on the other end may have
produced an error message immediately before exiting. This error message may
be currently waiting to be read by the local send-pack process.
Therefore, we should try to read from the remote end, even when pack-objects
dies unexepectedly. We accomplish this by _always_ calling receive_status()
after pack_objects(). If the remote end managed to produce a well-formed
status report before exiting, then receive_status() simply presents that to
the user. Even if the data from the remote end cannot be understood by
receive_status(), it will print that data as part of its error message. In
any case, we give the user as much information about the failure as possible.
Signed-off-by: Johan Herland <redacted>
---
builtin/send-pack.c | 13 +++----------
1 files changed, 3 insertions(+), 10 deletions(-)
@@ -251,7 +251,7 @@ int send_pack(struct send_pack_args *args,intstatus_report=0;intuse_sideband=0;unsignedcmds_sent=0;-intret;+intret=0;structasyncdemux;/* Does the other end support the reporting? */
@@ -339,25 +339,18 @@ int send_pack(struct send_pack_args *args,}if(new_refs&&cmds_sent){-if(pack_objects(out,remote_refs,extra_have,args)<0){-for(ref=remote_refs;ref;ref=ref->next)-ref->status=REF_STATUS_NONE;+if((ret=pack_objects(out,remote_refs,extra_have,args))){if(args->stateless_rpc)close(out);if(git_connection_is_socket(conn))shutdown(fd[0],SHUT_WR);-if(use_sideband)-finish_async(&demux);-return-1;}}if(args->stateless_rpc&&cmds_sent)packet_flush(out);if(status_report&&cmds_sent)-ret=receive_status(in,remote_refs);-else-ret=0;+ret|=receive_status(in,remote_refs);if(args->stateless_rpc)packet_flush(out);
From: Johan Herland <hidden> Date: 2016-06-15 22:51:18
When using server_supports() to match a given "feature" against the server
capabilities, follow these rules:
- "feature" must appear at the beginning of server_capabilities, or the
byte immediately before the matched location in server_capabilities
must be a SP; and
- if "feature" does not end with an equal sign, it does not expect a
value. The byte after the matched location in server_capabilities must
be either the end of string or a SP. A feature that expects a value is
checked with 'server_supports("feature=")' and the matched location in
server_capabilities can be followed by anything (i.e. if at the end of
string or a SP, it gets an empty string as the value, and otherwise it
will get the stretch of bytes after the '=' up to the next SP).
Given the server_capabilities string "foo=ab bar=froboz boz",
this patch should make it behave as follows:
server_supports("foo=") matches "foo=ab", returns "ab";
server_supports("ab") does not match anything;
server_supports("bar") does not match anything;
server_supports("boz") matches (and returns "boz"), without failing
at the end of bar=froboz that comes earlier.
Suggested-by: Junio C Hamano <redacted>
Signed-off-by: Johan Herland <redacted>
---
cache.h | 2 +-
connect.c | 30 +++++++++++++++++++++++++++---
2 files changed, 28 insertions(+), 4 deletions(-)
From: Johan Herland <hidden> Date: 2016-06-15 22:51:18
Currently we refuse combining --max-pack-size with --stdout since there's
no way to make multiple packs when the pack is written to stdout. However,
we want to be able to limit the maximum size of the pack created by
--stdout (and abort pack-objects if we are unable to meet that limit).
Therefore, when used together with --stdout, we reinterpret --max-pack-size
to indicate the maximum pack size which - if exceeded - will cause
pack-objects to abort with an error message.
Signed-off-by: Johan Herland <redacted>
---
Documentation/git-pack-objects.txt | 3 ++
builtin/pack-objects.c | 9 ++++---
t/t5300-pack-object.sh | 43 ++++++++++++++++++++++++++++++++++++
3 files changed, 51 insertions(+), 4 deletions(-)
@@ -112,6 +112,9 @@ base-name:: If specified, multiple packfiles may be created. The default is unlimited, unless the config variable `pack.packSizeLimit` is set.+++When used together with --stdout, the command will fail with an error+message if the pack output exceeds the given limit. --max-commit-count=<n>:: This option is only useful together with --stdout.
@@ -229,7 +229,7 @@ static unsigned long write_object(struct sha1file *f,if(!entry->delta)usable_delta=0;/* no delta */-elseif(!pack_size_limit)+elseif(!pack_size_limit||pack_to_stdout)usable_delta=1;/* unlimited packfile */elseif(entry->delta->idx.offset==(off_t)-1)usable_delta=0;/* base was written to another pack */
@@ -478,6 +478,9 @@ static void write_pack_file(void)*Ifso,rewriteitlikeinfast-import*/if(pack_to_stdout){+if(nr_written!=nr_remaining)+die("unable to make pack within the pack size"+" limit (%lu bytes)",pack_size_limit);sha1close(f,sha1,CSUM_CLOSE);}elseif(nr_written==nr_remaining){sha1close(f,sha1,CSUM_FSYNC);
@@ -2327,9 +2330,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)if(!pack_to_stdout&&!pack_size_limit)pack_size_limit=pack_size_limit_cfg;-if(pack_to_stdout&&pack_size_limit)-die("--max-pack-size cannot be used to build a pack for transfer.");-if(pack_size_limit&&pack_size_limit<1024*1024){+if(!pack_to_stdout&&pack_size_limit&&pack_size_limit<1024*1024){warning("minimum pack size limit is 1 MiB");pack_size_limit=1024*1024;}
From: Johan Herland <hidden> Date: 2016-06-15 22:51:18
This adds some technical documentation on the 'limit-*' family of
capabilities that will be added in the following commits.
Also refactor the generation of the capabilities declaration in receive-pack.
This will also be further expanded in the following commits.
Signed-off-by: Johan Herland <redacted>
---
Documentation/technical/pack-protocol.txt | 6 ++--
Documentation/technical/protocol-capabilities.txt | 22 +++++++++++++++++++++
builtin/receive-pack.c | 16 +++++++++++---
3 files changed, 37 insertions(+), 7 deletions(-)
@@ -391,8 +391,8 @@ The reference discovery phase is done nearly the same way as it is in the fetching protocol. Each reference obj-id and name on the server is sent in packet-line format to the client, followed by a flush-pkt. The only real difference is that the capability listing is different - the only-possible values are 'report-status', 'delete-refs', 'side-band-64k' and-'ofs-delta'.+possible values are 'report-status', 'delete-refs', 'side-band-64k',+'ofs-delta' and 'limit-*'. Reference Update Request and Packfile Transfer ----------------------------------------------
@@ -21,6 +21,9 @@ NOT advertise capabilities it does not understand. The 'report-status' and 'delete-refs' capabilities are sent and recognized by the receive-pack (push to server) process.+Any 'limit-*' capabilities may only be sent by the receive-pack+process. It is never requested by client.+ The 'side-band-64k' and 'ofs-delta' capabilities are sent and recognized by both upload-pack and receive-pack protocols.
@@ -185,3 +188,22 @@ it is capable of accepting a zero-id value as the target value of a reference update. It is not sent back by the client, it simply informs the client that it can be sent zero-id values to delete references.++limit-*+-------++If the server sends one or more capabilities that start with "limit-",+it means that there are certain limits to what kind of pack the server+will receive. More specifically, these capabilities must be of the form+"limit-<what>=<num>" where "<what>" (a sequence of lower-case letters,+digits and "-") describes which property of the pack is limited, and+"<num>" (a sequence of decimal digits) specifies the limit value.+Capabilities of this type are not sent back by the client; instead the+client must verify that the created packfile does not exceed the given+limits. This check should happen prior to transferring the packfile to+the server. If the check fails, the client must abort the upload, and+report the reason for the aborted push back to the user.+The following "limit-*" capabilites are recognized:++More "limit-*" capabilities may be added in the future. The client+is free to ignore any "limit-*" capabilities it does not understand.
From: Johan Herland <hidden> Date: 2016-06-15 22:51:18
Currently, when pushing a pack to the server that has specified a pack size
limit, we don't detect that we exceed that limit until we have already
generated (and started transmitting) that much pack data.
Ideally, we should be able to predict the approximate pack size _before_ we
start generating and transmitting the pack data, and abort early if the
estimated pack size exceeds the pack size limit.
This patch tries to provide such an estimate: It looks at the objects that
are to be included in the pack, and for already-packed objects, it assumes
that their compressed in-pack size is a good estimate of how much they will
contribute to the pack currently being generated. This assumption should be
valid as long as the objects are reused as-is.
For loose objects that are to be included in the pack, we currently have no
good estimate as to how much they will contribute to the pack size. Since
it's better to underestimate (because an overestimation will prevent us
from sending a pack that might actually be within the pack size limit),
we don't include loose objects at all in the pack size estimate. This makes
the estimate somewhat useless in common workflows (where the push happens
before (most of) the pushed objects are packed).
The estimate is generated before the "Compressing" and "Writing" phases of
the push, so if the estimate exceeds the pack size limit, we abort before
sending any pack data to the server.
If the estimate turns out to be too low (e.g. because we're pushing many
loose objects), there is still code in place to abort the push when we
reach the pack size limit during transmission.
Signed-off-by: Johan Herland <redacted>
---
I'm not really happy with excluding loose objects in the pack size
estimate. However, the size contributed by loose objects varies wildly
depending on whether a (good) delta is found. Therefore, any estimate
done at an early stage is bound to be wildly inaccurate. We could maybe
use some sort of absolute minimum size per object instead, but I
thought I should publish this version before spending more time futzing
with it...
A drawback of not including loose objects in the pack size estimate,
is that pushing loose objects is a very common use case (most people
push more often than they 'git gc'). However, for the pack sizes that
servers are most likely to refuse (hundreds of megabytes), most of
those objects will probably already be packed anyway (e.g. by
'git gc --auto'), so I still hope the pack size estimate will be useful
when it really matters.
...Johan
builtin/pack-objects.c | 23 +++++++++++++++++++++++
1 files changed, 23 insertions(+), 0 deletions(-)
From: Johan Herland <hidden> Date: 2016-06-15 22:51:18
Add a new receive.packSizeLimit config variable which defines an upper
limit on the pack size to accept in a single push.
This limit is advertised to clients, using the new "limit-pack-size=<num>"
capability. The client side - aka. send-pack - parses this capability and
forwards it to pack-objects, using the --max-pack-size option.
pack-objects then checks the generated pack against the limit and aborts
the pack transmission if the pack becomes too large.
However, older clients that do not understand the capability will not check
their pack against the limit, and will end up pushing the pack to the server.
Currently there is no extra check on the server to detect a push that exceeds
receive.packSizeLimit. However, such a check could be done in a pre-receive
or update hook.
Documentation and tests are included.
Signed-off-by: Johan Herland <redacted>
---
Documentation/config.txt | 9 +++
Documentation/technical/protocol-capabilities.txt | 1 +
builtin/receive-pack.c | 10 +++-
builtin/send-pack.c | 14 ++++-
send-pack.h | 1 +
t/t5400-send-pack.sh | 62 +++++++++++++++++++++
6 files changed, 93 insertions(+), 4 deletions(-)
@@ -1592,6 +1592,15 @@ receive.unpackLimit:: especially on slow filesystems. If not set, the value of `transfer.unpackLimit` is used instead.+receive.packSizeLimit::+ If the pack file transferred in a push exceeds this limit,+ then the entire push will be refused. This is meant to prevent+ an unintended large push (typically a result of the user not+ being aware of exactly what is being pushed, e.g. pushing a+ large rewritten history) from entering the repo. If not set,+ there is no upper limit on the size of the pack transferred+ in a single push.+ receive.commitCountLimit:: If the number of commits received in a push exceeds this limit, then the entire push will be refused. This is meant to prevent
@@ -205,6 +205,7 @@ the server. If the check fails, the client must abort the upload, and report the reason for the aborted push back to the user. The following "limit-*" capabilites are recognized:+ - limit-pack-size=<num> (Maximum size (in bytes) of uploaded pack) - limit-commit-count=<num> (Maximum number of commits in a pack) More "limit-*" capabilities may be added in the future. The client
@@ -287,4 +287,66 @@ test_expect_success 'push is allowed when commit limit is not exceeded' 'test"$parent_head"="$child_head"'+test_expect_success'verify that limit-pack-size capability is not advertised by default''+rewound_push_setup&&+(+cdparent&&+test_might_failgitreceive-pack.<../pkt-flush>output&&+test_must_failgrep-q"limit-pack-size"output+)+'++test_expect_success'verify that receive.packSizeLimit triggers limit-pack-size capability''+(+cdparent&&+gitconfigreceive.packSizeLimit10&&+test_might_failgitreceive-pack.<../pkt-flush>output&&+grep-q"limit-pack-size=10"output+)+'++test_expect_success'deny pushing when receive.packSizeLimit is exceeded''+(+cdchild&&+gitreset--hardorigin/master&&+echothree>file&&gitcommit-a-mthree&&+test_must_failgitsend-pack../parentmaster2>errs&&+grep-q"pack size limit"errs+)&&+parent_head=$(cdparent&&gitrev-parse--verifymaster)&&+child_head=$(cdchild&&gitrev-parse--verifymaster)&&+test"$parent_head"!="$child_head"+'++test_expect_success'repeated push failure proves that objects were not stored remotely''+(+cdchild&&+test_must_failgitsend-pack../parentmaster2>errs&&+grep-q"pack size limit"errs+)&&+parent_head=$(cdparent&&gitrev-parse--verifymaster)&&+child_head=$(cdchild&&gitrev-parse--verifymaster)&&+test"$parent_head"!="$child_head"+'++test_expect_success'increase receive.packSizeLimit''+(+cdparent&&+gitconfigreceive.packSizeLimit1000000&&+test_might_failgitreceive-pack.<../pkt-flush>output&&+grep-q"limit-pack-size=1000000"output+)+'++test_expect_success'push is allowed when pack size is not exceeded''+(+cdchild&&+gitsend-pack../parentmaster2>errs&&+test_must_failgrep-q"pack size limit"errs+)&&+parent_head=$(cdparent&&gitrev-parse--verifymaster)&&+child_head=$(cdchild&&gitrev-parse--verifymaster)&&+test"$parent_head"="$child_head"+'+ test_done
From: Johan Herland <hidden> Date: 2016-06-15 22:51:18
The new --max-commit-count option behaves similarly to --max-object-count,
when used together with --stdout: It limits the number of commits in the
pack written to stdout. If the pack would exceed this limit, pack-objects
will abort with an error message.
Unlike --max-pack-size and --max-object-count, --max-commit-count must
always be used together with --stdout. This is because using the commit
count to split packs is not at all a good heuristic, since Git does not
necessarily distribute commit objects uniformly across packs.
Documentation and tests are included.
Signed-off-by: Johan Herland <redacted>
---
Documentation/git-pack-objects.txt | 8 ++++++++
builtin/pack-objects.c | 24 +++++++++++++++++++++---
t/t5300-pack-object.sh | 34 ++++++++++++++++++++++++++++++++++
3 files changed, 63 insertions(+), 3 deletions(-)
@@ -113,6 +113,14 @@ base-name:: The default is unlimited, unless the config variable `pack.packSizeLimit` is set.+--max-commit-count=<n>::+ This option is only useful together with --stdout.+ Specifies the maximum number of commits allowed in the created+ pack. If the number of commits would exceed the given limit,+ pack-objects will fail with an error message.+ The number can be suffixed with "k", "m", or "g".+ The default is unlimited.+ --honor-pack-keep:: This flag causes an object already in a local pack that has a .keep file to be ignored, even if it would have
@@ -2322,6 +2334,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)pack_size_limit=1024*1024;}+if(!pack_to_stdout&&commit_count_limit)+die("--max-commit-count is only useful together with --stdout.");+if(!pack_to_stdout&&thin)die("--thin cannot be used to build an indexable pack.");
@@ -2348,6 +2363,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)if(non_empty&&!nr_result)return0;+if(pack_to_stdout&&commit_count_limit&&commit_count_limit<nr_commits)+die("unable to make pack within the commit count limit"+" (%lu commits)",commit_count_limit);if(nr_result)prepare_pack(window,depth);write_pack_file();
@@ -396,6 +396,40 @@ test_expect_success 'verify resulting packs' 'gitverify-packtest-11-*.pack'+test_expect_success'make a few more commits''+gitreset--hard$commit&&+echo"change">file&&+gitaddfile&&+gitcommit-msecond&&+commit2=`gitrev-parse--verifyHEAD`&&+echo"more change">>file&&+gitcommit-a-mthird&&+commit3=`gitrev-parse--verifyHEAD`&&+echo"even more change">>file&&+gitcommit-a-mfourth&&+commit4=`gitrev-parse--verifyHEAD`&&{+echo$commit&&+echo$commit2&&+echo$commit3&&+echo$commit4+}>>commit-list+'++test_expect_success'--stdout works with large enough --max-commit-count''+gitpack-objects--revs--stdout--max-commit-count=4<commit-list>test-17.pack&&+gitindex-pack--stricttest-17.pack+'++test_expect_success'verify resulting pack''+gitverify-packtest-17.pack+'++test_expect_success'--stdout fails when pack exceeds --max-commit-count''+test_must_failgitpack-objects--revs--stdout--max-commit-count=3<commit-list>test-18.pack2>errs&&+test_must_failgitindex-pack--stricttest-18.pack&&+grep-q"commit count limit"errs+'+## WARNING!#
From: Johan Herland <hidden> Date: 2016-06-15 22:51:18
Add a new receive.commitCountLimit config variable which defines an upper
limit on the number of commits to accept in a single push.
This limit is advertised to clients, using the new "limit-commit-count=<num>"
capability. The client side - aka. send-pack - parses this capability and
forwards it to pack-objects, using the recently added --max-commit-count
option. pack-objects then checks the generated pack against the limit and
aborts the pack generation if the pack would have too many commits.
However, older clients that do not understand the capability will not check
their pack against the limit, and will end up pushing the pack to the server.
Currently there is no extra check on the server to detect a push that exceeds
receive.commitCountLimit. However, such a check could be done in a pre-receive
or update hook.
Documentation and tests are included.
Signed-off-by: Johan Herland <redacted>
---
Documentation/config.txt | 9 +++
Documentation/technical/protocol-capabilities.txt | 2 +
builtin/receive-pack.c | 9 +++
builtin/send-pack.c | 10 +++
send-pack.h | 1 +
t/t5400-send-pack.sh | 65 +++++++++++++++++++++
6 files changed, 96 insertions(+), 0 deletions(-)
@@ -1592,6 +1592,15 @@ receive.unpackLimit:: especially on slow filesystems. If not set, the value of `transfer.unpackLimit` is used instead.+receive.commitCountLimit::+ If the number of commits received in a push exceeds this limit,+ then the entire push will be refused. This is meant to prevent+ an unintended large push (typically a result of the user not+ being aware of exactly what is being pushed, e.g. pushing a+ large rewritten history) from entering the repo. If not set,+ there is no upper limit on the number of commits transferred+ in a single push.+ receive.denyDeletes:: If set to true, git-receive-pack will deny a ref update that deletes the ref. Use this to prevent such a ref deletion via a push.
@@ -205,5 +205,7 @@ the server. If the check fails, the client must abort the upload, and report the reason for the aborted push back to the user. The following "limit-*" capabilites are recognized:+ - limit-commit-count=<num> (Maximum number of commits in a pack)+ More "limit-*" capabilities may be added in the future. The client is free to ignore any "limit-*" capabilities it does not understand.
@@ -253,6 +260,7 @@ int send_pack(struct send_pack_args *args,unsignedcmds_sent=0;intret=0;structasyncdemux;+constchar*p;/* Does the other end support the reporting? */if(server_supports("report-status"))
@@ -263,6 +271,8 @@ int send_pack(struct send_pack_args *args,args->use_ofs_delta=1;if(server_supports("side-band-64k"))use_sideband=1;+if((p=server_supports("limit-commit-count=")))+args->max_commit_count=strtoul(p,NULL,10);if(!remote_refs){fprintf(stderr,"No refs in common and none specified; doing nothing.\n"
@@ -222,4 +222,69 @@ test_expect_success 'deny pushing to delete current branch' ')'+echo"0000">pkt-flush++test_expect_success'verify that limit-commit-count capability is not advertised by default''+rewound_push_setup&&+(+cdparent&&+test_might_failgitreceive-pack.<../pkt-flush>output&&+test_must_failgrep-q"limit-commit-count"output+)+'++test_expect_success'verify that receive.commitCountLimit triggers limit-commit-count capability''+(+cdparent&&+gitconfigreceive.commitCountLimit1&&+test_might_failgitreceive-pack.<../pkt-flush>output&&+grep-q"limit-commit-count=1"output+)+'++test_expect_success'deny pushing when receive.commitCountLimit is exceeded''+(+cdchild&&+gitreset--hardorigin/master&&+echothree>file&&gitcommit-a-mthree&&+echofour>file&&gitcommit-a-mfour&&+test_must_failgitsend-pack../parentmaster2>errs&&+grep-q"commit count limit"errs+)&&+parent_head=$(cdparent&&gitrev-parse--verifymaster)&&+child_head=$(cdchild&&gitrev-parse--verifymaster)&&+test"$parent_head"!="$child_head"+'++test_expect_success'repeated push failure proves that objects were not stored remotely''+(+cdchild&&+test_must_failgitsend-pack../parentmaster2>errs&&+grep-q"commit count limit"errs+)&&+parent_head=$(cdparent&&gitrev-parse--verifymaster)&&+child_head=$(cdchild&&gitrev-parse--verifymaster)&&+test"$parent_head"!="$child_head"+'++test_expect_success'increase receive.commitCountLimit''+(+cdparent&&+gitconfigreceive.commitCountLimit2&&+test_might_failgitreceive-pack.<../pkt-flush>output&&+grep-q"limit-commit-count=2"output+)+'++test_expect_success'push is allowed when commit limit is not exceeded''+(+cdchild&&+gitsend-pack../parentmaster2>errs&&+test_must_failgrep-q"commit count limit"errs+)&&+parent_head=$(cdparent&&gitrev-parse--verifymaster)&&+child_head=$(cdchild&&gitrev-parse--verifymaster)&&+test"$parent_head"="$child_head"+'+ test_done
@@ -391,7 +391,8 @@ The reference discovery phase is done nearly the same way as it is in the fetching protocol. Each reference obj-id and name on the server is sent in packet-line format to the client, followed by a flush-pkt. The only real difference is that the capability listing is different - the only-possible values are 'report-status', 'delete-refs' and 'ofs-delta'.+possible values are 'report-status', 'delete-refs', 'side-band-64k' and+'ofs-delta'. Reference Update Request and Packfile Transfer ----------------------------------------------
@@ -21,8 +21,8 @@ NOT advertise capabilities it does not understand. The 'report-status' and 'delete-refs' capabilities are sent and recognized by the receive-pack (push to server) process.-The 'ofs-delta' capability is sent and recognized by both upload-pack-and receive-pack protocols.+The 'side-band-64k' and 'ofs-delta' capabilities are sent and+recognized by both upload-pack and receive-pack protocols. All other capabilities are only recognized by the upload-pack (fetch from server) process.
From: Johan Herland <hidden> Date: 2016-06-15 22:51:18
Add a new receive.objectCountLimit config variable which defines an upper
limit on the number of objects to accept in a single push. The server
aborts the transfer if the pack header received from the client indicates
a number of objects that exceeds this upper limit.
This limit is not advertised to clients, but is only enforced server-side.
When the limit is exceeded, the server sends a helpful error message to the
client, and then aborts the transfer, leaving the client with a broken pipe.
Server administrators might want to use this config variable to prevent
unintended large pushes from entering the repo (typically a result of the
user not being aware of exactly what is being pushed, e.g. pushing a large
rewritten history). Note that this config variable is not intended to protect
against DoS attacks, since there are countless other ways to attempt to DoS a
server without violating this limit.
Traditionally, this kind of limit would be imposed by a pre-receive or update
hook, but both of those run _after_ the pack has been received and stored by
receive-pack, so they cannot prevent the pack from being stored on the server.
Documentation and tests are included.
Signed-off-by: Johan Herland <redacted>
---
Documentation/config.txt | 9 +++++++++
builtin/receive-pack.c | 11 +++++++++--
t/t5400-send-pack.sh | 44 ++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 62 insertions(+), 2 deletions(-)
@@ -1610,6 +1610,15 @@ receive.commitCountLimit:: there is no upper limit on the number of commits transferred in a single push.+receive.objectCountLimit::+ If the number of objects received in a push exceeds this limit,+ then the entire push will be refused. This is meant to prevent+ an unintended large push (typically a result of the user not+ being aware of exactly what is being pushed, e.g. pushing a+ large rewritten history) from entering the repo. If not set,+ there is no upper limit on the number of objects transferred+ in a single push.+ receive.denyDeletes:: If set to true, git-receive-pack will deny a ref update that deletes the ref. Use this to prevent such a ref deletion via a push.
@@ -349,4 +349,48 @@ test_expect_success 'push is allowed when pack size is not exceeded' 'test"$parent_head"="$child_head"'+test_expect_success'deny pushing when receive.objectCountLimit is exceeded''+rewound_push_setup&&+(+cdparent&&+gitconfigreceive.objectCountLimit1+)&&+(+cdchild&&+gitreset--hardorigin/master&&+echothree>file&&gitcommit-a-mthree&&+test_must_failgitsend-pack../parentmaster2>errs&&+grep-q"receive\\.objectCountLimit"errs+)&&+parent_head=$(cdparent&&gitrev-parse--verifymaster)&&+child_head=$(cdchild&&gitrev-parse--verifymaster)&&+test"$parent_head"!="$child_head"+'++test_expect_success'repeated push failure proves that objects were not stored remotely''+(+cdchild&&+test_must_failgitsend-pack../parentmaster2>errs&&+grep-q"receive\\.objectCountLimit"errs+)&&+parent_head=$(cdparent&&gitrev-parse--verifymaster)&&+child_head=$(cdchild&&gitrev-parse--verifymaster)&&+test"$parent_head"!="$child_head"+'++test_expect_success'push is allowed when object limit is increased''+(+cdparent&&+gitconfigreceive.objectCountLimit10+)&&+(+cdchild&&+gitsend-pack../parentmaster2>errs&&+test_must_failgrep-q"receive\\.objectCountLimit"errs+)&&+parent_head=$(cdparent&&gitrev-parse--verifymaster)&&+child_head=$(cdchild&&gitrev-parse--verifymaster)&&+test"$parent_head"="$child_head"+'+ test_done
On Sun, May 22, 2011 at 17:52, Johan Herland [off-list ref] wrote:
Currently, when pushing a pack to the server that has specified a pack size
limit, we don't detect that we exceed that limit until we have already
generated (and started transmitting) that much pack data.
Ideally, we should be able to predict the approximate pack size _before_ we
start generating and transmitting the pack data, and abort early if the
estimated pack size exceeds the pack size limit.
This patch tries to provide such an estimate: It looks at the objects that
are to be included in the pack, and for already-packed objects, it assumes
that their compressed in-pack size is a good estimate of how much they will
contribute to the pack currently being generated. This assumption should be
valid as long as the objects are reused as-is.
This looks good to me.
I'm not really happy with excluding loose objects in the pack size
estimate. However, the size contributed by loose objects varies wildly
depending on whether a (good) delta is found. Therefore, any estimate
done at an early stage is bound to be wildly inaccurate. We could maybe
use some sort of absolute minimum size per object instead, but I
thought I should publish this version before spending more time futzing
with it...
A drawback of not including loose objects in the pack size estimate,
is that pushing loose objects is a very common use case (most people
push more often than they 'git gc'). However, for the pack sizes that
servers are most likely to refuse (hundreds of megabytes), most of
those objects will probably already be packed anyway (e.g. by
'git gc --auto'), so I still hope the pack size estimate will be useful
when it really matters.
That is my impression too. Most servers using this feature will
probably put a limit of at least 10MB. Once you get into the 25-100M
range, the client probably has already packed the bulk of that
content. Especially if we also have Junio's new stream large blobs to
packs during git add patch. So as you point out, cases where this is
mostly useful (really huge push) this is likely to still trigger
correctly.
We can still get a tighter estimate if we wanted to. I wouldn't mix it
into this patch, but make a new one on top of it. During delta
compression we hold onto deltas, or at least compute and retain the
size of the chosen delta. We could re-check the pack size after the
Compressing phase by including the delta sizes in the estimate, and if
we are over, abort before writing.
For non-delta, non-reuse we may be able to guess by just using the
loose object size. The loose object is most likely compressed at the
same compression ratio as the outgoing pack stream will use, so a
deflate(inflate(loose)) cycle is going to be very close in total bytes
used. If we over shoot the limit by more than some fudge factor (say
8K in 1M limit or 0.7%), abort before writing.
--
Shawn.
From: Johan Herland <hidden> Date: 2016-06-15 22:51:18
On Monday 23. May 2011, Shawn Pearce wrote:
We can still get a tighter estimate if we wanted to. I wouldn't mix
it into this patch, but make a new one on top of it. During delta
compression we hold onto deltas, or at least compute and retain the
size of the chosen delta. We could re-check the pack size after the
Compressing phase by including the delta sizes in the estimate, and
if we are over, abort before writing.
Ok. Not sure when I'll have the time/courage to dive into this, but I'll
at least give it a try.
For non-delta, non-reuse we may be able to guess by just using the
loose object size. The loose object is most likely compressed at the
same compression ratio as the outgoing pack stream will use, so a
deflate(inflate(loose)) cycle is going to be very close in total
bytes used. If we over shoot the limit by more than some fudge
factor (say 8K in 1M limit or 0.7%), abort before writing.
I already have an unsubmitted patch on top of the series that includes
the on-disk/compressed size of loose objects in the estimate. However,
it's quite intrusive (need to extend sha1_object_info() to return
compressed size of loose objects). Also, since I don't yet take the
delta compression into account, these numbers are obviously unreliable.
That said, in the cases where loose objects are not deltified it seems
the compressed/loose versions are about 3 to 7 bytes larger than the
corresponding compressed/packed versions. I guess this is due to the
loose files using a "<type> SP <size> NUL" text header (deflated),
whereas the pack uses a more compact binary format (not deflated).
We could test a large corpus (e.g. linux-kernel) to find the average
difference between compressed/loose size and compressed/packed size, and
then multiply this with the number of non-delta, non-reuse object to
determine the fudge factor you describe above.
Have fun! :)
...Johan
--
Johan Herland, [off-list ref]
www.herland.net
From: Junio C Hamano <hidden> Date: 2016-06-15 22:51:18
Johan Herland [off-list ref] writes:
quoted hunk
This adds some technical documentation on the 'limit-*' family of
capabilities that will be added in the following commits.
Also refactor the generation of the capabilities declaration in receive-pack.
This will also be further expanded in the following commits.
Signed-off-by: Johan Herland <redacted>
---
...
From: Junio C Hamano <hidden> Date: 2016-06-15 22:51:18
Johan Herland [off-list ref] writes:
However, older clients that do not understand the capability will not check
their pack against the limit, and will end up pushing the pack to the server.
Currently there is no extra check on the server to detect a push that exceeds
receive.commitCountLimit. However, such a check could be done in a pre-receive
or update hook.
I found the above a reasonable thing to do. In other words, this is an
advisory configuration at this point (and from a cursory scanning of the
rest of the series, throughout the series), and that is OK.
@@ -1592,6 +1592,15 @@ receive.unpackLimit:: especially on slow filesystems. If not set, the value of `transfer.unpackLimit` is used instead.+receive.commitCountLimit::+ If the number of commits received in a push exceeds this limit,+ then the entire push will be refused. This is meant to prevent+ an unintended large push (typically a result of the user not+ being aware of exactly what is being pushed, e.g. pushing a+ large rewritten history) from entering the repo. If not set,+ there is no upper limit on the number of commits transferred+ in a single push.
But then it may probably be a good idea to reword this a bit, to clarify
the refusal happens voluntarily by the pusher. E.g.
Tell "git push" not to push more than this many commits at once
into this repository. This is meant to prevent ... in a single
push. Note that older versions of "git push" may ignore this
advisory, so if you really want to refuse such a push, you would
need to arrange to do so in either the pre-receive hook or the
update hook.
@@ -205,5 +205,7 @@ the server. If the check fails, the client must abort the upload, and report the reason for the aborted push back to the user. The following "limit-*" capabilites are recognized:+ - limit-commit-count=<num> (Maximum number of commits in a pack)+
I think s/in a pack/to transfer/ is more appropriate.
It is a non-essential detail that the current implementation carries only
one pack in a single session between send-pack and receive-pack. When we
update the protocol (with another capability) so that we can send more
than one packs in a single session, we would want the maximum number of
commits to be honored.
Come to think of it, I do not necessarily agree with the earlier "max
commit count can only be used with max pack size"; I can accept it if the
statement is qualified with "for now", though.
It is entirely reasonable to say that I want to split packs in 2GB chunks,
and I want to keep the number of commits in the resulting packs (notice
the plural) under this fixed ceiling to avoid mistakes, no?
@@ -253,6 +260,7 @@ int send_pack(struct send_pack_args *args, unsigned cmds_sent = 0; int ret = 0; struct async demux;+ const char *p; /* Does the other end support the reporting? */ if (server_supports("report-status"))
@@ -263,6 +271,8 @@ int send_pack(struct send_pack_args *args, args->use_ofs_delta = 1; if (server_supports("side-band-64k")) use_sideband = 1;+ if ((p = server_supports("limit-commit-count=")))+ args->max_commit_count = strtoul(p, NULL, 10);
If we find garbage in *p, we would just run with a random limit, which may
cause the pack-objects to abort, but that still is a controlled failure
and is acceptable.
From: Johan Herland <hidden> Date: 2016-06-15 22:51:18
On Monday 23 May 2011, Junio C Hamano wrote:
Johan Herland [off-list ref] writes:
quoted
This adds some technical documentation on the 'limit-*' family of
capabilities that will be added in the following commits.
Also refactor the generation of the capabilities declaration in
receive-pack. This will also be further expanded in the following
commits.
Signed-off-by: Johan Herland <redacted>
---
...
From: Johan Herland <hidden> Date: 2016-06-15 22:51:18
On Tuesday 24 May 2011, Junio C Hamano wrote:
Johan Herland [off-list ref] writes:
quoted
However, older clients that do not understand the capability will not
check their pack against the limit, and will end up pushing the pack
to the server. Currently there is no extra check on the server to
detect a push that exceeds receive.commitCountLimit. However, such a
check could be done in a pre-receive or update hook.
I found the above a reasonable thing to do. In other words, this is an
advisory configuration at this point (and from a cursory scanning of the
rest of the series, throughout the series), and that is OK.
@@ -1592,6 +1592,15 @@ receive.unpackLimit:: especially on slow filesystems. If not set, the value of `transfer.unpackLimit` is used instead.+receive.commitCountLimit::+ If the number of commits received in a push exceeds this limit,+ then the entire push will be refused. This is meant to prevent+ an unintended large push (typically a result of the user not+ being aware of exactly what is being pushed, e.g. pushing a+ large rewritten history) from entering the repo. If not set,+ there is no upper limit on the number of commits transferred+ in a single push.
But then it may probably be a good idea to reword this a bit, to clarify
the refusal happens voluntarily by the pusher. E.g.
Tell "git push" not to push more than this many commits at once
into this repository. This is meant to prevent ... in a single
push. Note that older versions of "git push" may ignore this
advisory, so if you really want to refuse such a push, you would
need to arrange to do so in either the pre-receive hook or the
update hook.
@@ -205,5 +205,7 @@ the server. If the check fails, the client must
abort the upload, and
report the reason for the aborted push back to the user.
The following "limit-*" capabilites are recognized:
+ - limit-commit-count=<num> (Maximum number of commits in a pack)
+
I think s/in a pack/to transfer/ is more appropriate.
It is a non-essential detail that the current implementation carries only
one pack in a single session between send-pack and receive-pack. When we
update the protocol (with another capability) so that we can send more
than one packs in a single session, we would want the maximum number of
commits to be honored.
Agreed.
Come to think of it, I do not necessarily agree with the earlier "max
commit count can only be used with max pack size"; I can accept it if the
statement is qualified with "for now", though.
I'll add the qualification.
It is entirely reasonable to say that I want to split packs in 2GB
chunks, and I want to keep the number of commits in the resulting packs
(notice the plural) under this fixed ceiling to avoid mistakes, no?
I guess it depends on whether you interpret the commit count limit as a per-
pack threshold that triggers pack splitting (similar to how we interpret the
pack size limit), or as an upper bound which aborts pack-objects if
exceeded.
I initially found it more intuitive to interpret all of these as a fixed
upper bound when paired with --stdout (since that implicitly limits us to a
single pack), and as a pack splitting threshold when used without --stdout
(except that triggering pack splits based on commit count is not useful).
40 is 19 plus terminating NUL plus 20-decimal digits to hold the count?
Indeed. I will document this more clearly.
quoted
@@ -263,6 +271,8 @@ int send_pack(struct send_pack_args *args, args->use_ofs_delta = 1; if (server_supports("side-band-64k")) use_sideband = 1;+ if ((p = server_supports("limit-commit-count=")))+ args->max_commit_count = strtoul(p, NULL, 10);
If we find garbage in *p, we would just run with a random limit, which
may cause the pack-objects to abort, but that still is a controlled
failure and is acceptable.
Agreed.
...Johan
--
Johan Herland, [off-list ref]
www.herland.net